G7.6: fleet canary — probes what a user does, not what is cheap
All checks were successful
check / gate (push) Successful in 19s
All checks were successful
check / gate (push) Successful in 19s
Today's outage is the whole design brief: /health returned 200 for the entire
hour that login was dead. A canary watching /health would have stayed green
while nobody in the ecosystem could sign in. So the login probe is here and it
is the one that matters.
Three rules it obeys:
- never green for something it did not prove (I-8)
- alert on TRANSITIONS, not every run — a canary people filter is a dead
canary, which is how the last one sat 37 days dead unnoticed
- run where the watched thing cannot take it down: Veron 1, never Kit 0
Two independent signals, so losing one still leaves the other: an email via
Resend on state change, and a non-zero exit that turns the CI run red in the
forge itself.
Alerts say what broke in human terms — 'a human can actually sign in' — rather
than only naming an endpoint.
Verified against production: 7/7 green including login at 17.1s; a forced 404
reports DOWN; a 1s threshold reports SLOW at 23.3s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
48
.gitea/workflows/canary.yml
Normal file
48
.gitea/workflows/canary.yml
Normal file
@@ -0,0 +1,48 @@
|
||||
# The fleet canary (G7.6).
|
||||
#
|
||||
# Runs on Veron 1, DELIBERATELY not on Kit 0. A canary hosted on the box it
|
||||
# watches dies with that box and reports nothing at the exact moment it matters.
|
||||
#
|
||||
# The previous fleet canary (kit-army-config/docs/deployed-state.json) sat dead
|
||||
# for 37+ days because its workflow returned startup_failure on every run and
|
||||
# nothing watched the watcher. This one has two independent signals: an email on
|
||||
# state change, and a red CI run in the forge. Losing one still leaves the other.
|
||||
|
||||
name: canary
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Every 10 minutes. Frequent enough that an outage is measured in minutes,
|
||||
# infrequent enough that the login probe (~18s of real work on a loaded box)
|
||||
# is not itself a load source.
|
||||
- cron: "*/10 * * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
probe:
|
||||
runs-on: veron-1
|
||||
timeout-minutes: 8
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
# State persists between runs so "still broken" can be told apart from
|
||||
# "just broke" — that is what keeps this from emailing every 10 minutes
|
||||
# during an outage, and a canary people filter is a dead canary.
|
||||
- name: restore canary state
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: canary-state.json
|
||||
key: canary-state-${{ github.run_id }}
|
||||
restore-keys: canary-state-
|
||||
|
||||
- name: probe
|
||||
env:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
CANARY_LOGIN_EMAIL: ${{ secrets.CANARY_LOGIN_EMAIL }}
|
||||
CANARY_LOGIN_PASSWORD: ${{ secrets.CANARY_LOGIN_PASSWORD }}
|
||||
CANARY_ALERT_TO: ${{ secrets.CANARY_ALERT_TO }}
|
||||
run: python3 scripts/canary.py
|
||||
Reference in New Issue
Block a user