Caught by TESTING the alert path instead of assuming it. Without an explicit
User-Agent, urllib sends 'Python-urllib/3.x' and Resend rejects it 403, while
the identical request via curl succeeds.
The failure mode this avoids is the worst one a canary has: it would have
detected every outage correctly and told nobody. Same bot-filtering trap as the
Gitea migrate call earlier today — worth recognising on sight.
Also prints the HTTP body on failure. '403 Forbidden' alone sends you hunting
for a bad key; the body names the real cause.
Verified: alert sent (200).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Today's outage is the whole design brief: /health returned 200 for the entire
hour that login was dead. A canary watching /health would have stayed green
while nobody in the ecosystem could sign in. So the login probe is here and it
is the one that matters.
Three rules it obeys:
- never green for something it did not prove (I-8)
- alert on TRANSITIONS, not every run — a canary people filter is a dead
canary, which is how the last one sat 37 days dead unnoticed
- run where the watched thing cannot take it down: Veron 1, never Kit 0
Two independent signals, so losing one still leaves the other: an email via
Resend on state change, and a non-zero exit that turns the CI run red in the
forge itself.
Alerts say what broke in human terms — 'a human can actually sign in' — rather
than only naming an endpoint.
Verified against production: 7/7 green including login at 17.1s; a forced 404
reports DOWN; a 1s threshold reports SLOW at 23.3s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Direction is deliberate. For the migration quarter GitHub stays upstream and
Windy Git is a PULL mirror, so it cannot diverge — worst case it is stale, not
wrong. Making Windy Git authoritative before Grant flips G11.6 would create a
two-writer problem nobody asked for. I-4's push-mirror is the steady state for
repos that originate here.
Refuses windy-pro by name: six checkouts, a build counter forked three ways, and
two sessions recording different HEADs hours apart. That gets resolved by
reading, not by importing (G11.5).
Records the Cloudflare trap: the public endpoint answers 403 error 1010 because
CF blocks urllib's user-agent as a bot signature. It reads like Gitea rejecting
the token and is not — the identical call against localhost:3080 on the host
succeeds. Bulk import belongs on the host anyway.
Verified: windy-calendar imported, 731 KB, mirroring.
Survey: 135 private repos, and GitHub Actions cannot run on ANY of them. Sampled
windy-pro, windy-mind, eternitas, windy-registry, windy-agent, WindyCloud — every
recent run is startup_failure, as recently as today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Strand G0 complete and VERIFIED against real Postgres, not asserted.
- FastAPI plane, fail-closed provider seams, repair-pointer error taxonomy
- migration 001: all 10 tables incl. repo_type NOT NULL and model_cards (I-7)
- 17 invariant tests, ruff clean, vocabulary audit clean
Two bugs found by RUNNING it that review would not have caught:
1. SQLAlchemy Enum persists .name, not .value — so RepoState.deleted_soft
and CreatedVia.imported would have written labels migration 001 never
declared, failing at runtime rather than at review. Pinned via
values_callable.
2. op.create_table asks each Enum to emit its own CREATE TYPE with no
checkfirst, so the second reference raised DuplicateObject and the
migration died halfway. Types are now created once, referenced with
create_type=False.
Proven live, with the hostile env var set:
- I-12: COMMIT_SHA=deadbeef... in the environment, /version reports real HEAD.
That env pin is the documented root cause of nine sibling services
misreporting their commit; here it is structurally ignored.
- I-8: three unconfigured providers -> status degraded, HTTP 503, each saying
'refusing to report healthy'. No mock, no false green.
- G0.4: upgrade -> downgrade -> upgrade round-trip clean (10 -> 0 -> 10).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
