windy-git/docs/incidents/2026-08-12-account-server-login-outage.md

Incident — account.windyword.ai login outage, 2026-08-12

Found while adding a dashboard tile. Not caused by Windy Git. Recorded here because Windy Git is the ecosystem's verification cell and this is the failure mode its CI exists to catch earlier.

Impact

POST /api/v1/auth/login timed out for at least ~50 minutes (45 s+, no response). /health and /.well-known/jwks.json also timed out for part of it. This is the identity service every Windy product authenticates against — Word, Chat, Mail, Cloud, Eternitas hand-offs, and Windy Git's own OIDC.

Root cause — two faults multiplying

1. A hot retry loop. windy-agent-roster called POST /api/v1/identity/mail/address-by-windy-id for every identity it knows, got a timeout, retried immediately, and never backed off. Measured: 74 failures/minute, 2,220 in 30 minutes, sustained. Every log line read Pro mail-lookup failed for <uuid>: The operation was aborted due to timeout.

2. Every one of those calls is expensive. account-server/src/db/postgres-adapter.ts:114 runs execFileSync with a new node -e child process per query — new pg client, new TCP connection, new TLS handshake, blocking the event loop up to 30 s. Confirmed live: a host process reading node -e const { Client } = require('pg'); .... Each 404 cost 0.6–2.1 s of blocking work.

The multiplication is the story. Either alone is survivable. Together they form a positive feedback loop: slow responses cause timeouts, timeouts cause immediate retries, retries add load, load makes responses slower. The listening socket inside the container showed Recv-Q 510 — 510 connections accepted by the kernel that node was too blocked to pick up. Login sat in that queue.

Background condition: 54 containers on a 4-vCPU box, 12 of them dev/demo (eternitas-dev-*, windymind-dev-*, nacholos-demo-*, nachope-demo-*). Baseline load 17–19. That is the amplifier — a process fork is cheap on an idle box and ruinous on a saturated one.

Resolution — two steps

1. Stop the bleeding. docker stop windy-agent-roster at 16:50:57Z. Login recovered from timeout to HTTP 200 immediately.

2. Ship the real fix and bring agent chat back. The retry fix already existed: a parallel session diagnosed the same incident from the windy-pro side and landed c79f196 — "fix(roster): back off failed mail lookups instead of retrying every 30s forever" (#172) — with a test (services/agent-roster/tests/mail-lookup-backoff.test.js). Kit 0 was one commit behind and did not have it. That is the whole reason the loop ran.

Deployed by fast-forwarding /root/windy-chat ac61db6 → c79f196 and rebuilding only agent-roster (--no-deps; nothing else on the box touched). Deliberately git merge --ff-only, never reset --hard — this checkout has a documented history of local edits a hard reset would silently eat.

The fix caches failures per owner with exponential backoff (60 s → 30 min cap) and suppresses logging after three attempts. Observed working in production: attempt 2, backing off 120s.

	before	after
login	timeout at 45 s+	HTTP 200 in ~18 s
/health	timeout	200 in 0.28 s
jwks	timeout	200 in 0.16 s
account-server CPU	168–210 %	0.00 %
roster mail-lookup failures	74/min	0/min
account-server calls to that route	~50/min	0/min
agent chat	down (stopped)	back up, healthy

18 s is restored, not healthy. A login should be well under a second. That number is the fork-per-query adapter under a loaded box, and it is what remains after the loop was removed.

What is still true

• The mail lookups still fail — the backoff stops them amplifying, it does not make them succeed. Each owner now retries once per backoff window instead of every 30 s. The underlying 404/timeout is unexplained. The route exists (identity.ts:1014-1022, reads x-service-token). Either the identities are genuinely absent or the caller sends the wrong shape — worth knowing before the roster returns.
• The postgres adapter is unchanged. It is the single largest stability liability in the ecosystem, 468 call sites, and a multi-week job.

What would have caught this

Nothing did. There is no alerting on account-server latency, and the fleet canary has been dead since 2026-07-03. The service was (unhealthy) with a failing healthcheck streak of 74 and nothing said so.

Recommendations, ordered

1. Fix the retry — done, c79f196, deployed 2026-08-12.
2. Alert on the healthcheck. A 74-deep failing streak on the identity service should page, not sit.
3. Move dev/demo off the production box. 12 containers of non-production load on the box that runs identity, the CA, mail, Matrix and the broker.
4. Then the postgres-adapter migration. Hottest paths first — login is the obvious first path.

The lesson worth keeping

The fix was written, tested, reviewed and merged — and the outage happened anyway, because Kit 0 was one commit behind. A merged fix that has not reached production is not a fix; it is a belief.

That is the same root cause both August audits named — nothing anywhere checks whether a decision reached production — arriving as a live outage rather than a finding in a report. Deploy verification is not paperwork.

Diagnostic note for whoever is next

docker logs / docker stats loops across 54 containers cost real CPU on a saturated box. Load rose from 17 to 24.9 while this was being investigated, and some of that was the investigation. Take one clean measurement, then back off.