69 Commits

Author SHA1 Message Date
Kit OC5
a2daca61ef ci: mount windy-pro's read-only build inputs into dind; allow exactly that path for jobs
Non-secret inputs git-ignored in windy-pro (models, linux-x64 portable
bundle, enter-monitor build) that build-desktop needs. Mounted :ro into
dind; valid_volumes allows only /ci-inputs/windy-pro; refresh-ci-inputs.sh
copies them from the frozen release clone (read-only on the source).
Invariant I-5 narrowed, not dropped: exactly that one path, read-only in
dind, no other service mounts it, still no docker socket (proven to fail
on :rw). Orchestrator-approved (option a). Applied in an idle window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:56:48 -04:00
Kit OC5
1bc55eaec9 compute guard: flag direct AI-provider use (Windy Mind is the only door), warn-only
All checks were successful
check / gate (push) Successful in 28s
canary / probe (push) Successful in 14s
Grant's rule (09-23): every model call goes through Windy Mind. The bridge
now posts windy-git/compute-guard on every PR head (lines the PR ADDS vs its
merge-base) and default-branch head (whole tree): provider hosts, provider
SDK imports/deps and raw provider key names. Warn-only: success + "⚠ WARN"
and a link to the first hit; COMPUTE_GUARD_MODE=block turns it red later.

Exceptions live in ci/compute-guard-allow.yml, each with a reason (Mind
itself, user-BYOK windy-agent / windy-code extension / windy-pro desktop +
MindPanel, windy-connect config writers). Tests, docs, comments, lockfiles,
vendored code and CI config are never scanned. Reads the sync's bare clones
(no docker exec); cached per (repo, sha, rules). Non-fatal; never a fake OK.

First cases = COMPUTE_BYPASS_AUDIT.md. Today on default branches: 38
findings in 3 repos (windy-chat audit #2, windy-pro account-server #3/#4,
windytalk reference/), 0 elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:42:10 -04:00
Kit OC5
fdb0f5989e ci: run dind under Sysbox, not privileged (rollback override kept)
Some checks failed
canary / probe (push) Has been cancelled
check / gate (push) Has been cancelled
dind was privileged: true, so a job that escaped into dind was root on
Veron 1, which is Grant's workstation. Under sysbox-runc (sysbox-ce 0.7.1,
installed 09-23 with no docker restart) dind root is an unprivileged host
uid. Smoke-tested standalone: nested containers, internet, a services-style
postgres on a private network and a python image all pass unprivileged.
Fresh volume dind-storage-sysbox; the old dind-storage stays for
docker-compose.privileged.yml, the one-command rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:23:55 -04:00
Kit OC5
9be2952ff8 rerun_ci: detect a running oneshot sync correctly (is-active lies for oneshot)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:22:59 -04:00
Kit OC5
fe3bce39ff bridge: post pending for queued jobs a runner can take
All checks were successful
check / gate (push) Successful in 28s
Gitea 1.24 lists only picked-up jobs, so a queued PR showed NOTHING on
GitHub and lanes asked whether their push was lost (Windy Mind #131,
Windy Cloud today). The bridge now reads waiting jobs from the gitea DB
and posts pending where nothing newer was picked up; a queued re-run
supersedes the stale failure it replaces.

Only status 5 jobs whose runs-on labels a live runner has: blocked jobs
often end skipped and label-unrunnable jobs are cancelled unpicked, and
neither ever reaches /actions/tasks, so their pending would never resolve.
Lookup is bounded (30 s) and non-fatal: the IO-stall lesson.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:20:32 -04:00
Kit OC5
6b0b60eb4a scripts: rerun_ci.sh — re-fire a PR's CI without the web button
Gitea 1.24 has no rerun API and the web button needs Grant's SSO identity.
Guarded branch rewind that the next sync undoes; restores the branch itself
on timeout. Used today for eternitas #166 and windy-mind #131 after the
Veron IO stall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:17:10 -04:00
Kit OC5
60708dd8db push velocity: correlate the SSO-id subquery on the grouped column
All checks were successful
check / gate (push) Successful in 21s
canary / probe (push) Successful in 9s
Dry-run against the real gitea DB: 'subquery uses ungrouped column u.id'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:06:27 -04:00
Kit OC5
aedc772d29 push velocity: isolate its query so a failure never costs ci.run rows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:06:27 -04:00
Kit OC5
acb8ec3a16 push velocity: key humans on windy_identity_id (SSO link); no id -> system + caller
Telemetry UPDATE 2 actor rule: agent/human rows without actor_id are
quarantined. Forge humans sign in only via Windy SSO, so Gitea's
external_login_user.external_id is their windy_identity_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:06:27 -04:00
Kit OC5
0051c72037 telemetry: detect push velocity from Gitea's action table (alert only)
git push never touches our API, so throttle.py can't see it. Gitea's
action table records every push; the 5-min sync-side emitter now reads
it and emits forge.push_velocity when an account crosses 60 pushes/1h,
500 pushes/24h (standard-band base) or 10 ref deletes/24h. One row per
account per rule per window while over; windyadmin (the sync) exempt.
Nothing sits in the push path and nothing is refused. HOLD until
Telemetry Boss declares the shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:06:27 -04:00
Kit OC5
4c76b1b12a canary: end the hub session the login probe opens (journey cleanup rule)
All checks were successful
check / gate (push) Successful in 32s
canary / probe (push) Successful in 12s
identity.login created a live hub session every 10 min and never ended it.
It now logs out with the token it got: retried on 5xx / no response
(8 x 15 s), 401/404/410 = already over, any other 4xx fails fast, and a
cleanup it can't finish is reported as identity.logout DOWN "CLEANUP
FAILED" (alerts + red run). The hub's /auth/logout revokes every refresh
token of the account (verified live), so the next run's logout heals a
leftover; no ledger needed. Proven end to end: login 200, logout 200,
10/10 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 13:39:55 -04:00
Kit OC5
2d4fadb090 sync: bound the janitor and telemetry steps (docker exec hangs in an IO stall)
Some checks failed
canary / probe (push) Has been cancelled
check / gate (push) Has been cancelled
09-23 16:43Z the Veron data2 SMR stall left runc exec in D state; the
janitor's docker exec never returned, so the sync sat 'activating' and no
repo mirrored or got a status for any lane. Both steps are non-fatal;
now they time out (120 s / 180 s) and the run continues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:54:34 -04:00
Kit OC5
e3b69fa759 telemetry: UPDATE 7 — read the ingest body; count quarantined + dropped on heartbeats
Some checks failed
canary / probe (push) Has been cancelled
check / gate (push) Has been cancelled
The ledger answers 202 even when it quarantines rows. Both emitters now log
a warning with the reasons and report service.health.telemetry_quarantined
and telemetry_dropped (API: buffer overflow; sync: 0 by construction, since
a failed send keeps cursor + spool). HOLD until Telemetry Boss declares both
keys on windy-git's two service.health shapes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:32:28 -04:00
Kit OC5
4acf50d9ef bridge tests: fake serves workflow contents; cover invalid-workflow status
All checks were successful
check / gate (push) Successful in 23s
b7a7e94 made the bridge read workflow files, which the strict fake Gitea
refused (7 red). The fake now serves contents (404 when absent), and new
tests cover: error posted with no runs, valid files add nothing, no repost,
.gitea/workflows wins over .github/workflows, and each workflow_problem
shape. pyyaml declared in dev extras (the bridge imports it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:31:19 -04:00
Kit OC5
b7a7e94df0 bridge: post an error status when Windy Git ignores an invalid workflow
Gitea drops an invalid workflow file with one log line and fires no run, so
the GitHub PR showed nothing and lanes waited for CI that never came
(windytalk #100). The bridge now reads each workflow file at the commit it
reports on and posts windy-git/<wf>/workflow = error with the reason.
Verified: 0 false positives on all 23 bridged repos' main; catches
windytalk #100's broken commits (invalid YAML at line 12), fix commit clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:12 -04:00
c83f808a60 bridge: retry transport blips (TLS timeout/reset), never HTTP errors
All checks were successful
check / gate (push) Successful in 20s
canary / probe (push) Successful in 9s
A single GitHub TLS handshake timeout failed the whole sync, flipped its
windy-job heartbeat to ok:false and would page for nothing. Up to 3
attempts with backoff for URLError/timeout/reset; HTTP errors return
immediately as before. Test covers both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:15:22 -04:00
eb27e63db3 telemetry: adopt the end-to-end synthetic convention (UPDATE 4)
All checks were successful
check / gate (push) Successful in 24s
canary / probe (push) Successful in 9s
Replaces the keyed marker from 1c3b5b0 with the ecosystem convention:
any X-Windy-Synthetic value marks the request synthetic; the flag lives in
a per-request contextvar, labels this request's rows, and is FORWARDED on
downstream calls (Eternitas trust lookup, Gitea API). The canary sends
"1". Rows are still recorded; the label separates, never suppresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:06:51 -04:00
1c3b5b0638 telemetry: synthetic:true on canary refusals (keyed, not a bare flag)
All checks were successful
check / gate (push) Successful in 25s
canary / probe (push) Successful in 7s
The canary deliberately sends forged tokens every 10 min; those refusal
rows read as attacks. It now sends X-Windy-Synthetic carrying a shared
secret (Gitea repo secret CANARY_SYNTHETIC_KEY = WINDYGIT_SYNTHETIC_KEY in
Veron .env); the API marks the row synthetic only on a constant-time
match, so an attacker cannot label their own refusals synthetic to hide.
synthetic is declared on forge.auth.failed (Telemetry Boss, UPDATE 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:54:56 -04:00
90643fe48e telemetry step 2: API boot/health + forge.auth.failed (declared)
All checks were successful
check / gate (push) Successful in 25s
canary / probe (push) Successful in 6s
Membrane first: I-2 and MEMBRANE.v1 now list the windy-admin ledger
(POST /v1/events). api/app/telemetry.py: service.boot once per start
(commit_sha omitted when unknown, I-12), an hourly in-process
service.health with the shared keys (requests, errors_5xx/4xx,
refusals_4xx, p95_ms only when there was traffic), and one
forge.auth.failed row per refused request: declared 13-code enum,
http_status, caller class, route TEMPLATE (never the concrete path),
actor_type system with no actor_id (all-lanes rule). No token = nothing
sent or buffered; flush failures keep rows (bounded) and never raise.
Token from root-only /etc/windygit/telemetry.env (optional env_file).
8 behavioural tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:47:29 -04:00
00ec963f82 telemetry: fix ci.run completeness — cursor on (finish time, job id)
Some checks failed
check / gate (push) Has been cancelled
Telemetry Boss found jobs_finished=43 vs 8 ci.run rows. Root cause: the
high-water mark was the job id, but jobs FINISH out of id order, so every
long job that started before the mark and finished after it was silently
never emitted. Now a (finish time, id) cursor; finish = stopped, or
updated for skipped jobs with no stop time. Heartbeat finished/failed/
cancelled counts are derived from exactly the rows emitted, so
sum(jobs_finished) == count(ci.run) by construction (dry run on real
data: 97 == 97, failed 2 == 2, cancelled 13 == 13). posted_to_github now
set from the bridge's own rules. duration_ms = Gitea whole seconds x 1000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:42:36 -04:00
b5e4eaf57a telemetry: ci.job_cancelled from the janitor; interval_s on heartbeat
All checks were successful
check / gate (push) Successful in 22s
canary / probe (push) Successful in 6s
The janitor now returns one JSON line per job it cancels (repo, workflow,
job, reason, runs_on, waited_s) into a spool; the emitter ships them as
ci.job_cancelled (declared with Telemetry Boss) and truncates the spool
only after a 2xx. Run status recompute folded into the same statement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:27:53 -04:00
baaa542bae docs: audit disposition 09-23 — R2 god token replaced by bucket-scoped token
All checks were successful
check / gate (push) Successful in 43s
canary / probe (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:15:35 -04:00
0634a6cb1b style: ruff fix in telemetry_emit
All checks were successful
check / gate (push) Successful in 29s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:08:21 -04:00
1a171eabd5 telemetry: CI emitter for admin.windyword.ai (inert until token)
Some checks failed
check / gate (push) Failing after 20s
canary / probe (push) Successful in 6s
ci.run (one row per finished job, exactly once via a high-water mark;
branch_kind default|pr|other so the dashboard can show "main is red") and
service.health (interval counts: finished/failed/cancelled, waiting,
running, runners online, oldest wait). Shapes declared with Windy
Telemetry 40; sends nothing until WINDYGIT_TELEMETRY_TOKEN exists.
State is only advanced after a 2xx, so a failed post retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:06:54 -04:00
28c31236b8 ci: janitor also clears jobs blocked forever on failed needs
When a needed job fails, Gitea leaves dependants BLOCKED (7) even after
the run finishes; eternitas build jobs sat there 8h. Mark them skipped
(what GitHub shows) once the run is done and 30 min have passed.
Found by the new telemetry dry run (oldest_waiting_s = 29160).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 11:06:33 -04:00
cd5967031b ci: janitor cancels jobs no runner can ever take
All checks were successful
check / gate (push) Successful in 20s
canary / probe (push) Successful in 6s
windy-pro alone left ~4 jobs per run waiting forever (build-electron on
macos/windows/ubuntu-latest, deploy if:false): Gitea evaluates job if:
only at pick time, the labels do not exist here, and waiting jobs are
invisible in /actions/tasks. 37 such jobs across 10 runs today. After
30 min they are cancelled and the run status recomputed. Runs each sync,
non-fatal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 10:58:53 -04:00
f246417095 ci: windy-pro desktop/installer jobs are NON-BLOCKING (Grant, 09-23)
build-desktop, test-installer and reality-check still run on Windy Git
and stay visible there, but the bridge no longer posts them to GitHub, so
they cannot turn windy-pro's combined status red. Windy Git side only;
the desktop code is Grant's to fix. BRIDGE_NON_BLOCKING, per repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 10:58:07 -04:00
50c1464043 security: never bundle credential repos to R2 in plaintext
All checks were successful
check / gate (push) Successful in 23s
canary / probe (push) Successful in 7s
kit-army-config (the lockbox) and every *-soul / anima repo carry
credentials; the nightly R2 bundles are unencrypted, so the R2 key was a
key to every secret. Excluded by name (BACKUP_EXCLUDE); they are backed up
encrypted by the Windy Drops lane (restic) and stay mirrored on Veron.
Behavioural test runs the script's own exclusion function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 10:48:42 -04:00
7a63f90da3 ci: bridge windy-mind (private) verdicts to GitHub
All checks were successful
check / gate (push) Successful in 23s
canary / probe (push) Successful in 6s
windy-mind has been writable + CI on Windy Git since 08-13 (deploy.yml
disabled, uv pinned); it only lacked GitHub commit statuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 09:59:28 -04:00
b8f97f0731 ops: sync never pushes archive/* branches to Windy Git
All checks were successful
check / gate (push) Successful in 20s
archive/<machine>-<date>/<branch> are off-machine safety copies of
unpushed work (one-repo doctrine). GitHub holds them; running CI on them
is waste. Negative refspec ^refs/heads/archive/* on the push (git 2.43
on Veron). Requested by 8c for windy-pro's Mac mini archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 09:57:57 -04:00
95c33c8004 ops: host systemd units in git; windy-pro tags never reach Windy Git
All checks were successful
check / gate (push) Successful in 22s
- deploy/systemd/: sync/backup timers+services, tunnel, and the windy-job
  heartbeat drop-ins (silent-failure audit). They existed only on Veron,
  the same drift that left the runbook wrong. GITHUB_TOKEN is stripped
  (repo is public); it stays in the root-only unit on the host.
- sync: SYNC_NO_TAGS (default windy-pro). build-electron fires on v* tags
  and targets ubuntu/macos/windows-latest, labels no runner has, so it
  would queue forever, invisibly. Desktop releases are built elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 09:51:51 -04:00
d5181f1c6d ci: G11.5 resolved — onboard windy-pro; cloud cells + windytalk; promote script
All checks were successful
check / gate (push) Successful in 21s
canary / probe (push) Successful in 6s
- import_from_github.py no longer refuses windy-pro: lane 8c audited all
  14 checkouts (WINDYPRO_CHECKOUTS.md), GitHub main is canonical.
- sync + bridge: windy-cloud-domains, windy-cloud-vps, windytalk (default
  branch master), windy-pro; windy-cloud-sites added to the bridge (it was
  synced but never bridged).
- scripts/promote_to_ci.sh: the mirror->writable procedure as one script,
  with the delete-before-import hazard documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 09:49:15 -04:00
e6530d3171 test: behavioral G3.5 webhook tests (audit: tests were source-string asserts)
All checks were successful
check / gate (push) Successful in 20s
canary / probe (push) Successful in 9s
Seven HTTP-level tests through the real route: sha256= prefix and bare
digests accepted, digest of re-serialised JSON refused, forged/wrong-key/
missing signatures refused, unset secret -> 503, a revocation with a bad
signature never reaches the handler, the reachability ping never acts.
Mutation-checked: dropping the prefix strip fails the behavioral test
while the old string-grep invariant still passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:26:14 -04:00
419443573a ci: onboard windy-hand; stop running windy-agent CI twice
All checks were successful
check / gate (push) Successful in 23s
windy-hand promoted to writable + bridged. windy-agent is PUBLIC and its
GitHub Actions already run on Veron's GitHub runner; running its 3-version
pytest matrix here too was pure duplicate load (3 x ~4.5 cores for 25+
min, host load 64 on 24 cores). Actions are now off for windy-agent on
Windy Git; it stays in REPOS as a synced copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:23:25 -04:00
4a34b35441 security: CI egress filter — jobs reach the internet, never Veron/LAN
Measured: an unprivileged job container inside the CI dind could open SSH,
Ollama and dev servers on Veron (192.168.1.73) and the rest of the LAN,
WireGuard and Tailscale — lateral movement for any malicious dependency,
no escape needed. egress.sh (idempotent; windygit-ci-egress.service at
boot) hooks the jobs bridge: runner<->dind, replies, DNS and public
egress allowed; RFC1918, CGNAT, link-local and the host itself dropped.
Verified from a job container: 6/6 private targets blocked, DNS,
internet and the public forge OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:20:45 -04:00
dfe5543eda docs: runbook + AGENTS match reality (item 5 of the launch bar)
Some checks failed
check / gate (push) Has been cancelled
canary / probe (push) Successful in 34s
RUNBOOK-VERON: deploy uses fetch + merge --ff-only and api-only rebuilds
(the old text used git pull, contradicting its own warning); new sections
for host timers, CI (6 runners x1, windyadmin-scoped, 90m ceiling, queue
truth in the gitea DB, logs in R2), sign-in posture and break-glass;
standing checkout = OC5. AGENTS.md no longer says GENESIS / no code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:19:12 -04:00
40cb455d0d ci: onboard windy-translate, windytranslate-site, windytraveler-site
Some checks failed
check / gate (push) Has been cancelled
Promoted to writable; the two sites' CF deploy.yml disabled (they would
need the CF god token in a job container). All three only have
ubuntu-latest workflows today, so nothing runs until their lanes switch
runs-on to [self-hosted, linux, x64].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:13:40 -04:00
8c404eb410 security: turn off Gitea OAuth auto-registration
Any Windy Word account (public signup, unverified email) auto-registered a
forge account on first sign-in — reproduced with a throwaway account —
and the act runners are instance-wide, so a stranger's workflow would run
on Veron's privileged dind beside the R2 god token. §7 makes opening the
forge to non-Grant users Grant's call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 03:09:43 -04:00
5b16114b98 auth: token contract v1 (aud windy_git, both issuers); CI for eternitas
Some checks failed
check / gate (push) Successful in 37s
canary / probe (push) Has been cancelled
- hub_jwt: aud list is ["windy_git"] (contract v1 array). Dropped
  "windy-git": that is Gitea's OIDC client_id, so a forge id_token would
  have passed the aud check. `type: human` is now REQUIRED (id_tokens have
  none), which makes accepting the discovery-URL issuer safe.
- runner job ceiling 30m -> 90m: eternitas's serial pytest is ~50 min and
  would have been killed mid-suite.
- eternitas (private) added to the GitHub status bridge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:58:56 -04:00
18ea9a4686 auth: G3.2 hub JWKS verifier — humans can sign in to the plane (SSO #14)
The human path refused every token in production (503
human_signin_not_ready) because no verifier existed. api/app/hub_jwt.py
verifies hub access tokens against account.windyword.ai's JWKS:

- RS256 only (closes alg:none and HS256-with-public-key confusion)
- iss must be "windy-identity" — what hub ACCESS tokens carry (observed
  live); id_tokens (discovery-URL issuer) are not accepted as bearers
- aud optional today, must name Windy Git when present; hub_require_aud
  flips it mandatory once the hub emits it. PyJWT's own aud check is off
  on purpose: it rejects ANY aud-bearing token when no audience is given.
- type must be human; identity = windy_identity_id, never sub (per-row id)
- production verifies even if require_verified_jwt is off

11 behavioral tests sign real RS256 tokens with a local key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:55:17 -04:00
32e8ac8474 ci: bridge windy-registry (private) PR/main verdicts to GitHub
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:53:20 -04:00
8e9fa3116c ci: six runners; SSO #8 Gitea sign-in hardening (staged)
- runner-5/6: 50+ jobs were queued with ~11 private repos onboarded. dind
  keeps the 12-core ceiling, so this adds concurrency, not CPU.
- Gitea: password + passkey sign-in forms off (break-glass = CLI), and
  ACCOUNT_LINKING auto -> login. auto linked any hub login whose email
  matched an existing account, and SITE ADMIN windyadmin carries Grant's
  email. Grant is linked by the hub's stable sub, which matches first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:51:56 -04:00
74ad4950b2 ci: onboard windy-drops, windy-code-web, windy-code, windy-traveler
All checks were successful
check / gate (push) Successful in 1m1s
canary / probe (push) Successful in 38s
All four promoted from pull mirrors to writable. windy-code keeps only
canonical-domains-lint active: its other 15 workflows target hosted
macOS/Windows/ubuntu-latest runners and would queue forever here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:49:00 -04:00
e7bbf9af51 ci: prune.sh must address dind over TCP (it has no unix socket)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:47:35 -04:00
45686283be ci: bound CI storage; don't bridge image-build jobs
- deploy/runner/prune.sh + windygit-ci-prune.timer (6h): age-based prune
  of the CI-only dind (containers, finished-job volumes, images/builder
  cache >7d) plus a hard 60 GB cap. Only that daemon, over its own TCP
  socket; never the host's Docker. It was 38 GB and unbounded — the same
  class of growth that filled Kit 0 on 09-01.
- pr_status_bridge: jobs named *docker* are not posted. Job containers
  have no daemon by design (I-5), so they are red on every commit; a
  permanent red X teaches everyone to ignore red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:47:28 -04:00
e4a15869c0 ci: onboard windy-connect + windy-search to private-repo CI
Some checks failed
check / gate (push) Has been cancelled
windy-connect promoted from pull mirror to writable (release.yml, which
publishes to PyPI on tag push, disabled — the sync pushes tags).
windy-search was already writable; its scheduled drift-check is disabled
because it now runs as cron on Kit 0. Both added to BRIDGE_REPOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:44:19 -04:00
dcf9286f16 ci: make Windy Git CI permanent for the private repos
All checks were successful
check / gate (push) Successful in 21s
canary / probe (push) Successful in 6s
- Four runners x capacity 1 instead of one x capacity 4. Concurrent jobs in
  one act_runner share /root/.cache/act; a refresh racing a copy killed 3 of
  windy-chat's ~20 jobs at setup-node (lstat ... no such file). Separate
  processes have separate caches. Same parallelism, same capped dind.
- Behavioral tests for pr_status_bridge (latest verdict wins, no reposting,
  skipped never painted green, fork PRs never run, pagination, PR lifecycle).
- import_from_github.py reads IMPORT_GITEA_URL, not GITEA_BASE_URL: sourcing
  the deploy .env pointed it at http://gitea:3000 and it died on DNS after the
  mirror it replaces had already been deleted.
- CUTOVER.md: the private-repo CI path, onboarding steps, and the
  /actions/tasks-hides-queued-runs trap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:18:04 -04:00
1b09b9b0d3 ci: sync windy-chat + windy-mail, bridge PR CI verdicts back to GitHub
All checks were successful
check / gate (push) Successful in 20s
canary / probe (push) Successful in 6s
GitHub Actions can't run on the private platform repos. Windy Git already
has their code and a working runner, so:

- windy-chat and windy-mail were read-only pull mirrors (which can never
  run Actions); they are now writable, deploy.yml/build-image.yml disabled,
  and synced from GitHub like the others.
- scripts/pr_status_bridge.py mirrors open same-repo GitHub PRs into Windy
  Git (so pull_request workflows fire) and posts each job's result back as
  a GitHub commit status (windy-git/<workflow>/<job>) on PR heads and the
  default-branch head. Fork PRs are never run. Runs after every sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 02:13:21 -04:00
390c1e7479 ops: move tunnel metrics to 2001, sync windy-git into itself
Some checks failed
check / gate (push) Successful in 19s
canary / probe (push) Failing after 7s
windygit-tunnel had crash-looped ~91k times: another project's
cornercall-tunnel holds 127.0.0.1:2000, and cloudflared exits when it
cannot bind its metrics port. Ingress only survived because a stray
cloudflared.service ran the same config. That unit is now disabled and
/etc/cloudflared/config.yml uses metrics 127.0.0.1:2001.

Also add windy-git to the GitHub->Windy Git sync list; its self-hosted
copy was stuck 3 commits behind (only check + canary workflows, no
deploys, so syncing is safe).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 01:29:57 -04:00
2b30b0ac99 docs: record the runner bump, the act cache flake, and the remaining reds
Second root cause found and fixed: act_runner 0.2.11 predates
`runs.using: node24`, so any repo on actions/checkout@v5 died before its first
step. Bumped to 0.6.1; Windy-Clone went 4/4 red to 4/4 green.

Also upgrades the act-cache note from "watch item" to a confirmed job failure
(lstat on a vanished file mid-tar), with the wipe command and the annotated-tag
dead end that looks like a wrong checkout but isn't.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 19:09:26 -04:00
cd7dd9b7ae ci: bump act_runner 0.2.11 -> 0.6.1 for node24 action support
0.2.11's bundled act only knows runs.using node12/node16/node20, so any repo
pinning a current action major (actions/checkout@v5, actions/setup-python@v6)
fails before its first step with "The runs.using key in action.yml must be one
of: [...], got node24". Windy-Clone is how this surfaced.

Verified node24 is absent from the 0.2.11 binary and present in 0.6.1, and that
every key in deploy/runner/config.yaml still exists in 0.6.1's schema.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 19:00:07 -04:00
9fc27eabd6 docs: record the real CI root cause — setup-uv resolves uv via the forge API
The localhost->postgres fix was correct but was never what failed these jobs;
they died at step 2. setup-uv v4+ resolves "latest" through GITHUB_API_URL,
which act_runner points at our own forge, so it 404s and every later step is
skipped by success(). Pinned an explicit uv version across 11 repos.

Also records the diagnosis traps that cost the most time: Gitea's job status
enum (1=success, 2=failure), act misattributing the error to the previous step,
the jobs-log API needing a repo-matched id, and Secure cookies defeating a
loopback curl login.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 17:57:04 -04:00
Grant Whitmer
db055a1922 docs: refresh the turnover prompt for the actual next task
Some checks failed
check / gate (push) Successful in 18s
canary / probe (push) Failing after 6s
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 17:31:23 -04:00
Grant Whitmer
86326ca6a7 docs: record three-repo CI fix results — 1 of 3 verified
All checks were successful
check / gate (push) Successful in 19s
windy-registry's postgres integration went failure -> success, proving the fix.
windy-mind and WindyCloud still fail for a cause I could not determine: the
jobs API returns 'job not found' for the ids the runs report, so logs were not
retrievable that way. Next session should read them from the Gitea web UI.

Records the trap that the three repos did NOT share one pattern — a naive
localhost->postgres swap would have left WindyCloud on port 15432.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 17:30:44 -04:00
Grant Whitmer
e0d5d118be docs: turnover for a fresh session
All checks were successful
check / gate (push) Successful in 35s
canary / probe (push) Successful in 8s
State, the immediate task (three-repo localhost->service-name CI fix), the traps
already paid for, and a copy-paste prompt.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 15:00:28 -04:00
Grant Whitmer
01e36155a3 ci: remove the service-networking probe; scope the python-pin test
Some checks failed
check / gate (push) Successful in 20s
canary / probe (push) Failing after 1m33s
The probe's own log was never retrievable through the jobs API, but the
question it asked was answered better by a direct comparison of two real
workflows on the same runner and image:

  windy-git gate       @postgres:5432   -> passes its migration round-trip
  eternitas migrations @localhost:5432  -> failed

Also scopes test_g73 to workflows that actually run Python. It failed the probe
for not pinning a version when the probe only shelled out to psql — the test
being wrong rather than the workflow.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 14:41:14 -04:00
Grant Whitmer
fe8f84bbdf ci: probe how service containers are addressed on this runner
Some checks failed
check / gate (push) Failing after 17s
canary / probe (push) Successful in 16s
Several migrated workflows hardcode postgres at localhost:5432, which is
correct on GitHub-hosted runners (services are port-mapped to the VM) and
suspect on Gitea Actions (the job runs IN a container, so localhost is the job).
Prove which form works before rewriting anyone's workflow.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 13:14:24 -04:00
Grant Whitmer
eae1bff50b G2.3: Windy Git branding — and get it out of one host's disk into the repo
All checks were successful
check / gate (push) Successful in 19s
canary / probe (push) Successful in 8s
The front end was 100% stock Gitea: green teacup, "Gitea: Git with a cup of
tea", "A painless, self-hosted Git service". G2.3 was specified in the plan with
an acceptance test and never executed, and nothing enforced it.

Now: Windy Git name, wind-mark logo, brand-blue accent, and a landing page that
says what this actually is. Uses Gitea's SUPPORTED surface (custom templates +
public assets) so upstream upgrades keep arriving — no source modified (D-2/I-1).

Two traps this cost, both now documented and tested:

1. GITEA__DEFAULT__APP_NAME does not work. Gitea reads APP_NAME from the TOP
   LEVEL of app.ini; the env var created a literal [default] section that Gitea
   ignores, so the installer's stock APP_NAME kept winning while the config
   looked correct. The env-to-ini pass also APPENDED a second APP_NAME rather
   than replacing the first — a new variant of the documented G4A.3 trap.

2. Cloudflare caches /assets/* for 6h and no token in this stack can purge, so
   the new logo and CSS were invisible while being correct at origin. Brand
   assets now carry a VERSION IN THE FILENAME; bump it on every change.

Committed with an idempotent apply.sh, because applying it straight to Veron's
disk first was itself the config-drift trap this project documents: a rebuild
would have silently reverted to stock Gitea.

85 tests green.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-14 09:08:48 -04:00
Grant Whitmer
e7dee39151 throttle: stop claiming to limit pushes we cannot see
All checks were successful
check / gate (push) Successful in 19s
canary / probe (push) Successful in 9s
I reintroduced the exact defect I had just criticised. ACTION_BASE listed
"push" and "push.force", but git push goes straight to Gitea over HTTPS and
never touches this API — so nothing records a push, a count would be zero
forever, and enforce() would look up a limit, count nothing, and allow
everything. A silent no-op wearing the costume of a control, made worse by a
config name that implies the protection exists.

Split into ACTION_BASE (actually enforced: repo.create, grant.create) and
NOT_ENFORCED_HERE (push, push.force) with the reason and the remedy written
down: enforcing push velocity needs a Gitea-side pre-receive or push webhook
reporting into agent_actions.

enforce("push") now raises rather than silently allowing, and a test asserts the
two sets stay disjoint.

Found by auditing whether the auth fix could be walked around — every
/api/v1/repos/* route does require a caller, and the only unauthenticated
endpoints are /health, /version and the HMAC-verified webhook.

83 tests green.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:33:30 -04:00
Grant Whitmer
25368547cb docs: runbook — never pull -q, never force-push a deployed branch
All checks were successful
check / gate (push) Successful in 21s
canary / probe (push) Successful in 11s
Two deploy traps paid for on 2026-08-14: 'git pull -q' hid a divergent-branch
error so a deploy ran against stale code while reporting success, and the
divergence came from amending a commit a deploy checkout already tracked.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:28:57 -04:00
Grant Whitmer
c60bfb2b89 I-12: fail the build when COMMIT_SHA is empty
All checks were successful
check / gate (push) Successful in 19s
/version went null after a deploy — the exact "service cannot name its own
commit" defect this project was built to prevent, caught by its own honesty
check.

Cause: the sed replaced "" with "" (a no-op when COMMIT_SHA is empty) and the
grep then matched that same empty string, so the guard verified nothing. A build
with no COMMIT_SHA passed and shipped a container reporting commit_sha: null.

Now the build fails loudly instead.

Second cause of the stale deploy, and it was mine: an earlier `git commit
--amend` + force-push rewrote history the Veron deploy checkout was already
sitting on, leaving it divergent so `git pull -q` failed SILENTLY (-q hid
"Need to specify how to reconcile divergent branches"). Two lessons: do not
force-push a branch a deploy checkout tracks, and do not pull with -q in a
deploy script.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:27:38 -04:00
Grant Whitmer
a0ed4a5ec0 SECURITY: make EPT routing independent of signature well-formedness
All checks were successful
check / gate (push) Successful in 20s
looks_like_ept used jwt.get_unverified_header, which validates the WHOLE token
and therefore rejects anything with a malformed signature segment. Routing
consequently depended on signature well-formedness: an EPT-shaped token with a
bad signature fell through to the HUMAN path, where it was refused for the wrong
reason and — with require_verified_jwt off (dev) — could have been read as a
human identity via its `sub` claim.

Now the header segment is decoded directly, so routing depends only on what the
token CLAIMS to be; whether it is authentic remains verify_ept's job.

Also routes alg:none to the EPT verifier regardless of typ, since a `none`
token is never valid for any caller. Both forged shapes now return 401
ept_invalid — the honest code — instead of 503 "feature not ready".

Found by noticing a forged EPT returned 503 where the verifier should have
answered 401, rather than accepting "it was refused, close enough".

80 tests green.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:23:50 -04:00
Grant Whitmer
830ca48705 docs: mark revocation finding resolved — it was critical, not low
All checks were successful
check / gate (push) Successful in 19s
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:20:46 -04:00
Grant Whitmer
deb3a8fafc test: prove the revocation wiring end-to-end (resolve_passport -> decide_trust)
Some checks failed
check / gate (push) Has been cancelled
Monkeypatched httpx so resolve_passport sees a real revoked trust body
(status=revoked, band=unproven, allowed=[]) and must raise
PassportNotInGoodStanding — proving the WIRING, not just the decision. This is
the path that stops a validly-signed EPT that outlived its passport's
revocation (~365-day tokens).

Live-confirmed alongside: Eternitas refuses to mint EPTs for revoked bots
("credentials are not issued for non-active bots"), so the only exposure was a
pre-existing token — exactly what this now catches. The active agent's EPT still
returns 200. A fully-live revoked test would require revoking a real fleet
passport (destructive), so the wiring is proven deterministically instead.

79 tests green.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:20:44 -04:00
Grant Whitmer
587fb05265 SECURITY: enforce passport status — revocation now takes effect live
All checks were successful
check / gate (push) Successful in 21s
A revoked passport returns HTTP 200, status=revoked, band=unproven,
allowed_actions=[] (verified live 2026-08-13). resolve_passport keyed refusal
only on HTTP 4xx and band=="untrusted", so it returned band 'unproven' and the
agent was seated. Revocation was NOT enforced on the live auth path at all — and
now that agent auth actually works, a revoked agent could authenticate and act.

Extracts decide_trust(body) -> (band, actions) | raise. Only status=="active"
is allowed; revoked/suspended/frozen/unknown all refuse, fail-closed on the
field that carries the most consequential fact about an identity. The agent call
site turns that into a clean 403 passport_revoked.

This is the REAL revocation gate — the token cannot be un-issued, but its
standing is re-checked on every request, so revocation takes effect on the next
call with no webhook required. The Eternitas webhook remains useful for
invalidating locally-issued credentials/grants (G6.3, not built yet), but it was
never the primary gate and its being unwired is no longer a live exposure.

Behavioral tests: revoked body refused, active accepted, unknown/missing status
fails closed.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:18:16 -04:00
Grant Whitmer
c96d1d3102 canary: guard both forgery shapes now that real verification is live
All checks were successful
check / gate (push) Successful in 20s
The EPT-shaped forgery is the one that matters after G3.2 — it is what
signature verification actually guards. The JWT-shaped one still exercises the
human gate. Both must_refuse; a 2xx on either pages.

Verified live: forged EPT -> 401 ept_invalid, forged JWT -> 503,
genuine EPT -> 200.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:13:44 -04:00
Grant Whitmer
79dcea4d3e G3.2/G3.4: real EPT verification + wire the throttle, reopening agent auth
All checks were successful
check / gate (push) Successful in 19s
canary / probe (push) Successful in 10s
REOPENS the agent path — but only because possession is now actually proven.

EPT verification (api/app/ept.py): ES256 against Eternitas's published key set
at /.well-known/eternitas-keys. algorithms=["ES256"] makes alg:none and
algorithm confusion unrepresentable rather than merely unlikely; issuer and exp
are enforced by the library; an unknown kid is refused.

Order is deliberate: signature FIRST, trust lookup second. These EPTs live ~365
days and carry rev/tru baked in at issuance, so a year-old "rev: false" proves
nothing — revocation and band still come from a live lookup on every request.

Found while building it: real EPTs put the passport in the "sub" claim. The old
code read "passport"/"sub_passport", which no genuine EPT carries — so real
agents were never recognised and ONLY forged tokens ever authenticated. The
bypass was not just a hole, it was the only thing that worked.

Throttle (api/app/throttle.py): BAND_MULTIPLIER and rate_*_per_day were defined
and read by nothing. Now enforced on repo.create and grant.create, counted
against agent_actions (one source of truth, not a private counter that drifts
from the audit log). Fails CLOSED — a limiter that fails open protects you until
the moment something is wrong. Untrusted band is 403 read-only, not 429, because
"slow down" would be a lie.

Tests: 14 behavioral, signing real ES256 tokens with a locally-generated key so
they exercise the crypto path with no network dependency — genuine tokens
accepted, and alg:none / foreign key / tampered payload / expired / wrong issuer
/ unknown kid / missing claims all refused. 74 green.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 23:09:47 -04:00
Grant Whitmer
c257cc55c6 docs: second-auditor review (Fable) — 1 critical fixed live, 4 open
All checks were successful
check / gate (push) Successful in 20s
canary / probe (push) Successful in 11s
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 22:54:15 -04:00
Grant Whitmer
85fa65a52b canary: continuously verify the forged-token bypass stays closed
All checks were successful
check / gate (push) Successful in 18s
Adds must_refuse checks: a 2xx is the alarm, a 401/403/503 is health. The
forged alg:none agent token is probed every 10 min; if it ever returns 2xx the
canary goes red and pages. Proven both ways — 503 reads ok, a 200 endpoint
alarms 'ACCEPTED — this MUST be refused'. The security property is now enforced
by a running check, not assumed.

Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
2026-08-13 22:52:56 -04:00
66 changed files with 4953 additions and 118 deletions

View File

@@ -2,11 +2,17 @@
Read this before touching anything. Then read `DNA_STRAND_MASTER_PLAN.md`, which is the source of truth.
## Current state
## Current state (2026-09-23)
**GENESIS.** No code. No `make dev` yet — building it is codon **G0.8**.
**LIVE on Veron 1** — `app.windygit.com` (Gitea 1.24.6, Windy SSO only),
`api.windygit.com` (our plane: humans via hub JWKS, agents via Eternitas EPT),
`models.windygit.com`. Strands G0–G5, G7, G11 done; see the plan for the rest.
The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Cloudflare Tunnel), then **G2** (Gitea, stock and branded), then **G3** (identity), then **G4** (storage). G0–G4 are sequential. G5–G12 are concurrent once G4 lands.
It is also **the permanent CI for the private platform repos** (GitHub Actions
cannot run on them): `scripts/sync_from_github.sh` + `scripts/pr_status_bridge.py`,
onboarding in `docs/CUTOVER.md`, operations in `docs/RUNBOOK-VERON.md`.
Standing dev checkout: **OC5 `~/windy-git`**. Deploy copy: Veron `/srv/windygit/src`.
## The rules that will get you reverted if you break them
@@ -28,7 +34,7 @@ The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Clo
- Errors are 4-field repair pointers: `{code, speak, machine_cause, remediation_tool}`. No exceptions, including validation errors.
- Every tool response carries `state_proof` + `next_actions`.
- Telemetry `actor_type` comes from the enum `{human, agent, system}`. **`'service'` is not legal** — it 422s and silently drops the whole batch. A sibling service is losing telemetry to exactly this today.
- Runner labels are explicit and pinned. **`ubuntu-latest` is banned** — all four `windy-registry` workflows use it and every run fails.
- Runner labels are explicit and pinned: `[self-hosted, linux, x64]` or `veron-1`. **`ubuntu-latest` is banned** — no runner here has it, so the job queues forever.
## Membrane

View File

@@ -81,7 +81,7 @@ Numbered because code cites them. Changing one requires an ADR that names it.
1. **I-1 · Gitea is a component, never a merged tree.** Our code lives in our services and calls Gitea's REST API. Any patch to Gitea source lives in `patches/` as a numbered, rebasable diff with a one-line justification, and `make check` fails if `patches/` grows past **3** files without an ADR.
2. **I-2 · The membrane is ENUMERATED.**
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo).
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo). · windy-admin ledger `POST https://admin.windyword.ai/v1/events` (field telemetry, 2026-09-23: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed` — shapes declared with the ledger owner first; no content, no passports, no emails)
**Calls in:** `POST /internal/repo-from-folder` (Cloud portal: git-enable a folder) · `POST /internal/mirror-status` (ops).
**Events out:** `repo.created`, `repo.pushed`, `release.published`, `model.published`, `ci.completed`.
**Events in:** `passport.revoked` (fail-closed), `storage.quota.exceeded`, `identity.created`.

View File

@@ -18,7 +18,14 @@ COPY alembic ./alembic
COPY alembic.ini ./
COPY scripts ./scripts
RUN sed -i "s|^BAKED_COMMIT_SHA: str = \"\"|BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"|" api/app/buildinfo.py \
# I-12: an EMPTY COMMIT_SHA must fail the build, not sail through it.
# Previously the sed replaced "" with "" (a no-op) and the grep then matched
# that same empty string, so a build with no COMMIT_SHA passed and shipped a
# container reporting commit_sha: null — exactly the "service cannot name its
# own commit" defect this project exists to prevent. Caught 2026-08-14 when
# /version went null after a deploy.
RUN test -n "${COMMIT_SHA}" || (echo "FATAL: COMMIT_SHA build arg is empty (I-12)" && false) \
&& sed -i "s|^BAKED_COMMIT_SHA: str = \"\"|BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"|" api/app/buildinfo.py \
&& sed -i "s|^BAKED_BUILT_AT: str = \"\"|BAKED_BUILT_AT: str = \"${BUILT_AT}\"|" api/app/buildinfo.py \
&& grep -q "BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"" api/app/buildinfo.py

View File

@@ -22,7 +22,7 @@ boot guard in `api/app/main.py` that refuses to start there in production.
| 8600 | `windy-git-api` — our plane |
| **3080** | Gitea — host 3000 and 3300 are taken by resident projects on Veron 1 |
| 5432 | Postgres |
| 2000 | cloudflared metrics (probe target) |
| 2001 | cloudflared metrics — NOT 2000: `cornercall-tunnel` (another project) takes 2000, and a metrics bind failure kills the whole tunnel |
## Ingress — Cloudflare Tunnel `windy-git`

View File

@@ -25,7 +25,10 @@ import httpx
from fastapi import Header, Request
from api.app.config import Settings
from api.app.ept import EptInvalid, looks_like_ept, verify_ept
from api.app.errors import RepairPointer, passport_unresolvable
from api.app.hub_jwt import HubTokenInvalid, verify_hub_token
from api.app.telemetry import synthetic_headers
log = logging.getLogger(__name__)
@@ -72,6 +75,36 @@ BAND_MULTIPLIER: dict[str, float] = {
}
class PassportNotInGoodStanding(Exception):
"""The passport resolved, but Eternitas does not list it as active
(revoked / suspended / frozen / unknown status)."""
def __init__(self, passport: str, status: str) -> None:
self.passport = passport
self.status = status
super().__init__(f"{passport} status={status!r}")
def decide_trust(body: dict, passport: str) -> tuple[str, tuple[str, ...]]:
"""The trust body -> (band, allowed_actions), or refuse.
THE GATE THAT WAS MISSING. A revoked passport returns HTTP 200 with
`status: revoked`, `band: unproven`, `allowed_actions: []` — verified live
2026-08-13. The previous code keyed refusal only on HTTP 4xx and on
band=="untrusted", so a revoked agent (200, band unproven) authenticated and
acted normally. Revocation was not enforced on the live path at all; the
webhook that was supposed to be the backup was never the primary gate.
Only `status == "active"` is allowed. Anything else — including a status
Eternitas invents tomorrow — refuses. Fail-closed on the field that carries
the most consequential fact about an identity.
"""
status = str(body.get("status", "")).lower()
if status != "active":
raise PassportNotInGoodStanding(passport, status or "missing")
return body.get("band", "unproven"), tuple(body.get("allowed_actions", ()))
async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tuple[str, ...]]:
"""G3.6 — THE STATUS-CODE LAW.
@@ -93,7 +126,7 @@ async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tupl
)
url = f"{settings.eternitas_base_url}/api/v1/trust/{passport}"
headers = {"X-API-Key": settings.eternitas_platform_api_key}
headers = {"X-API-Key": settings.eternitas_platform_api_key, **synthetic_headers()}
last_status = 0
for attempt in range(3):
async with httpx.AsyncClient(timeout=httpx.Timeout(8.0, connect=3.0)) as client:
@@ -105,8 +138,9 @@ async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tupl
continue
last_status = r.status_code
if r.status_code == 200:
body = r.json()
return body.get("band", "unproven"), tuple(body.get("allowed_actions", []))
# decide_trust raises PassportNotInGoodStanding on a non-active
# status; that propagates past the retry loop as a hard refusal.
return decide_trust(r.json(), passport)
if r.status_code in (400, 404):
# Malformed or not-issued. Refuse immediately — retrying cannot help
# and pretending it might is how a soft-allow gets written.
@@ -159,69 +193,77 @@ async def get_caller(
token = authorization.split(" ", 1)[1].strip()
# --- agent (Eternitas EPT) --------------------------------------------
passport = _unverified_claim(token, "passport") or _unverified_claim(token, "sub_passport")
if passport:
# ⚠️ SECURITY — trust is not authentication.
#
# A trust lookup answers "is this passport reputable?". It does NOT
# answer "does this caller actually hold this passport?". Skipping the
# second question is an authentication bypass: anyone who knows a
# passport number (they appear in logs, the lockbox and revocation
# messages) could present an UNSIGNED token naming it and be treated as
# that agent. Verified live 2026-08-13 — a forged `alg:none` token
# returned HTTP 200.
#
# ES256/JWKS verification of the EPT against Eternitas is not built yet
# (the G3.2/G9.1 verifier). Until it is, the agent path FAILS CLOSED in
# production — exactly as the human path below already does. This is not
# a downgrade of the "agents are citizens" design; it is refusing to
# seat a citizen whose ID we cannot yet check. It reopens automatically
# the moment `verify_ept_signature` exists and this gate consults it.
if settings.is_production and settings.require_verified_jwt:
# Possession FIRST, reputation second. The signature proves the caller holds
# this passport; the trust lookup then says what it may do. Doing only the
# second was the 2026-08-13 impersonation bypass.
if looks_like_ept(token):
try:
verified = verify_ept(token, settings.eternitas_base_url)
except EptInvalid as exc:
raise RepairPointer(
status_code=503,
code="agent_signin_not_ready",
speak="Helper sign-in isn't switched on yet. Nothing you have is affected.",
machine_cause=(
"EPT signature verification (G3.2/G9.1) is not implemented; "
"refusing an unverified agent token in production. A trust "
"lookup proves reputation, not possession."
),
remediation_tool=None,
)
status_code=401,
code="ept_invalid",
speak="We couldn't confirm that helper's ID, so we didn't let it in.",
machine_cause=f"EPT verification failed: {exc}",
remediation_tool="windy_git.reissue_agent_token",
) from exc
band, actions = await resolve_passport(settings, passport)
# The token is authentic. It is NOT evidence of current standing: these
# EPTs live ~365 days and carry `rev`/`tru` baked in at issuance, so a
# year-old `rev: false` proves nothing. Revocation and band come from a
# live lookup, every time.
try:
band, actions = await resolve_passport(settings, verified.passport)
except PassportNotInGoodStanding as exc:
# The signature is authentic, but the identity is no longer good.
# Revocation takes effect here, live, on the next request — no
# webhook required. That is the honest place for it: the token can't
# be un-issued, but its standing is checked every time.
raise RepairPointer(
status_code=403,
code="passport_revoked",
speak="That helper's access has been turned off.",
machine_cause=f"eternitas status for {verified.passport} is {exc.status!r}, not active",
remediation_tool=None,
) from exc
if band.lower() == "untrusted":
raise RepairPointer(
status_code=403,
code="agent_read_only",
speak="That helper can look, but it isn't allowed to make changes yet.",
machine_cause=f"passport {passport} band=untrusted is read-only",
machine_cause=f"passport {verified.passport} band=untrusted is read-only",
remediation_tool=None,
)
return Caller(
actor_type=ActorType.agent,
passport=passport,
passport=verified.passport,
band=band,
allowed_actions=actions,
)
# --- human (account-server RS256) -------------------------------------
if settings.is_production and settings.require_verified_jwt:
# I-8, applied to ourselves. G3.2's JWKS verifier is not written yet, and
# an unverified JWT is an authentication bypass rather than a shortcut.
# Refusing is the only honest answer until the verifier exists.
raise RepairPointer(
status_code=503,
code="human_signin_not_ready",
speak="Signing in isn't switched on yet. Nothing you have is affected.",
machine_cause=(
"JWKS verification (G3.2) is not implemented; refusing to accept "
"an unverified human token in production"
),
remediation_tool=None,
)
# --- human (hub RS256 access token, G3.2) ------------------------------
# Production ALWAYS verifies, whatever require_verified_jwt says: the flag
# only exists to let local dev run against unsigned fixture tokens.
if settings.require_verified_jwt or settings.is_production:
try:
human = verify_hub_token(
token,
settings.account_server_base_url,
issuers=tuple(settings.hub_issuers),
audiences=tuple(settings.hub_audiences),
require_aud=settings.hub_require_aud,
)
except HubTokenInvalid as exc:
raise RepairPointer(
status_code=401,
code="token_invalid",
speak="We couldn't confirm that sign-in. Try signing in again.",
machine_cause=f"hub token verification failed: {exc}",
remediation_tool=None,
) from exc
return Caller(actor_type=ActorType.human, identity_id=human.identity_id)
# Local dev only (require_verified_jwt=False outside production).
identity_id = _unverified_claim(token, "windy_identity_id") or _unverified_claim(token, "sub")
if not identity_id:
raise RepairPointer(
@@ -241,10 +283,8 @@ def _unverified_claim(token: str, claim: str) -> str | None:
on the result re-establishes trust independently: an agent's authority comes
from a live Eternitas trust lookup, never from the token's own assertions.
⚠️ Full RS256/ES256 JWKS verification for the human path lands in G3.2's
verifier and MUST be in place before `api.windygit.com` accepts a human
token from outside. Until then the human path is reachable only from inside
the tunnel, and `settings.require_verified_jwt` refuses it in production.
Humans are verified by hub_jwt.verify_hub_token (G3.2); this reader backs
only the local-dev path, which production never takes.
"""
import base64
import json

View File

@@ -53,16 +53,33 @@ class Settings(BaseSettings):
# ---- account-server OIDC (human identity) -----------------------------
account_server_base_url: str = "https://account.windyword.ai"
# G3.2 — what a hub ACCESS token must say about itself (see hub_jwt.py).
# Token contract v1 (lane 8c, 2026-09-23): access tokens may carry either
# issuer. id_tokens are kept out by `type` + `windy_identity_id` + aud, not
# by issuer.
hub_issuers: list[str] = ["windy-identity", "https://account.windyword.ai"]
# Contract v1: aud is an ARRAY; first-party tokens list every product, and
# Windy Git's entry is `windy_git` (underscore). ⚠️ NEVER add "windy-git"
# (hyphen): that is Gitea's OIDC client_id, so an id_token minted for the
# forge would carry it and pass as a bearer here.
hub_audiences: list[str] = ["windy_git"]
# Flip to True once the hub emits aud on every access token.
hub_require_aud: bool = False
# ---- field telemetry (admin.windyword.ai ledger) ----------------------
# Unset token = nothing sent, nothing buffered. The token lives in the
# root-only /etc/windygit/telemetry.env on Veron, never in the repo.
windygit_telemetry_token: str = ""
telemetry_ingest_url: str = "https://admin.windyword.ai/v1/events"
# Internal callers (the Cloud portal calling /internal/*). A first-class
# caller class, not a bypass: unset means service calls are REFUSED.
service_token: str = ""
# ⚠️ FAIL-CLOSED GATE. Full RS256/ES256 JWKS verification lands in G3.2.
# Until it does, the human token path must not be reachable in production —
# accepting an unverified JWT is not a shortcut, it is an authentication
# bypass. Agents are unaffected: their authority comes from a live Eternitas
# trust lookup, not from anything the token asserts about itself.
# ⚠️ FAIL-CLOSED GATE. Human tokens are verified against the hub's JWKS
# (G3.2, hub_jwt.py). False only enables the unverified local-dev path, and
# production verifies regardless — an unverified JWT is a bypass, not a
# shortcut.
require_verified_jwt: bool = True
# ---- storage law (I-3, G4.4) ------------------------------------------

157
api/app/ept.py Normal file
View File

@@ -0,0 +1,157 @@
"""EPT signature verification (G3.2 / G9.1) — the gate that makes an agent an agent.
Trust is not authentication. A trust lookup answers *"is this passport
reputable?"*; only a signature answers *"does this caller actually hold it?"*.
Skipping the second question was a live impersonation bypass on 2026-08-13 — a
forged `alg:none` token naming a passport read out of the logs returned HTTP 200.
What this module refuses, deliberately and by construction:
* **`alg: none`** — the original exploit. `algorithms=["ES256"]` makes it
unrepresentable rather than merely unlikely.
* **Algorithm confusion.** Only ES256 is accepted. If an attacker presents an
HS256 token, PyJWT will not try to use an EC public key as an HMAC secret,
which is the classic way "verified" JWTs get forged.
* **An unknown `kid`.** The key must be one Eternitas currently publishes.
* **A wrong issuer or an expired token** — checked by the library, not by us.
What it deliberately does NOT decide: whether the agent is *allowed* to act.
The EPT carries `rev` and `tru` claims baked in at issuance, and these tokens
live for a year (observed `exp` ≈ 365 days). A year-old `rev: false` is not
evidence of anything. **Revocation and trust must come from a live lookup**, so
this module returns only identity and the caller re-checks standing.
"""
from __future__ import annotations
import base64
import json
import logging
import time
from dataclasses import dataclass
import httpx
import jwt
from jwt import PyJWKClient
log = logging.getLogger(__name__)
ISSUER = "eternitas.ai"
ALGORITHMS = ["ES256"] # exactly one. Never widen this list.
_jwks_client: PyJWKClient | None = None
_jwks_url: str | None = None
class EptInvalid(Exception):
"""The token is not a valid, currently-signed Eternitas EPT."""
@dataclass(frozen=True)
class VerifiedEpt:
passport: str
operator: str | None
bot_name: str | None
issued_at: int | None
expires_at: int | None
def _client(base_url: str) -> PyJWKClient:
"""One cached JWKS client. PyJWKClient caches keys and refetches on an
unknown kid, so a key rotation heals itself without a redeploy."""
global _jwks_client, _jwks_url
url = f"{base_url.rstrip('/')}/.well-known/eternitas-keys"
if _jwks_client is None or _jwks_url != url:
_jwks_client = PyJWKClient(url, cache_keys=True, lifespan=300)
_jwks_url = url
return _jwks_client
def verify_ept(token: str, eternitas_base_url: str) -> VerifiedEpt:
"""Verify an EPT's signature and claims. Raises EptInvalid on ANY doubt.
There is no partial success and no "probably fine" path: every failure mode
below produces the same refusal, because a caller that cannot prove
possession is indistinguishable from an attacker.
"""
try:
signing_key = _client(eternitas_base_url).get_signing_key_from_jwt(token)
except Exception as exc: # noqa: BLE001 - unknown kid, unreachable JWKS, malformed
raise EptInvalid(f"no usable signing key: {type(exc).__name__}: {exc}") from exc
try:
claims = jwt.decode(
token,
signing_key.key,
algorithms=ALGORITHMS, # ES256 only — closes alg:none and alg confusion
issuer=ISSUER,
options={
"require": ["sub", "iss", "exp"],
"verify_signature": True,
"verify_exp": True,
"verify_iss": True,
},
)
except jwt.PyJWTError as exc:
raise EptInvalid(f"{type(exc).__name__}: {exc}") from exc
# The REAL claim name. Eternitas puts the passport in `sub`; the previous
# code looked for `passport` / `sub_passport`, which no genuine EPT carries —
# so real agents were never recognised and only forged tokens ever "worked".
passport = claims.get("sub")
if not isinstance(passport, str) or not passport.strip():
raise EptInvalid("EPT carried no passport in `sub`")
return VerifiedEpt(
passport=passport,
operator=claims.get("ope"),
bot_name=claims.get("bot"),
issued_at=claims.get("iat"),
expires_at=claims.get("exp"),
)
def looks_like_ept(token: str) -> bool:
"""Cheap, unauthenticated triage: is this token even *claiming* to be an EPT?
Used ONLY to route a token to the right verifier. It decides nothing about
trust — an attacker controls every byte it reads.
Decodes the header segment directly rather than via
`jwt.get_unverified_header`, which validates the whole token structure and
therefore rejects anything with a malformed SIGNATURE. That made routing
depend on signature well-formedness: an EPT-shaped token with a bad
signature fell through to the human path, where it was refused for the wrong
reason ("signing in isn't switched on") and — with `require_verified_jwt`
off — could have been read as a human identity via its `sub` claim.
Routing must depend only on what the token claims to be. Whether it is
authentic is `verify_ept`'s job, and it says no.
"""
try:
head_b64 = token.split(".", 1)[0]
head_b64 += "=" * (-len(head_b64) % 4)
header = json.loads(base64.urlsafe_b64decode(head_b64))
except Exception: # noqa: BLE001
return False
if not isinstance(header, dict):
return False
return header.get("typ") == "EPT" or header.get("alg") in ("ES256", "none")
async def eternitas_reachable(base_url: str) -> bool:
"""Whether the key set can be fetched at all. Used by /health/full so an
unreachable JWKS is reported rather than discovered during an outage."""
try:
async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, connect=3.0)) as c:
r = await c.get(
f"{base_url.rstrip('/')}/.well-known/eternitas-keys",
headers={"User-Agent": "windy-git/1.0"},
)
return r.status_code == 200 and "keys" in r.json()
except Exception: # noqa: BLE001
return False
def seconds_until_expiry(ept: VerifiedEpt) -> int | None:
return None if ept.expires_at is None else int(ept.expires_at - time.time())

133
api/app/hub_jwt.py Normal file
View File

@@ -0,0 +1,133 @@
"""Human token verification (G3.2) — hub access tokens from account.windyword.ai.
Until this existed the human path refused every token in production (503
`human_signin_not_ready`), because reading an unverified JWT's claims is an
authentication bypass, not a shortcut. This module is what lets it say yes.
The token it accepts is the hub's ACCESS token, as observed live 2026-09-23:
header {alg: RS256, typ: JWT, kid: <published at /.well-known/jwks.json>}
claims iss = "windy-identity" (contract v1 also allows the discovery URL)
type = "human", exp - iat = 900 s
sub = per-row user id ← NOT the cross-product identity
windy_identity_id = the Windy Account UUID (what Gitea's OIDC links on)
no `aud` yet
What it refuses, by construction:
* **Anything but RS256.** One algorithm, never a list. Closes `alg: none` and
HS256-with-the-public-key confusion.
* **An unknown `kid`**, a wrong issuer, an expired token — library-checked.
* **An id_token used as a bearer.** id_tokens prove a login happened to a
relying party (for the forge: aud `windy-git`), not that this caller may act
here. They carry no `type` and no `windy_identity_id`, and their aud is a
client id, not the product name `windy_git` — any one of the three refuses.
* **A non-human `type`.** An agent's authority comes from its EPT and a live
Eternitas lookup, never from a hub token dressed as a person.
* **A token with no `windy_identity_id`.** `sub` is a different namespace (the
per-row user id); falling back to it would silently mint identities that
match nothing Gitea knows.
`aud` (token contract v1, lane 8c): an array; first-party tokens list every
product and Windy Git's is `windy_git`. Optional until the hub emits it; when
present it MUST include `windy_git`.
`hub_require_aud=True` makes it mandatory — flip it once the hub emits it.
"""
from __future__ import annotations
from dataclasses import dataclass
import jwt
from jwt import PyJWKClient
ALGORITHMS = ["RS256"] # exactly one. Never widen this list.
_jwks_client: PyJWKClient | None = None
_jwks_url: str | None = None
class HubTokenInvalid(Exception):
"""Not a valid, currently-signed hub access token for a human."""
@dataclass(frozen=True)
class VerifiedHuman:
identity_id: str
email: str | None
expires_at: int | None
def _client(base_url: str) -> PyJWKClient:
"""Cached JWKS client; refetches on an unknown kid so rotation self-heals."""
global _jwks_client, _jwks_url
url = f"{base_url.rstrip('/')}/.well-known/jwks.json"
if _jwks_client is None or _jwks_url != url:
_jwks_client = PyJWKClient(url, cache_keys=True, lifespan=300)
_jwks_url = url
return _jwks_client
def verify_hub_token(
token: str,
base_url: str,
*,
issuers: tuple[str, ...],
audiences: tuple[str, ...],
require_aud: bool,
signing_key=None,
) -> VerifiedHuman:
"""Verify a hub access token. Raises HubTokenInvalid on ANY doubt.
`signing_key` exists for tests only (a locally generated key, no network).
"""
try:
key = (
signing_key
if signing_key is not None
else _client(base_url).get_signing_key_from_jwt(token).key
)
except Exception as exc: # noqa: BLE001 - unknown kid, unreachable JWKS, malformed
raise HubTokenInvalid(f"no usable signing key: {type(exc).__name__}: {exc}") from exc
try:
claims = jwt.decode(
token,
key,
algorithms=ALGORITHMS,
issuer=list(issuers),
options={
"require": ["iss", "exp", "iat"],
"verify_signature": True,
"verify_exp": True,
"verify_iss": True,
# Checked by hand below: PyJWT rejects any token CARRYING aud
# when no audience is passed, which would break the moment the
# hub starts emitting it — the exact trap the SSO matrix names.
"verify_aud": False,
},
)
except jwt.PyJWTError as exc:
raise HubTokenInvalid(f"{type(exc).__name__}: {exc}") from exc
aud = claims.get("aud")
if aud is None:
if require_aud:
raise HubTokenInvalid("token carries no aud and hub_require_aud is on")
else:
presented = {aud} if isinstance(aud, str) else set(aud) if isinstance(aud, list) else set()
if not presented & set(audiences):
raise HubTokenInvalid(f"aud {sorted(presented)} does not name Windy Git")
# REQUIRED, not defaulted: id_tokens carry no `type`, and this is one of the
# two claims (with windy_identity_id) that keep them from acting as bearers.
if claims.get("type") != "human":
raise HubTokenInvalid(f"token type {claims.get('type')!r} is not a human access token")
identity = claims.get("windy_identity_id") or claims.get("windyIdentityId")
if not isinstance(identity, str) or not identity.strip():
raise HubTokenInvalid("token carries no windy_identity_id")
return VerifiedHuman(
identity_id=identity, email=claims.get("email"), expires_at=claims.get("exp")
)

View File

@@ -6,11 +6,13 @@ component and is reached only over its REST API (D-2 / I-1).
from __future__ import annotations
import asyncio
import logging
import socket
import time
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
@@ -25,6 +27,7 @@ from api.app.providers.registry import (
R2Provider,
)
from api.app.routes import health, repos, webhooks
from api.app.telemetry import SYNTHETIC, Telemetry, caller_class, is_synthetic
logging.basicConfig(
level=logging.INFO,
@@ -44,9 +47,7 @@ def _refuse_kit_zero(settings) -> None:
if not settings.is_production:
return
try:
local_ips = {
info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)
}
local_ips = {info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)}
except socket.gaierror:
return
if settings.kit0_host in local_ips:
@@ -98,8 +99,23 @@ async def lifespan(app: FastAPI):
# systemd Restart=always, plus the runbook's `systemctl status`.
]
telemetry = Telemetry(
settings.telemetry_ingest_url,
settings.windygit_telemetry_token,
environment=settings.environment,
commit_sha=info.commit_sha,
version=info.version,
)
app.state.telemetry = telemetry
telemetry.boot()
await telemetry.flush()
task = asyncio.create_task(telemetry.run()) if telemetry.enabled else None
yield
if task is not None:
task.cancel()
await telemetry.flush()
if engine is not None:
await engine.dispose()
@@ -119,8 +135,44 @@ app.include_router(repos.router)
app.include_router(webhooks.router)
@app.middleware("http")
async def _count_requests(request: Request, call_next):
"""Heartbeat counts (requests, 4xx/5xx, refusals, p95). Never raises."""
start = time.perf_counter()
marker = SYNTHETIC.set(is_synthetic(request.headers))
try:
response = await call_next(request)
finally:
SYNTHETIC.reset(marker)
tel = getattr(request.app.state, "telemetry", None)
if tel is not None:
tel.record_request(
response.status_code,
(time.perf_counter() - start) * 1000,
refused=getattr(request.state, "refused", False),
)
return response
@app.exception_handler(RepairPointer)
async def _repair_pointer_handler(_, exc: RepairPointer) -> JSONResponse:
async def _repair_pointer_handler(request: Request, exc: RepairPointer) -> JSONResponse:
tel = getattr(request.app.state, "telemetry", None)
detail = exc.detail if isinstance(exc.detail, dict) else {}
code = detail.get("code")
if tel is not None and code in tel.auth_codes:
# A refusal is a failure row (field-visibility rule 1). The caller is
# unauthenticated by definition, so: system actor, no actor_id, and
# the route TEMPLATE, never the concrete path.
request.state.refused = True
route = request.scope.get("route")
tel.auth_failed(
code=code,
http_status=exc.status_code,
caller=caller_class(request.headers),
route=getattr(route, "path", None),
upstream_status=getattr(exc, "upstream_status", None),
synthetic=is_synthetic(request.headers),
)
return JSONResponse(status_code=exc.status_code, content=exc.detail)

View File

@@ -105,7 +105,7 @@ class DatabaseProvider(Provider):
# TunnelProvider was removed deliberately. See the note in main.py: cloudflared
# binds 127.0.0.1:2000 on the HOST, and this process runs in a container whose
# binds 127.0.0.1:2001 on the HOST, and this process runs in a container whose
# only route to the host is the bridge gateway (172.17.0.1), where nothing is
# listening. Binding the metrics endpoint wider would fix the probe and make a
# metrics bind failure able to take down ingress -- a worse trade than losing

View File

@@ -26,6 +26,7 @@ from pydantic import BaseModel, Field, field_validator
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from api.app import throttle
from api.app.auth import ActorType, Caller, get_caller
from api.app.errors import RepairPointer
from api.app.models.core import (
@@ -208,6 +209,11 @@ async def create_repo(
remediation_tool=None,
)
# Throttle before any side effect. Checking after would let a rate-limited
# agent still create the Gitea repo and only then be told no.
async with _sessionmaker(request)() as session:
await throttle.enforce(session, settings, caller, "repo.create")
gitea = GiteaClient(settings)
owner = _owner_login(caller)
await gitea.ensure_user(owner, f"{owner}@windygit.com")
@@ -234,6 +240,8 @@ async def create_repo(
),
)
session.add(repo)
await session.flush()
await throttle.record(session, caller, "repo.create", repo_id=repo.id)
await session.commit()
await session.refresh(repo)
@@ -337,6 +345,7 @@ async def create_grant(
"""
settings = request.app.state.settings
async with _sessionmaker(request)() as session:
await throttle.enforce(session, settings, caller, "grant.create")
repo = await _load_repo(session, repo_id, caller)
is_owner = (caller.identity_id and repo.identity_id == caller.identity_id) or (
caller.passport and repo.passport == caller.passport
@@ -364,6 +373,8 @@ async def create_grant(
expires_at=expires,
)
session.add(grant)
await session.flush()
await throttle.record(session, caller, "grant.create", repo_id=repo.id)
await session.commit()
await session.refresh(grant)
grant_id, role = grant.id, grant.role

View File

@@ -20,6 +20,7 @@ import httpx
from api.app.config import Settings
from api.app.errors import RepairPointer, provider_unconfigured
from api.app.telemetry import synthetic_headers
_TIMEOUT = httpx.Timeout(20.0, connect=5.0)
@@ -36,6 +37,7 @@ class GiteaClient:
return {
"Authorization": f"token {self._s.gitea_admin_token}",
"Content-Type": "application/json",
**synthetic_headers(), # end-to-end synthetic convention (Telemetry UPDATE 4)
}
async def _request(self, method: str, path: str, **kw: Any) -> httpx.Response:

255
api/app/telemetry.py Normal file
View File

@@ -0,0 +1,255 @@
"""Field telemetry to the admin ledger (admin.windyword.ai) — step 2, 2026-09-23.
Shapes are DECLARED with Telemetry Boss (the ledger owner); the server
quarantines any row that doesn't match, so never add a key or a code here
without re-declaring it first:
service.boot once per process start {commit_sha, version, environment}
service.health hourly, in-process interval_s, uptime_s, requests,
errors_5xx, errors_4xx,
refusals_4xx, p95_ms
forge.auth.failed every refused request {code, http_status, caller,
route?, upstream_status?}
Refusals come first: a refused caller is the most expensive silent failure
("the button did nothing"). An UNAUTHENTICATED caller has no trustworthy id, so
per the all-lanes actor rule the row is actor_type "system", no actor_id, and
the caller class goes in metadata.caller.
Privacy: codes, statuses, route TEMPLATES, counts, durations. Never a passport
number, an email, a token fragment or a concrete path with names in it.
No token → nothing is sent and nothing is buffered. A failed flush keeps the
rows (bounded) and retries on the next tick; it never raises into a request.
"""
from __future__ import annotations
import asyncio
import contextvars
import json
import logging
import time
import urllib.request
from datetime import UTC, datetime
log = logging.getLogger("windy-git.telemetry")
PLATFORM, SERVICE = "windy-git", "api"
FLUSH_EVERY_S = 60
HEALTH_EVERY_S = 3600
MAX_BUFFER = 5000
MAX_LATENCY_SAMPLES = 20000
# The declared forge.auth.failed code enum (Telemetry Boss, 2026-09-23). A code
# outside this set is NOT a refusal row — it counts in errors_* instead.
AUTH_CODES = frozenset(
{
"not_signed_in",
"token_invalid",
"token_unrecognised",
"ept_invalid",
"passport_revoked",
"passport_unresolvable",
"agent_read_only",
"agent_rate_limited",
"quota_exceeded",
"trust_unavailable",
"throttle_unavailable",
"service_token_invalid",
"service_auth_unconfigured",
}
)
def _iso(epoch: float) -> str:
return datetime.fromtimestamp(epoch, UTC).isoformat().replace("+00:00", "Z")
# Ecosystem convention (Telemetry UPDATE 4): synthetic traffic travels END TO
# END. Originators (canaries, probes, journeys) send `X-Windy-Synthetic: 1`;
# every service marks all of that request's rows synthetic:true AND forwards the
# header on every downstream call. Absent = real. Never strip it, never set it
# on real traffic. The label separates rows — it never suppresses them.
SYNTHETIC: contextvars.ContextVar[bool] = contextvars.ContextVar("windy_synthetic", default=False)
def is_synthetic(headers) -> bool:
return bool((headers.get("x-windy-synthetic") or "").strip())
def synthetic_headers() -> dict:
"""Merge into every downstream request made while serving this one."""
return {"X-Windy-Synthetic": "1"} if SYNTHETIC.get() else {}
def caller_class(headers) -> str:
"""Declared values: anonymous_human | anonymous_agent | unknown."""
from api.app.ept import looks_like_ept
if headers.get("x-service-token"):
return "unknown"
auth = headers.get("authorization") or ""
if not auth.lower().startswith("bearer "):
return "unknown"
return "anonymous_agent" if looks_like_ept(auth.split(" ", 1)[1].strip()) else "anonymous_human"
class Telemetry:
def __init__(
self,
url: str,
token: str,
*,
environment: str = "",
commit_sha: str | None = None,
version: str = "",
) -> None:
self.url, self.token = url, token
self.environment, self.commit_sha, self.version = environment, commit_sha, version
self.started = time.time()
self.buffer: list[dict] = []
self.auth_codes = AUTH_CODES
self._reset_window()
@property
def enabled(self) -> bool:
return bool(self.token)
def _reset_window(self) -> None:
self.window_start = time.time()
self.requests = self.errors_5xx = self.errors_4xx = self.refusals_4xx = 0
self.latencies_ms: list[float] = []
# UPDATE 7: rows the ledger quarantined (it still answers 202) and rows
# this process lost (buffer overflow). Non-zero = a bug in this emitter.
self.quarantined = self.dropped = 0
# ---- recording (never raises into a request) --------------------------
def record_request(self, status: int, duration_ms: float, *, refused: bool = False) -> None:
self.requests += 1
if status >= 500:
self.errors_5xx += 1
elif refused:
self.refusals_4xx += 1
elif status >= 400:
self.errors_4xx += 1
if len(self.latencies_ms) < MAX_LATENCY_SAMPLES:
self.latencies_ms.append(duration_ms)
def _event(self, event_type: str, metadata: dict, *, ts: float | None = None) -> None:
if not self.enabled:
return
self.buffer.append(
{
"ts": _iso(ts or time.time()),
"platform": PLATFORM,
"service": SERVICE,
"event_type": event_type,
"actor_type": "system",
"metadata": metadata,
}
)
if len(self.buffer) > MAX_BUFFER:
self.dropped += len(self.buffer) - MAX_BUFFER
del self.buffer[: len(self.buffer) - MAX_BUFFER]
def boot(self) -> None:
meta = {"version": self.version, "environment": self.environment}
if self.commit_sha: # unknown is absent, never invented (I-12)
meta["commit_sha"] = self.commit_sha
self._event("service.boot", meta, ts=self.started)
def auth_failed(
self,
*,
code: str,
http_status: int,
caller: str,
route: str | None = None,
upstream_status: int | None = None,
synthetic: bool = False,
) -> None:
if code not in AUTH_CODES:
return
meta: dict = {
"code": code,
"http_status": int(http_status),
"caller": caller,
"synthetic": bool(synthetic),
}
if route:
meta["route"] = route
if upstream_status is not None:
meta["upstream_status"] = int(upstream_status)
self._event("forge.auth.failed", meta)
def health_row(self) -> dict:
now = time.time()
meta = {
"interval_s": int(now - self.window_start),
"uptime_s": int(now - self.started),
"requests": self.requests,
"errors_5xx": self.errors_5xx,
"errors_4xx": self.errors_4xx,
"refusals_4xx": self.refusals_4xx,
"telemetry_quarantined": self.quarantined,
"telemetry_dropped": self.dropped,
}
if self.latencies_ms: # no traffic = no p95, not a fake 0
s = sorted(self.latencies_ms)
meta["p95_ms"] = int(s[min(len(s) - 1, int(0.95 * (len(s) - 1) + 0.5))])
return meta
def health(self) -> None:
self._event("service.health", self.health_row())
self._reset_window()
# ---- sending ------------------------------------------------------------
def _post(self, batch: list[dict]) -> tuple[int, dict]:
req = urllib.request.Request(
self.url,
data=json.dumps({"events": batch}).encode(),
method="POST",
headers={
"Authorization": f"Bearer {self.token}",
"Content-Type": "application/json",
"User-Agent": "windy-git-api-telemetry/1",
},
)
with urllib.request.urlopen(req, timeout=20) as r:
try:
body = json.loads(r.read() or b"{}")
except ValueError:
body = {}
return r.status, body if isinstance(body, dict) else {}
async def flush(self) -> None:
if not self.enabled or not self.buffer:
return
batch = self.buffer[:500]
try:
status, body = await asyncio.to_thread(self._post, batch)
except Exception as exc: # noqa: BLE001 - telemetry must never take the API down
log.warning("telemetry flush failed (%d rows kept): %s", len(self.buffer), exc)
return
if 200 <= status < 300:
del self.buffer[: len(batch)]
self.note_quarantine(body)
def note_quarantine(self, body: dict) -> None:
# 202 does NOT mean every row landed: refused rows are dead-lettered.
q = body.get("quarantined")
if isinstance(q, int) and q > 0:
self.quarantined += q
log.warning("telemetry: %d row(s) QUARANTINED by the ledger: %s", q,
"; ".join(map(str, body.get("rejections") or [])) or "no reason given")
async def run(self) -> None:
"""The one in-process timer: flush every minute, heartbeat every hour."""
last_health = time.monotonic()
while True:
await asyncio.sleep(FLUSH_EVERY_S)
if time.monotonic() - last_health >= HEALTH_EVERY_S:
self.health()
last_health = time.monotonic()
await self.flush()

187
api/app/throttle.py Normal file
View File

@@ -0,0 +1,187 @@
"""EI-band velocity limits (G3.4) — throttle by trust, never by omission.
`BAND_MULTIPLIER` and the `rate_*_per_day` settings existed since G0 and were
**read by nothing**: a documented invariant with no implementation, which is the
exact failure pattern the ecosystem audits kept finding. This module is the
missing consumer.
The doctrine (§0.6) is *capability-completeness*: an agent is never denied a
capability it should have, it is **rate-limited by how much it has proven**.
Platinum gets 10x, gold 4x, standard 1x, watch 0.5x, and untrusted is read-only.
Counting is done against `agent_actions`, which is already the append-only record
of every agent write. That is deliberate: a limiter with its own private counter
disagrees with the audit log the moment either is restarted, and then nobody can
say what actually happened. One source of truth, queried.
**Fail-closed.** If the count cannot be taken, the action is refused. A limiter
that fails open is decoration — it protects you right up until the moment
something is wrong, which is the only moment it matters.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime, timedelta
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from api.app.auth import BAND_MULTIPLIER, ActorType, Caller
from api.app.config import Settings
from api.app.errors import RepairPointer
from api.app.models.core import AgentAction
log = logging.getLogger(__name__)
WINDOW = timedelta(days=1)
# Actions this module ACTUALLY enforces: they pass through our API, so we can
# both count and refuse them.
ACTION_BASE: dict[str, str] = {
"repo.create": "rate_repo_creates_per_day",
"grant.create": "rate_grants_per_day",
}
# ⚠️ DECLARED BUT NOT ENFORCEABLE HERE — and named, rather than quietly listed
# alongside the real ones.
#
# `git push` goes straight to Gitea over HTTPS and never touches this API, so
# nothing records a `push` action and a count of them would be zero forever.
# Listing these in ACTION_BASE (as this module first did) would make `enforce`
# look up a limit, count nothing, and allow everything — a silent no-op wearing
# the costume of a control. That is the same dead-code pattern this module was
# written to remove, and it is worse here because the config name implies the
# protection exists.
#
# Enforcing push velocity requires a Gitea-side hook (pre-receive or the push
# webhook) that reports into `agent_actions`. Until that exists these settings
# are inert, and saying so is the honest option.
NOT_ENFORCED_HERE: dict[str, str] = {
"push": "rate_pushes_per_day",
"push.force": "rate_force_pushes_per_day",
}
def limit_for(settings: Settings, action: str, band: str | None) -> int:
"""Effective per-day allowance. Unknown bands get the standard rate.
Unknown-band handling is a real decision, not a default. Eternitas began
emitting `unproven` on 2026-07-30 without it appearing in the documented
enum. Treating an unrecognised band as untrusted would lock out every freshly
hatched agent the day Eternitas adds a name; treating it as platinum would be
a hole. Standard-with-no-bonus is the honest middle.
"""
base = getattr(settings, ACTION_BASE[action])
mult = BAND_MULTIPLIER.get((band or "").lower(), 1.0)
return int(base * mult)
async def enforce(
session: AsyncSession,
settings: Settings,
caller: Caller,
action: str,
) -> None:
"""Raise 429 if this agent has spent its allowance. No-op for humans.
Humans are governed by account tier and their own session; this is the
agent-velocity control specifically (I-6: parity in capability, asymmetry in
throttle).
"""
if caller.actor_type != ActorType.agent or not caller.passport:
return
if action not in ACTION_BASE: # unknown action = unlimited would be a hole
raise RepairPointer(
status_code=500,
code="throttle_unknown_action",
speak="Something went wrong on our side. Nothing was changed.",
machine_cause=f"no rate base configured for action {action!r}",
remediation_tool=None,
)
band = (caller.band or "").lower()
# Untrusted is read-only. This is a capability decision, not a rate: it gets
# a 403 with a different explanation, because "slow down" would be a lie.
if BAND_MULTIPLIER.get(band, 1.0) <= 0:
raise RepairPointer(
status_code=403,
code="agent_read_only",
speak="That helper can look, but it isn't allowed to make changes yet.",
machine_cause=f"passport {caller.passport} band={band!r} is read-only",
remediation_tool=None,
)
allowed = limit_for(settings, action, band)
since = datetime.now(UTC) - WINDOW
try:
used = (
await session.execute(
select(func.count())
.select_from(AgentAction)
.where(
AgentAction.passport == caller.passport,
AgentAction.action == action,
AgentAction.result == "ok",
AgentAction.ts >= since,
)
)
).scalar_one()
except Exception as exc: # noqa: BLE001
# FAIL CLOSED. A limiter that fails open protects you until the moment
# something is wrong, which is the only moment it matters.
log.warning("throttle count failed for %s/%s: %s", caller.passport, action, exc)
raise RepairPointer(
status_code=503,
code="throttle_unavailable",
speak="We couldn't check that helper's limits, so we didn't make the change.",
machine_cause=f"agent_actions count failed: {type(exc).__name__}",
remediation_tool=None,
) from exc
if used >= allowed:
raise RepairPointer(
status_code=429,
code="agent_rate_limited",
speak=(
"That helper has done a lot in the last day, so we've paused it. "
"It'll be able to continue shortly."
),
machine_cause=(
f"passport {caller.passport} used {used}/{allowed} of {action} "
f"in 24h (band={band or 'unknown'})"
),
remediation_tool=None,
used=used,
allowed=allowed,
band=band or "unknown",
)
async def record(
session: AsyncSession,
caller: Caller,
action: str,
result: str = "ok",
repo_id=None,
) -> None:
"""Append the action that was just allowed.
Written AFTER the work succeeds, on purpose: counting attempts would let a
failing agent exhaust its own allowance by retrying, turning a transient
error into a lockout.
"""
if caller.actor_type != ActorType.agent or not caller.passport:
return
session.add(
AgentAction(
passport=caller.passport,
repo_id=repo_id,
action=action,
ei_at_action=caller.band,
result=result,
)
)
await session.flush()

View File

@@ -0,0 +1,92 @@
"""The canary's login probe must end the session it opens (journey cleanup rule)."""
from __future__ import annotations
import importlib.util
import io
import sys
import urllib.error
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
_spec = importlib.util.spec_from_file_location("canary", ROOT / "scripts" / "canary.py")
canary = importlib.util.module_from_spec(_spec)
sys.modules["canary"] = canary # dataclasses resolve their module by name
_spec.loader.exec_module(canary)
class _Resp:
def __init__(self, status=200, body=b"{}"):
self.status, self._body = status, body
def read(self):
return self._body
def __enter__(self):
return self
def __exit__(self, *a):
return False
def _err(code):
return urllib.error.HTTPError(canary.LOGOUT_URL, code, "x", {}, io.BytesIO(b""))
def _script(monkeypatch, outcomes):
calls = []
def fake(req, timeout=None):
calls.append((req.get_method(), req.full_url, req.get_header("Authorization")))
o = outcomes.pop(0)
if isinstance(o, Exception):
raise o
return o
monkeypatch.setattr(canary.urllib.request, "urlopen", fake)
return calls
def test_logout_ends_the_session(monkeypatch):
calls = _script(monkeypatch, [_Resp(200)])
r = canary.logout("tok", sleep=lambda s: None)
assert r.status == "ok"
assert calls == [("POST", canary.LOGOUT_URL, "Bearer tok")]
def test_5xx_and_no_response_are_retried_then_succeed(monkeypatch):
calls = _script(monkeypatch, [_err(502), OSError("reset"), _Resp(200)])
assert canary.logout("tok", sleep=lambda s: None).status == "ok"
assert len(calls) == 3
def test_already_over_counts_as_done(monkeypatch):
_script(monkeypatch, [_err(401)])
assert canary.logout("tok", sleep=lambda s: None).status == "ok"
def test_other_4xx_fails_fast_and_honestly(monkeypatch):
calls = _script(monkeypatch, [_err(400)])
r = canary.logout("tok", sleep=lambda s: None)
assert r.status == "down" and r.detail.startswith("CLEANUP FAILED") and len(calls) == 1
def test_retries_are_bounded_and_reported(monkeypatch):
calls = _script(monkeypatch, [_err(503)] * 8)
r = canary.logout("tok", attempts=8, sleep=lambda s: None)
assert r.status == "down" and "CLEANUP FAILED after 8 tries" in r.detail and len(calls) == 8
def test_login_probe_logs_out_with_the_token_it_got(monkeypatch):
calls = _script(monkeypatch, [_Resp(200, b'{"token": "abc"}'), _Resp(200)])
c = canary.Check("identity.login", "https://account.windyword.ai/api/v1/auth/login", "x",
method="POST", body={"email": "e", "password": "p"},
after=canary._logout_after_login)
r = canary._probe(c)
assert r.status == "ok" and [f.status for f in r.followups] == ["ok"]
assert calls[1] == ("POST", canary.LOGOUT_URL, "Bearer abc")
def test_login_without_token_is_a_cleanup_failure_not_a_pass():
[f] = canary._logout_after_login(b"{}")
assert f.status == "down" and "CLEANUP FAILED" in f.detail

View File

@@ -0,0 +1,184 @@
"""Compute guard: Windy Mind is the only door to AI compute (warn-only today)."""
from __future__ import annotations
import importlib.util
import subprocess
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[2]
_spec = importlib.util.spec_from_file_location("compute_guard", ROOT / "scripts" / "compute_guard.py")
cg = importlib.util.module_from_spec(_spec)
sys.modules["compute_guard"] = cg
_spec.loader.exec_module(cg)
ALLOW = cg.load_allow(ROOT / "ci" / "compute-guard-allow.yml")
@pytest.mark.parametrize(
"path, text, kind",
[
# audit #1 (windy-search, closed) and #2 (windy-chat, live): the shapes they had
("service/app/anthropic_client.py", 'URL = "https://api.anthropic.com/v1/messages"', "provider host"),
("service/app/config.py", 'token = os.environ["ANTHROPIC_OAUTH_TOKEN"]', "provider key"),
("services/agent-roster/lib/llm.js", "const url = 'https://api.groq.com/openai/v1/chat/completions'", "provider host"),
("docker-compose.yml", " GROQ_API_KEY: ${GROQ_API_KEY}", "provider key"),
# audit #3/#4 (windy-pro account-server)
("account-server/src/routes/transcription.ts", "const r = await fetch('https://api.openai.com/v1/audio/transcriptions'", "provider host"),
("account-server/src/config.ts", "openaiKey: process.env.OPENAI_API_KEY,", "provider key"),
# SDKs and deps
("app/llm.py", "from anthropic import Anthropic", "provider SDK"),
("app/llm.py", "import openai", "provider SDK"),
("app/llm.py", "import google.generativeai as genai", "provider SDK"),
("src/ai.ts", 'import Anthropic from "@anthropic-ai/sdk";', "provider SDK"),
("src/ai.js", "const Groq = require('groq-sdk')", "provider SDK"),
("package.json", ' "openai": "^4.52.0",', "provider SDK dep"),
("requirements.txt", "anthropic>=0.40", "provider SDK dep"),
("pyproject.toml", ' "google-generativeai>=0.8",', "provider SDK dep"),
],
)
def test_audit_shapes_are_flagged(path, text, kind):
assert kind in [k for k, _ in cg.scan_line(path, text)]
@pytest.mark.parametrize(
"path, text",
[
("app/mind.py", 'MIND = "https://mind.windyword.ai/v1/chat/completions"'), # the door itself
("app/models.py", "openai_compatible = True # Mind speaks the OpenAI wire format"),
("app/x.py", "from app.openai_shim import x"), # a local module, not the SDK
("package.json", ' "openai-types-lite": "1.0.0",'), # a different package
("README.txt", "set OPENAI_API_KEY"), # scanned-by-rule, excluded by SKIP separately
],
)
def test_near_misses_are_not_flagged(path, text):
if cg.SKIP.search(path):
return
assert cg.scan_line(path, text) == []
@pytest.mark.parametrize(
"path",
["tests/test_llm.py", "api/tests/x.py", "src/ai.test.ts", "web/foo.spec.js", "docs/setup.md",
"README.md", "package-lock.json", "uv.lock", "node_modules/openai/index.js", ".github/workflows/ci.yml",
"conftest.py", "app/llm_test.py"],
)
def test_tests_docs_lockfiles_vendored_ci_are_never_scanned(path):
assert cg.SKIP.search(path)
def test_allow_list_needs_a_reason_per_entry(tmp_path):
bad = tmp_path / "a.yml"
bad.write_text("allow:\n - repo: x\n paths: ['*']\n")
with pytest.raises(ValueError):
cg.load_allow(bad)
@pytest.mark.parametrize(
"repo, path, ok",
[
("windy-mind", "app/providers/anthropic.py", True),
("windy-agent", "agent/providers.py", True),
("windy-code", "extensions/windy-ai/src/aiProvider.ts", True),
("windy-code", "web/server/llm.ts", False), # BYOK is the extension only
("windy-connect", "backend/src/writers/claude_code.py", True),
("windy-chat", "services/agent-roster/lib/llm.js", False), # audit #2: must be flagged
("windy-pro", "account-server/src/routes/translations.ts", False),
],
)
def test_allow_list_entries(repo, path, ok):
assert cg.allowed(repo, path, ALLOW) is ok
DIFF = """diff --git a/app/llm.py b/app/llm.py
--- a/app/llm.py
+++ b/app/llm.py
@@ -10,0 +11,2 @@
+import anthropic
+client = anthropic.Anthropic()
diff --git a/tests/test_llm.py b/tests/test_llm.py
--- /dev/null
+++ b/tests/test_llm.py
@@ -0,0 +1 @@
+import anthropic
@@ -40 +42 @@
-x = 1
+x = 2
"""
def test_only_added_non_test_lines_are_findings():
fs = cg.parse_added("windy-chat", DIFF, ALLOW)
assert [(f.path, f.line, f.kind) for f in fs] == [("app/llm.py", 11, "provider SDK")]
def _repo(tmp_path, files: dict[str, str]) -> tuple[Path, str]:
work = tmp_path / "w"
work.mkdir()
run = lambda *a: subprocess.run(["git", *a], cwd=work, check=True, capture_output=True) # noqa: E731
run("init", "-q", "-b", "main")
for p, text in files.items():
(work / p).parent.mkdir(parents=True, exist_ok=True)
(work / p).write_text(text)
run("add", "-A")
run("-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "x")
bare = tmp_path / "r.git"
subprocess.run(["git", "clone", "-q", "--bare", str(work), str(bare)], check=True)
sha = subprocess.run(["git", "--git-dir", str(bare), "rev-parse", "main"],
capture_output=True, text=True, check=True).stdout.strip()
return bare, sha
def test_tree_scan_on_a_real_git_repo(tmp_path):
bare, sha = _repo(tmp_path, {
"app/llm.py": "import os\nKEY = os.environ['OPENAI_API_KEY']\n",
"app/ok.py": "MIND = 'https://mind.windyword.ai'\n",
"tests/test_llm.py": "import anthropic\n",
"docs/x.md": "api.anthropic.com\n",
})
fs = cg.scan_tree("windy-chat", bare, sha, ALLOW)
assert [(f.path, f.line, f.kind) for f in fs] == [("app/llm.py", 2, "provider key")]
def test_warn_mode_never_turns_red(monkeypatch):
monkeypatch.setattr(cg, "MODE", "warn")
state, desc, f = cg.status_for([cg.Finding("a.py", 3, "provider host", "api.openai.com")], whole_tree=False)
assert state == "success" and desc.startswith("⚠ WARN (not blocking): 1 direct AI-provider use added")
assert "a.py:3" in desc and f.path == "a.py"
def test_block_mode_fails(monkeypatch):
monkeypatch.setattr(cg, "MODE", "block")
state, desc, _ = cg.status_for([cg.Finding("a.py", 3, "provider host", "x")], whole_tree=True)
assert state == "failure" and desc.startswith("BLOCKED")
def test_clean_is_ok():
assert cg.status_for([], whole_tree=True)[:2] == (
"success", "OK: no direct AI-provider use in tree (Windy Mind is the only door)")
@pytest.mark.parametrize(
"text",
[
" # The ANTHROPIC_OAUTH_TOKEN setting was removed on 2026-09-23 ON PURPOSE", # windy-search
"# ANTHROPIC_API_KEY=",
" // fallback used to call https://api.groq.com directly",
" * @see https://api.openai.com/v1/audio",
"<!-- api.anthropic.com -->",
],
)
def test_comments_are_not_calls(text):
assert cg.scan_line("service/app/config.py", text) == []
def test_code_with_a_trailing_comment_still_counts():
assert cg.scan_line("a.js", "fetch('https://api.openai.com/v1') // TODO move to Mind")
def test_windy_pro_desktop_is_byok_but_the_account_server_is_not():
assert cg.allowed("windy-pro", "src/client/desktop/main.js", ALLOW)
assert not cg.allowed("windy-pro", "account-server/src/routes/translations.ts", ALLOW)

View File

@@ -0,0 +1,312 @@
"""Behavioral tests for EPT verification and EI throttling.
These sign real ES256 tokens with a locally-generated key and verify against a
locally-served key set, so they exercise the ACTUAL crypto path with no network
dependency and no reliance on Eternitas being reachable.
This file exists because the suite it joins was ~86 "does the source contain
this string" assertions and zero that ran the auth decision — which is how a
live impersonation bypass passed every test on 2026-08-13.
"""
from __future__ import annotations
import time
import jwt
import pytest
from cryptography.hazmat.primitives.asymmetric import ec
from api.app import ept as ept_mod
from api.app.auth import BAND_MULTIPLIER
from api.app.config import Settings
from api.app.ept import EptInvalid, verify_ept
from api.app.throttle import ACTION_BASE, limit_for
ISSUER = "eternitas.ai"
KID = "test-key-1"
@pytest.fixture
def signing(monkeypatch):
"""A real EC keypair; point the verifier's JWKS lookup at its public half."""
key = ec.generate_private_key(ec.SECP256R1())
class _FakeJWK:
def __init__(self, k):
self.key = k
class _FakeClient:
def __init__(self, *a, **kw):
pass
def get_signing_key_from_jwt(self, token):
header = jwt.get_unverified_header(token)
if header.get("kid") != KID:
raise Exception(f"unknown kid {header.get('kid')!r}")
return _FakeJWK(key.public_key())
monkeypatch.setattr(ept_mod, "_jwks_client", None)
monkeypatch.setattr(ept_mod, "PyJWKClient", _FakeClient)
return key
def _sign(key, claims, alg="ES256", kid=KID):
return jwt.encode(claims, key, algorithm=alg, headers={"kid": kid, "typ": "EPT"})
def _claims(**over):
c = {
"sub": "ET26-TEST-0001",
"iss": ISSUER,
"iat": int(time.time()) - 10,
"exp": int(time.time()) + 3600,
}
c.update(over)
return c
# ---- the property that was broken ----------------------------------------
def test_genuine_ept_is_accepted(signing):
v = verify_ept(_sign(signing, _claims()), "https://api.eternitas.ai")
assert v.passport == "ET26-TEST-0001"
def test_passport_comes_from_sub_not_a_passport_claim(signing):
"""Real EPTs put the passport in `sub`. The pre-fix code read `passport` /
`sub_passport`, which no genuine EPT carries — so real agents were never
recognised and only forged tokens ever worked."""
tok = _sign(signing, _claims(sub="ET26-REAL-9999", passport="ET26-LIES-0000"))
assert verify_ept(tok, "https://api.eternitas.ai").passport == "ET26-REAL-9999"
def test_alg_none_is_refused(signing):
"""The exact 2026-08-13 exploit."""
import base64
import json as _j
def seg(d):
return base64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
forged = f"{seg({'alg':'none','typ':'EPT','kid':KID})}.{seg(_claims())}."
with pytest.raises(EptInvalid):
verify_ept(forged, "https://api.eternitas.ai")
def test_signature_from_a_different_key_is_refused(signing):
attacker = ec.generate_private_key(ec.SECP256R1())
with pytest.raises(EptInvalid):
verify_ept(_sign(attacker, _claims()), "https://api.eternitas.ai")
def test_tampered_payload_is_refused(signing):
tok = _sign(signing, _claims())
h, _p, s = tok.split(".")
import base64
import json as _j
evil = base64.urlsafe_b64encode(
_j.dumps(_claims(sub="ET26-EVIL-0000")).encode()
).rstrip(b"=").decode()
with pytest.raises(EptInvalid):
verify_ept(f"{h}.{evil}.{s}", "https://api.eternitas.ai")
def test_expired_token_is_refused(signing):
"""Genuinely signed, genuinely expired — proves exp is enforced rather than
the token merely failing to parse."""
tok = _sign(signing, _claims(exp=int(time.time()) - 5, iat=int(time.time()) - 100))
with pytest.raises(EptInvalid):
verify_ept(tok, "https://api.eternitas.ai")
def test_wrong_issuer_is_refused(signing):
"""Correctly signed by a trusted key but claiming another issuer."""
with pytest.raises(EptInvalid):
verify_ept(_sign(signing, _claims(iss="evil.example.com")), "https://api.eternitas.ai")
def test_unknown_kid_is_refused(signing):
with pytest.raises(EptInvalid):
verify_ept(_sign(signing, _claims(), kid="attacker-key"), "https://api.eternitas.ai")
def test_missing_required_claims_are_refused(signing):
for missing in ("sub", "exp"):
c = _claims()
c.pop(missing)
with pytest.raises(EptInvalid):
verify_ept(_sign(signing, c), "https://api.eternitas.ai")
def test_only_es256_is_ever_accepted():
"""Widening this list reopens algorithm confusion."""
assert ept_mod.ALGORITHMS == ["ES256"]
# ---- the throttle that used to be dead code ------------------------------
def test_band_multiplier_is_actually_consumed():
"""BAND_MULTIPLIER was defined and read by nothing before this."""
s = Settings()
assert limit_for(s, "repo.create", "platinum") == s.rate_repo_creates_per_day * 10
assert limit_for(s, "repo.create", "gold") == s.rate_repo_creates_per_day * 4
assert limit_for(s, "repo.create", "standard") == s.rate_repo_creates_per_day
assert limit_for(s, "repo.create", "watch") == s.rate_repo_creates_per_day // 2
def test_unknown_band_gets_standard_not_unlimited_and_not_zero():
s = Settings()
assert limit_for(s, "repo.create", "a-band-invented-tomorrow") == s.rate_repo_creates_per_day
assert limit_for(s, "repo.create", None) == s.rate_repo_creates_per_day
def test_untrusted_band_is_read_only():
assert BAND_MULTIPLIER["untrusted"] == 0
def test_every_throttled_action_has_a_configured_base():
s = Settings()
for action, field in ACTION_BASE.items():
assert getattr(s, field) > 0, f"{action} has no positive base rate"
# ---- revocation enforced on the live trust path (not just the webhook) ----
def test_revoked_passport_is_refused_by_trust_decision():
"""A revoked passport returns HTTP 200, status=revoked, band=unproven,
allowed=[] (verified live 2026-08-13). The decision must refuse it — the
old code returned band 'unproven' and seated the agent."""
from api.app.auth import PassportNotInGoodStanding, decide_trust
revoked = {"status": "revoked", "band": "unproven", "allowed_actions": []}
with pytest.raises(PassportNotInGoodStanding):
decide_trust(revoked, "ET26-NJQT-QMR0")
def test_active_passport_is_accepted_by_trust_decision():
from api.app.auth import decide_trust
active = {"status": "active", "band": "gold", "allowed_actions": ["read", "send"]}
band, actions = decide_trust(active, "ET26-1EF9-VJAN")
assert band == "gold" and actions == ("read", "send")
def test_unknown_or_missing_status_fails_closed():
from api.app.auth import PassportNotInGoodStanding, decide_trust
for body in ({"band": "gold"}, {"status": "suspended"}, {"status": "frozen"},
{"status": ""}, {}):
with pytest.raises(PassportNotInGoodStanding):
decide_trust(body, "ET26-X")
@pytest.mark.asyncio
async def test_resolve_passport_raises_on_revoked_status_wiring(monkeypatch):
"""Proves the WIRING, not just the decision: resolve_passport must feed the
real trust body through decide_trust and propagate the refusal. A validly
minted EPT can outlive the passport by ~a year, so this is the path that
stops a revoked-but-still-signed token."""
from api.app import auth
from api.app.auth import PassportNotInGoodStanding, resolve_passport
from api.app.config import Settings
class _Resp:
status_code = 200
def json(self):
return {"status": "revoked", "band": "unproven", "allowed_actions": []}
class _Client:
def __init__(self, *a, **k):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def get(self, *a, **k):
return _Resp()
monkeypatch.setattr(auth.httpx, "AsyncClient", _Client)
settings = Settings(eternitas_platform_api_key="x", eternitas_base_url="https://api.eternitas.ai")
with pytest.raises(PassportNotInGoodStanding):
await resolve_passport(settings, "ET26-NJQT-QMR0")
@pytest.mark.asyncio
async def test_resolve_passport_returns_band_on_active_wiring(monkeypatch):
import httpx # noqa: F401
from api.app import auth
from api.app.auth import resolve_passport
from api.app.config import Settings
class _Resp:
status_code = 200
def json(self):
return {"status": "active", "band": "gold", "allowed_actions": ["read"]}
class _Client:
def __init__(self, *a, **k): pass
async def __aenter__(self): return self
async def __aexit__(self, *a): return False
async def get(self, *a, **k): return _Resp()
monkeypatch.setattr(auth.httpx, "AsyncClient", _Client)
settings = Settings(eternitas_platform_api_key="x")
band, actions = await resolve_passport(settings, "ET26-1EF9-VJAN")
assert band == "gold" and actions == ("read",)
def test_routing_does_not_depend_on_signature_wellformedness():
"""An EPT-shaped token must route to the EPT verifier even when its
signature is malformed. jwt.get_unverified_header() validates the whole
token and rejects bad signature padding, which used to push such tokens to
the human path — refused for the wrong reason, and readable as a human
identity via `sub` whenever require_verified_jwt was off."""
import base64
import json as _j
from api.app.ept import looks_like_ept
def seg(d):
return base64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
for hdr in ({"alg": "none", "typ": "EPT"}, {"alg": "ES256", "typ": "EPT"},
{"alg": "ES256"}):
tok = f"{seg(hdr)}.{seg({'sub': 'ET26-X'})}.x" # deliberately bad signature
assert looks_like_ept(tok), f"{hdr} did not route to the EPT verifier"
assert not looks_like_ept("not-a-token")
assert not looks_like_ept("")
def test_only_actions_that_route_through_this_api_are_claimed_enforced():
"""git push never touches this API, so a push limit here would count zero
forever and allow everything — a silent no-op wearing the costume of a
control. Push limits must stay in NOT_ENFORCED_HERE until a Gitea-side hook
reports pushes into agent_actions."""
from api.app.throttle import ACTION_BASE, NOT_ENFORCED_HERE
assert "push" not in ACTION_BASE
assert "push.force" not in ACTION_BASE
assert "push" in NOT_ENFORCED_HERE
assert not set(ACTION_BASE) & set(NOT_ENFORCED_HERE)
@pytest.mark.asyncio
async def test_enforce_refuses_an_action_it_cannot_actually_limit():
"""Asking to throttle 'push' must raise, not silently allow."""
from api.app.auth import ActorType, Caller
from api.app.config import Settings
from api.app.errors import RepairPointer
from api.app.throttle import enforce
caller = Caller(actor_type=ActorType.agent, passport="ET26-X", band="gold")
with pytest.raises(RepairPointer) as exc:
await enforce(None, Settings(), caller, "push")
assert exc.value.code == "throttle_unknown_action"

169
api/tests/test_hub_jwt.py Normal file
View File

@@ -0,0 +1,169 @@
"""G3.2 / I-8 — human tokens are verified, never read (SSO #14, 2026-09-23).
Behavioral: every case signs a real RS256 token with a locally generated key
and drives `get_caller`, so a green run means the gate refuses what it must —
not that some string appears in auth.py.
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import json
import time
import jwt
import pytest
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from api.app import hub_jwt
from api.app.config import Settings
from api.app.errors import RepairPointer
KEY = rsa.generate_private_key(public_exponent=65537, key_size=2048)
OTHER = rsa.generate_private_key(public_exponent=65537, key_size=2048)
IDENTITY = "5e1b9569-7f01-489d-bf14-6fe5a367fa3f"
def _claims(**over):
now = int(time.time())
c = {
"iss": "windy-identity",
"type": "human",
"sub": "row-id-not-identity",
"windy_identity_id": IDENTITY,
"email": "grant@example.com",
"iat": now,
"exp": now + 900,
}
c.update(over)
return {k: v for k, v in c.items() if v is not None}
def _sign(claims, key=KEY, alg="RS256"):
return jwt.encode(claims, key, algorithm=alg, headers={"kid": "test"})
class _Req:
def __init__(self, settings):
self.app = type("A", (), {"state": type("S", (), {"settings": settings})()})()
@pytest.fixture(autouse=True)
def _local_jwks(monkeypatch):
"""The hub's JWKS, served from KEY's public half — no network."""
class _Key:
key = KEY.public_key()
class _Client:
def get_signing_key_from_jwt(self, token):
return _Key()
monkeypatch.setattr(hub_jwt, "_client", lambda base_url: _Client())
async def _caller(token, **settings):
from api.app.auth import get_caller
s = Settings(environment="production", **settings)
return await get_caller(_Req(s), authorization=f"Bearer {token}", x_service_token=None)
async def _refused(token, **settings):
with pytest.raises(RepairPointer) as exc:
await _caller(token, **settings)
assert exc.value.status_code == 401 and exc.value.code == "token_invalid"
return exc.value
@pytest.mark.asyncio
async def test_genuine_hub_token_is_a_human_named_by_windy_identity_id():
c = await _caller(_sign(_claims()))
assert c.actor_type == "human"
assert c.identity_id == IDENTITY # NOT `sub`, which is the per-row user id
@pytest.mark.asyncio
async def test_forged_signature_is_refused():
await _refused(_sign(_claims(), key=OTHER))
@pytest.mark.asyncio
async def test_expired_token_is_refused():
await _refused(_sign(_claims(iat=int(time.time()) - 2000, exp=int(time.time()) - 60)))
@pytest.mark.asyncio
async def test_wrong_issuer_and_id_tokens_are_refused():
await _refused(_sign(_claims(iss="https://evil.example")))
# Contract v1: the discovery-URL issuer is legal for ACCESS tokens...
c = await _caller(_sign(_claims(iss="https://account.windyword.ai")))
assert c.identity_id == IDENTITY
# ...but an id_token minted for the forge (aud = Gitea's client id
# "windy-git", no type, sub = identity) must never act as a bearer here.
id_token = _claims(
iss="https://account.windyword.ai",
aud="windy-git",
type=None,
windy_identity_id=None,
sub=IDENTITY,
)
await _refused(_sign(id_token))
await _refused(_sign(dict(id_token, windy_identity_id=IDENTITY, type="human")))
@pytest.mark.asyncio
async def test_hs256_confusion_is_refused():
# The classic forgery: HMAC the token with the PUBLIC key as the secret.
pub = KEY.public_key().public_bytes(
serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo
)
header = base64.urlsafe_b64encode(json.dumps({"alg": "HS256", "typ": "JWT"}).encode()).rstrip(
b"="
)
body = base64.urlsafe_b64encode(json.dumps(_claims()).encode()).rstrip(b"=")
sig = base64.urlsafe_b64encode(
hmac.new(pub, header + b"." + body, hashlib.sha256).digest()
).rstrip(b"=")
await _refused((header + b"." + body + b"." + sig).decode())
@pytest.mark.asyncio
async def test_non_human_or_missing_type_is_refused():
await _refused(_sign(_claims(type="agent")))
await _refused(_sign(_claims(type=None)))
@pytest.mark.asyncio
async def test_missing_windy_identity_is_refused_not_read_from_sub():
await _refused(_sign(_claims(windy_identity_id=None)))
@pytest.mark.asyncio
async def test_aud_is_tolerated_when_it_names_windy_git_and_refused_otherwise():
"""PyJWT rejects ANY aud-bearing token when no audience is configured — the
trap that would break the day the hub starts emitting aud."""
c = await _caller(_sign(_claims(aud=["windy_chat", "windy_git", "windy_mail"])))
assert c.identity_id == IDENTITY
await _refused(_sign(_claims(aud=["windy_chat"])))
@pytest.mark.asyncio
async def test_require_aud_refuses_tokens_without_it():
await _refused(_sign(_claims()), hub_require_aud=True)
c = await _caller(_sign(_claims(aud=["windy_git"])), hub_require_aud=True)
assert c.identity_id == IDENTITY
@pytest.mark.asyncio
async def test_production_verifies_even_if_the_flag_is_off():
"""require_verified_jwt=False is a local-dev convenience; production must
never take the unverified path."""
await _refused(_sign(_claims(), key=OTHER), require_verified_jwt=False)
def test_algorithm_list_is_exactly_rs256():
assert hub_jwt.ALGORITHMS == ["RS256"]

View File

@@ -308,12 +308,13 @@ def test_g36_trust_client_never_soft_allows():
def test_g36_unverified_human_jwt_is_refused_in_production():
"""I-8 applied to ourselves: an unverified JWT is an authentication bypass,
not a shortcut. Until G3.2's JWKS verifier exists, production refuses."""
not a shortcut. G3.2's verifier now exists; behavioral proof that forged,
expired, mis-issued and mis-audienced tokens are refused lives in
test_hub_jwt.py. Here: the gate defaults closed."""
from api.app.config import Settings
assert Settings().require_verified_jwt is True
src = (ROOT / "api" / "app" / "auth.py").read_text()
assert "human_signin_not_ready" in src
assert "windy-git" not in Settings().hub_audiences # Gitea's client_id: id_token confusion
def test_no_auth_bypass_env_var_anywhere():
@@ -424,9 +425,28 @@ def test_i05_jobs_get_a_network_per_job_not_a_shared_bridge():
def test_i05_jobs_cannot_bind_mount_from_the_daemon_host():
cfg = (ROOT / "deploy" / "runner" / "config.yaml").read_text()
assert "valid_volumes: []" in cfg
assert 'docker_host: "-"' in cfg
"""Narrowed 2026-09-23 (orchestrator-approved): a job may bind-mount EXACTLY
one daemon path, windy-pro's non-secret build inputs, and only because dind
itself has that path READ-ONLY. Anything more (a second path, a writable
one, a glob) reopens the host to CI code. Still no docker socket for jobs."""
import re
import yaml
rd = ROOT / "deploy" / "runner"
cfg = yaml.safe_load((rd / "config.yaml").read_text())
allowed = cfg["container"]["valid_volumes"]
assert allowed in ([], ["/ci-inputs/windy-pro"]), f"I-5: jobs may mount nothing else: {allowed}"
assert cfg["container"]["docker_host"] == "-"
if allowed:
compose = yaml.safe_load((rd / "docker-compose.yml").read_text())
binds = [v for v in compose["services"]["dind"]["volumes"] if v.startswith("/")]
assert binds == ["/home/user1-gpu/ci-inputs/windy-pro:/ci-inputs/windy-pro:ro"], (
f"I-5: dind's only host bind must be the ci-inputs path, READ-ONLY: {binds}")
for name, svc in compose["services"].items():
if name != "dind":
for v in svc.get("volumes") or []:
assert not re.match(r"^/home/user1-gpu/ci-inputs", v), f"I-5: {name} mounts ci-inputs"
def test_i05_no_ci_container_can_reach_the_forge_network():
@@ -492,6 +512,11 @@ def test_g73_workflow_pins_a_python_that_satisfies_requires_python():
for wf in ROOT.rglob(".gitea/workflows/*.y*ml"):
text = wf.read_text()
# Only workflows that actually RUN Python need to pin it. A workflow
# that shells out to psql or curl does not, and demanding a pin from
# it is the test being wrong rather than the workflow.
if not _re.search(r"\bpython3?\b|pytest|pip ", text):
continue
# Either a pinned container image or an explicit setup-python version.
pin = _re.search(r"image:\s*python:(\d+)\.(\d+)", text) or _re.search(
r'python-version:\s*"?(\d+)\.(\d+)"?', text
@@ -658,23 +683,30 @@ def _fake_request(settings):
@_pytest.mark.asyncio
async def test_security_forged_agent_token_is_refused_in_production():
"""A token with alg:none naming a real passport must NOT authenticate.
This is the exploit that returned HTTP 200 on 2026-08-13, exercised through
the real get_caller decision rather than by grepping for a string."""
"""The 2026-08-13 exploit: an alg:none token naming a real passport returned
HTTP 200 as that agent. It must now be refused whichever gate catches it —
an EPT-shaped forgery by signature verification, a JWT-shaped one by the
human gate. What is asserted is REFUSAL, not a particular error code."""
from api.app.auth import get_caller
from api.app.config import Settings
from api.app.errors import RepairPointer
settings = Settings(environment="production", require_verified_jwt=True,
eternitas_platform_api_key="x", eternitas_base_url="https://api.eternitas.ai")
eternitas_platform_api_key="x")
req = _fake_request(settings)
with _pytest.raises(RepairPointer) as exc:
await get_caller(req, authorization=f"Bearer {_forged_bearer('ET26-1EF9-VJAN')}",
x_service_token=None)
# Must be refused, and must be refused BEFORE any trust lookup could seat it.
assert exc.value.status_code in (401, 503)
assert exc.value.code == "agent_signin_not_ready"
# Both shapes now route to the EPT verifier, because alg:none is never
# valid for ANY caller — so 401 "your token is bad" is the honest answer,
# not 503 "that feature isn't ready".
for typ, expected in (("JWT", "ept_invalid"), ("EPT", "ept_invalid")):
def seg(d):
return _b64.urlsafe_b64encode(_json.dumps(d).encode()).rstrip(b"=").decode()
forged = (f"{seg({'alg':'none','typ':typ})}"
f".{seg({'passport':'ET26-1EF9-VJAN','sub':'ET26-1EF9-VJAN'})}.sig")
with _pytest.raises(RepairPointer) as exc:
await get_caller(req, authorization=f"Bearer {forged}", x_service_token=None)
assert exc.value.status_code in (401, 403, 503), f"{typ} was not refused"
assert exc.value.code == expected, f"{typ} -> {exc.value.code}"
@_pytest.mark.asyncio
@@ -687,3 +719,55 @@ async def test_security_no_bearer_is_still_401():
with _pytest.raises(RepairPointer) as exc:
await get_caller(req, authorization=None, x_service_token=None)
assert exc.value.status_code == 401
def test_i12_build_fails_when_commit_sha_is_empty():
"""The sed+grep pair silently accepted an empty COMMIT_SHA: it replaced ""
with "" and then matched that same empty string, shipping a container that
reported commit_sha: null. That is the exact defect I-12 exists to prevent,
and it happened on 2026-08-14."""
df = (ROOT / "Dockerfile").read_text()
assert 'test -n "${COMMIT_SHA}"' in df
# --------------------------------------------------------------------------
# G2.3 — branding lives in the repo, not only on one host's disk
# --------------------------------------------------------------------------
def test_g23_branding_is_version_controlled():
"""It was applied directly to Veron's disk first, which is the config-drift
trap this project documents: the running system and the repo disagree, and
a rebuild silently reverts to stock Gitea."""
b = ROOT / "deploy" / "branding"
for f in ("apply.sh", "README.md", "templates/home.tmpl",
"templates/custom/header.tmpl"):
assert (b / f).exists(), f"missing {f}"
def test_g23_brand_css_filename_is_versioned():
"""Cloudflare caches /assets/* for 6h and no token here can purge, so a
fixed filename leaves stale bytes live for hours."""
import re as _re
hdr = (ROOT / "deploy" / "branding" / "templates" / "custom" / "header.tmpl").read_text()
m = _re.search(r"theme-windy\.v(\d+)\.css", hdr)
assert m, "brand CSS must carry a version in its FILENAME"
assert (ROOT / "deploy" / "branding" / "public" / "assets" / "css"
/ f"theme-windy.v{m.group(1)}.css").exists()
def test_backup_never_bundles_credential_repos_to_r2():
"""kit-army-config (the lockbox) and the *-soul / anima repos carry
credentials; the R2 bundles are plaintext. Behavioural: run the script's
own exclusion function against the names."""
import subprocess
script = (ROOT / "scripts" / "backup.sh").read_text()
fn = script[script.index('EXCLUDE="'):script.index("cleanup()")]
# A file named like a pattern in cwd must not break the match (glob expansion).
probe = "cd \"$(mktemp -d)\" && touch x-soul && " + fn + (
'for n in kit-army-config anima windy-0-soul kit-0c5-soul herm-0-soul '
'soulsafe windy-chat eternitas; do excluded "$n" && echo "X $n" || echo "- $n"; done'
)
out = subprocess.run(["bash", "-c", probe], capture_output=True, text=True, check=True).stdout
skipped = {ln[2:] for ln in out.splitlines() if ln.startswith("X ")}
assert skipped == {"kit-army-config", "anima", "windy-0-soul", "kit-0c5-soul", "herm-0-soul"}

View File

@@ -0,0 +1,384 @@
"""Behavioral tests for scripts/pr_status_bridge.py.
The bridge is the ONLY CI signal the private platform repos get on GitHub, so
these drive its real functions against fake Gitea/GitHub APIs rather than
grepping its source: a status painted green that nobody tested is worse than
no status at all.
"""
from __future__ import annotations
import base64
import importlib.util
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[2]
_spec = importlib.util.spec_from_file_location(
"pr_status_bridge", ROOT / "scripts" / "pr_status_bridge.py"
)
bridge = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(bridge)
SHA = "a" * 40
def _run(i, wf, job, status, sha=SHA, n=1):
return {
"id": i,
"workflow_id": wf,
"name": job,
"status": status,
"head_sha": sha,
"run_number": n,
}
class Fake:
def __init__(self, runs=(), statuses=(), gh_prs=(), wg_prs=(), workflows=None):
self.runs, self.statuses = list(runs), list(statuses)
self.workflows = workflows or {} # {path: yaml text} at every commit
self.gh_prs, self.wg_prs = list(gh_prs), list(wg_prs)
self.posted, self.opened, self.closed = [], [], []
def gitea(self, method, path, body=None):
if "/contents/" in path:
want = path.split("/contents/", 1)[1].split("?", 1)[0]
if want in self.workflows:
return 200, {"content": base64.b64encode(self.workflows[want].encode()).decode()}
files = [
{"type": "file", "name": k.rsplit("/", 1)[1], "path": k}
for k in self.workflows
if k.rsplit("/", 1)[0] == want
]
return (200, files) if files else (404, None)
if "/actions/tasks" in path:
page = int(path.rsplit("page=", 1)[1])
return 200, {"workflow_runs": self.runs[(page - 1) * 50 : page * 50]}
if method == "GET" and path.endswith("/pulls?state=open&limit=50"):
return 200, self.wg_prs
if method == "POST" and path.endswith("/pulls"):
self.opened.append(body)
return 201, {}
if method == "PATCH":
self.closed.append(path)
return 201, {}
raise AssertionError(path)
def github(self, method, path, body=None):
if "/statuses" in path and method == "GET":
return 200, self.statuses
if "/statuses/" in path and method == "POST":
self.posted.append(body)
return 201, {}
if "/pulls?" in path:
return 200, self.gh_prs
raise AssertionError(path)
@pytest.fixture
def fake(monkeypatch):
def make(queued=(), **kw):
f = Fake(**kw)
monkeypatch.setattr(bridge, "gitea", f.gitea)
monkeypatch.setattr(bridge, "github", f.github)
monkeypatch.setattr(bridge, "queued_jobs", lambda repo, sha: list(queued))
return f
return make
def test_posts_latest_verdict_per_job(fake):
f = fake(runs=[_run(1, "ci.yml", "test", "failure"), _run(2, "ci.yml", "test", "success", n=2)])
bridge.post_statuses("r", SHA)
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "success")]
assert f.posted[0]["target_url"].endswith("/actions/runs/2")
def test_unchanged_state_is_not_reposted(fake):
f = fake(
runs=[_run(1, "ci.yml", "test", "success")],
statuses=[{"context": "windy-git/ci/test", "state": "success"}],
)
bridge.post_statuses("r", SHA)
assert f.posted == []
def test_skipped_job_is_never_painted_green(fake):
f = fake(runs=[_run(1, "substrate-drift.yml", "check", "skipped")])
bridge.post_statuses("r", SHA)
assert f.posted == []
def test_other_commits_runs_are_ignored(fake):
f = fake(runs=[_run(1, "ci.yml", "test", "failure", sha="b" * 40)])
bridge.post_statuses("r", SHA)
assert f.posted == []
def test_runs_past_the_first_page_are_seen(fake):
noise = [_run(100 + i, "drift.yml", "x", "skipped", sha="c" * 40) for i in range(50)]
f = fake(runs=noise + [_run(1, "ci.yml", "test", "success")])
bridge.post_statuses("r", SHA)
assert [p["state"] for p in f.posted] == ["success"]
def _gh_pr(n, repo="sneakyfree/r"):
return {
"number": n,
"title": "t",
"html_url": "u",
"head": {"ref": f"b{n}", "sha": SHA, "repo": {"full_name": repo} if repo else None},
"base": {"ref": "main"},
}
def test_fork_prs_are_never_mirrored(fake, monkeypatch):
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
f = fake(gh_prs=[_gh_pr(1, repo="stranger/r"), _gh_pr(2, repo=None)])
assert bridge.sync_prs("r") == []
assert f.opened == []
def test_pr_mirror_opened_once_and_closed_when_github_closes(fake, monkeypatch):
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
f = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 3, "title": "[GH#5] gone"}])
assert bridge.sync_prs("r") == [SHA]
assert [o["head"] for o in f.opened] == ["b7"]
assert f.closed == ["/repos/windyadmin/r/pulls/3"]
f2 = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 4, "title": "[GH#7] t"}])
bridge.sync_prs("r")
assert f2.opened == [] and f2.closed == []
def test_image_build_jobs_are_not_posted(fake):
"""No Docker daemon in job containers (I-5): a build job's red is structural."""
f = fake(
runs=[_run(1, "ci.yml", "Docker Build", "failure"), _run(2, "ci.yml", "docker", "failure")]
)
bridge.post_statuses("r", SHA)
assert f.posted == []
def test_non_blocking_jobs_are_not_posted_for_that_repo_only(fake, monkeypatch):
"""Grant ruled windy-pro's desktop/installer jobs non-blocking: they must not
reach GitHub for windy-pro, and the rule must not leak to other repos."""
monkeypatch.setattr(bridge, "NON_BLOCKING", {"windy-pro": {"ci/build-desktop"}})
runs = [_run(1, "ci.yml", "build-desktop", "failure"), _run(2, "ci.yml", "test", "success")]
f = fake(runs=runs)
bridge.post_statuses("windy-pro", SHA)
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
f2 = fake(runs=runs)
bridge.post_statuses("windy-chat", SHA)
assert sorted(p["context"] for p in f2.posted) == [
"windy-git/ci/build-desktop",
"windy-git/ci/test",
]
def test_default_non_blocking_is_grants_ruling():
assert bridge.NON_BLOCKING.get("windy-pro") == {
"ci/build-desktop",
"ci/test-installer",
"ci/reality-check",
}
def test_transport_blips_are_retried_but_http_errors_are_not(monkeypatch):
import urllib.error
calls = {"n": 0}
class _R:
status = 200
def read(self):
return b"{}"
def __enter__(self):
return self
def __exit__(self, *a):
return False
def flaky(req, timeout):
calls["n"] += 1
if calls["n"] < 3:
raise urllib.error.URLError("_ssl.c:983: The handshake operation timed out")
return _R()
monkeypatch.setattr(bridge.urllib.request, "urlopen", flaky)
monkeypatch.setattr(bridge.time, "sleep", lambda s: None)
assert bridge._call("http://x", "t", "GET", "/p") == (200, {})
assert calls["n"] == 3
def forbidden(req, timeout):
calls["n"] += 1
raise urllib.error.HTTPError("http://x/p", 403, "no", {}, None)
calls["n"] = 0
monkeypatch.setattr(bridge.urllib.request, "urlopen", forbidden)
assert bridge._call("http://x", "t", "GET", "/p") == (403, None)
assert calls["n"] == 1
GOOD = "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: []\n"
BROKEN = "on: push\njobs:\n test:\n runs-on: x\n steps: [\n"
def test_invalid_workflow_gets_an_error_status_even_with_no_runs(fake):
# Gitea fires NO run for an invalid file: without this the PR shows nothing.
f = fake(workflows={".github/workflows/ci.yml": BROKEN})
bridge.post_statuses("windy-chat", SHA)
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/workflow", "error")]
assert "invalid YAML at line 5" in f.posted[0]["description"]
assert f.posted[0]["target_url"].endswith(f"/src/commit/{SHA}/.github/workflows/ci.yml")
def test_valid_workflows_post_nothing_extra(fake):
f = fake(runs=[_run(1, "ci.yml", "test", "success")], workflows={".github/workflows/ci.yml": GOOD})
bridge.post_statuses("windy-chat", SHA)
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
def test_workflow_error_is_not_reposted(fake):
f = fake(
workflows={".github/workflows/ci.yml": BROKEN},
statuses=[{"context": "windy-git/ci/workflow", "state": "error"}],
)
bridge.post_statuses("windy-chat", SHA)
assert f.posted == []
def test_gitea_dir_wins_over_github_dir(fake):
# Gitea runs .gitea/workflows when it has files and ignores .github/workflows.
f = fake(workflows={".gitea/workflows/ci.yml": GOOD, ".github/workflows/old.yml": BROKEN})
bridge.post_statuses("windy-chat", SHA)
assert f.posted == []
@pytest.mark.parametrize(
"text, problem",
[
(GOOD, None),
("on: push\njobs:\n a:\n uses: ./x.yml\n", None),
(BROKEN, "invalid YAML at line 5"),
("jobs:\n a:\n runs-on: x\n", "no `on:` trigger"),
("on: push\n", "no `jobs:`"),
("on: push\njobs:\n a:\n steps: []\n", "job `a` has no `runs-on:`"),
("- a\n", "not a YAML mapping"),
],
)
def test_workflow_problem(text, problem):
assert bridge.workflow_problem(text) == problem
def _q(n, wf, job):
return {"run_number": n, "workflow_id": wf, "name": job}
def test_queued_job_shows_pending_instead_of_nothing(fake):
f = fake(queued=[_q(5, "ci.yml", "test")])
bridge.post_statuses("windy-chat", SHA)
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "pending")]
assert f.posted[0]["target_url"].endswith("/actions/runs/5")
def test_queued_rerun_supersedes_the_stale_failure(fake):
f = fake(runs=[_run(1, "ci.yml", "test", "failure", n=4)], queued=[_q(7, "ci.yml", "test")])
bridge.post_statuses("windy-chat", SHA)
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "pending")]
def test_older_queued_job_never_overrides_a_newer_verdict(fake):
f = fake(runs=[_run(1, "ci.yml", "test", "success", n=9)], queued=[_q(3, "ci.yml", "test")])
bridge.post_statuses("windy-chat", SHA)
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "success")]
def test_queued_docker_and_non_blocking_jobs_stay_unposted(fake):
f = fake(queued=[_q(2, "ci.yml", "docker-build"), _q(2, "ci.yml", "build-desktop")])
bridge.post_statuses("windy-pro", SHA)
assert f.posted == []
def test_queued_lookup_refuses_unsafe_input():
assert bridge.queued_jobs("x'; drop table t;--", SHA) == []
assert bridge.queued_jobs("windy-chat", "not-a-sha") == []
def _db(monkeypatch, jobs, labels):
import json as _json
import subprocess as _sp
payload = _json.dumps({"jobs": jobs, "labels": [_json.dumps(x) for x in labels]})
monkeypatch.setattr(
bridge.subprocess, "run",
lambda *a, **k: _sp.CompletedProcess(a, 0, stdout=payload, stderr=""),
)
RUNNER = ["veron-1", "linux-x64", "self-hosted", "linux", "x64"]
def test_only_jobs_a_runner_can_take_are_pending(monkeypatch):
# macos-latest is cancelled unpicked by the janitor: pending would never resolve.
_db(monkeypatch, [
{"run_number": 3, "workflow_id": "ci.yml", "name": "test", "runs_on": '["self-hosted","linux","x64"]'},
{"run_number": 3, "workflow_id": "ci.yml", "name": "mac", "runs_on": '["macos-latest"]'},
], [RUNNER])
assert [j["name"] for j in bridge.queued_jobs("windy-chat", SHA)] == ["test"]
def test_lookup_failure_is_non_fatal(monkeypatch):
import subprocess as _sp
def boom(*a, **k):
raise _sp.TimeoutExpired("docker", 30)
monkeypatch.setattr(bridge.subprocess, "run", boom)
assert bridge.queued_jobs("windy-chat", SHA) == []
class _Guard:
def __init__(self, findings):
self.findings = findings
def check(self, repo, sha, default_branch, is_default_head):
return self.findings
@staticmethod
def status_for(findings, whole_tree):
if not findings:
return "success", "OK: clean", None
return "success", f"WARN {len(findings)}", findings[0]
class _F:
path, line = "app/llm.py", 7
def test_guard_posts_warn_with_a_link_to_the_first_finding(fake, monkeypatch):
f = fake()
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
bridge.post_compute_guard("windy-chat", SHA, "main", False)
assert [(p["context"], p["state"], p["description"]) for p in f.posted] == [
("windy-git/compute-guard", "success", "WARN 1")]
assert f.posted[0]["target_url"].endswith(f"/src/commit/{SHA}/app/llm.py#L7")
def test_guard_same_status_is_not_reposted(fake, monkeypatch):
f = fake(statuses=[{"context": "windy-git/compute-guard", "state": "success", "description": "WARN 1"}])
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
bridge.post_compute_guard("windy-chat", SHA, "main", False)
assert f.posted == []
def test_guard_that_cannot_run_posts_nothing(fake, monkeypatch):
f = fake()
monkeypatch.setitem(sys.modules, "compute_guard", _Guard(None))
bridge.post_compute_guard("windy-chat", SHA, "main", True)
assert f.posted == []

View File

@@ -0,0 +1,82 @@
"""Push-velocity detection (scripts/telemetry_emit.py): detect + alert only.
Driven through the real function with rows shaped like the Gitea query's.
"""
from __future__ import annotations
import importlib.util
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "scripts"))
_spec = importlib.util.spec_from_file_location("telemetry_emit", ROOT / "scripts" / "telemetry_emit.py")
te = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(te)
NOW = 1_800_000_000.0
def row(login="agent-et26abcd1234", uid=7, p1h=0, p24h=0, d24h=0, repos=1, wid=None):
return {"uid": uid, "login": login, "wid": wid, "p1h": p1h, "p24h": p24h, "d24h": d24h, "repos": repos}
def test_under_every_threshold_emits_nothing():
ev, keep = te.push_velocity_events([row(p1h=60, p24h=500, d24h=10)], NOW, {})
assert ev == [] and keep == {}
def test_burst_emits_one_declared_row_with_the_passport():
ev, keep = te.push_velocity_events([row(p1h=61, p24h=61, repos=3)], NOW, {})
assert len(ev) == 1
e = ev[0]
assert e["event_type"] == "forge.push_velocity" and e["service"] == "forge"
assert e["actor_type"] == "agent" and e["actor_id"] == "ET26-ABCD-1234"
assert e["metadata"] == {
"rule": "pushes_1h", "window_s": 3600, "count": 61, "threshold": 60,
"repos": 3, "gitea_user_id": 7,
}
assert keep == {"7:pushes_1h": NOW}
def test_still_over_is_reported_once_per_window_not_every_run():
_, keep = te.push_velocity_events([row(p1h=90)], NOW, {})
ev, keep = te.push_velocity_events([row(p1h=95)], NOW + 300, keep)
assert ev == [] and keep == {"7:pushes_1h": NOW}
ev, _ = te.push_velocity_events([row(p1h=95)], NOW + 3601, keep)
assert len(ev) == 1
def test_dropping_back_under_rearms():
_, keep = te.push_velocity_events([row(p1h=90)], NOW, {})
_, keep = te.push_velocity_events([row(p1h=5)], NOW + 300, keep)
assert keep == {}
ev, _ = te.push_velocity_events([row(p1h=90)], NOW + 600, keep)
assert len(ev) == 1
def test_the_sync_account_is_exempt():
ev, _ = te.push_velocity_events([row(login="windyadmin", uid=1, p1h=9999, p24h=9999)], NOW, {})
assert ev == []
def test_sso_human_is_keyed_on_windy_identity_id():
ev, _ = te.push_velocity_events([row(login="u-5e1b9569abc", wid="5e1b9569-full-id", d24h=11)], NOW, {})
assert [(e["actor_type"], e["actor_id"], e["metadata"]["rule"]) for e in ev] == [
("human", "5e1b9569-full-id", "ref_deletes_24h")
]
assert "caller" not in ev[0]["metadata"]
def test_no_provable_id_is_system_plus_caller_never_an_invented_id():
# UPDATE 2 actor rule: agent/human rows without an actor_id are quarantined.
for login in ("u-nolink", "agent-weird"):
ev, _ = te.push_velocity_events([row(login=login, p24h=501)], NOW, {})
assert ev[0]["actor_type"] == "system" and "actor_id" not in ev[0]
assert ev[0]["metadata"]["caller"] == "unknown"
def test_passport_round_trip():
assert te.passport_from_login("agent-et26p1zgttp8") == "ET26-P1ZG-TTP8"
assert te.passport_from_login("u-abc") is None

202
api/tests/test_telemetry.py Normal file
View File

@@ -0,0 +1,202 @@
"""Field telemetry from the API (step 2): behavioural, no network.
A refusal must become exactly one forge.auth.failed row in the DECLARED shape
(the ledger quarantines anything else); non-refusal errors must not; the
heartbeat must count what happened and never invent a p95 for no traffic.
"""
from __future__ import annotations
import httpx
import pytest
from fastapi import Depends, FastAPI
from api.app import telemetry as tmod
from api.app.errors import RepairPointer
DECLARED_AUTH_KEYS = {"code", "http_status", "caller", "route", "upstream_status", "synthetic"}
def _app(tel: tmod.Telemetry) -> FastAPI:
"""The real middleware + handler, re-registered on a bare app (no DB)."""
from api.app import main
app = FastAPI()
app.state.telemetry = tel
app.middleware("http")(main._count_requests)
app.exception_handler(RepairPointer)(main._repair_pointer_handler)
def refuse(code: str, status: int):
def dep():
raise RepairPointer(
status_code=status,
code=code,
speak="no",
machine_cause="test",
remediation_tool=None,
)
return dep
@app.get("/api/v1/repos/{repo}/grants", dependencies=[Depends(refuse("passport_revoked", 403))])
async def grants(repo: str):
return {}
@app.get("/api/v1/nope", dependencies=[Depends(refuse("repo_not_found", 404))])
async def nope():
return {}
@app.get("/ok")
async def ok():
return {"ok": True}
return app
async def _get(app, path, headers=None):
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://t") as c:
return await c.get(path, headers=headers or {})
def _tel():
return tmod.Telemetry(
"http://ledger.invalid/v1/events",
"tok",
environment="test",
commit_sha="abc1234",
version="0.1.0",
)
@pytest.mark.asyncio
async def test_refusal_emits_one_declared_row_with_route_template_not_path():
tel = _tel()
r = await _get(
_app(tel),
"/api/v1/repos/grandmas-secret-project/grants",
{"Authorization": "Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkVQVCJ9.e30.x"},
)
assert r.status_code == 403
rows = [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
assert len(rows) == 1
row = rows[0]
assert row["actor_type"] == "system" and "actor_id" not in row
assert set(row["metadata"]) <= DECLARED_AUTH_KEYS
assert row["metadata"]["code"] == "passport_revoked"
assert row["metadata"]["http_status"] == 403
assert row["metadata"]["caller"] == "anonymous_agent"
assert row["metadata"]["route"] == "/api/v1/repos/{repo}/grants"
assert "grandmas-secret-project" not in str(row)
@pytest.mark.asyncio
async def test_non_auth_errors_are_counted_but_not_refusal_rows():
tel = _tel()
await _get(_app(tel), "/api/v1/nope")
assert not [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
assert tel.errors_4xx == 1 and tel.refusals_4xx == 0
@pytest.mark.asyncio
async def test_heartbeat_counts_requests_refusals_and_p95():
tel = _tel()
app = _app(tel)
for _ in range(3):
await _get(app, "/ok")
await _get(app, "/api/v1/repos/x/grants")
meta = tel.health_row()
assert meta["requests"] == 4 and meta["refusals_4xx"] == 1 and meta["errors_5xx"] == 0
assert isinstance(meta["p95_ms"], int)
assert {"interval_s", "uptime_s"} <= set(meta)
def test_no_traffic_means_no_p95_not_a_fake_zero():
assert "p95_ms" not in _tel().health_row()
def test_no_token_sends_and_buffers_nothing():
tel = tmod.Telemetry("http://ledger.invalid", "")
tel.boot()
tel.auth_failed(code="token_invalid", http_status=401, caller="unknown")
assert tel.buffer == []
def test_unknown_code_is_never_sent_as_a_refusal():
tel = _tel()
tel.auth_failed(code="made_up_code", http_status=401, caller="unknown")
assert tel.buffer == []
def test_boot_omits_an_unknown_commit_rather_than_inventing_one():
tel = tmod.Telemetry("http://x", "tok", commit_sha=None, version="0.1.0")
tel.boot()
assert "commit_sha" not in tel.buffer[0]["metadata"]
def test_caller_classes_are_the_declared_three():
assert tmod.caller_class({}) == "unknown"
assert tmod.caller_class({"x-service-token": "s"}) == "unknown"
assert (
tmod.caller_class({"authorization": "Bearer eyJhbGciOiJSUzI1NiJ9.e30.x"})
== "anonymous_human"
)
@pytest.mark.asyncio
async def test_synthetic_header_marks_the_row_and_absent_means_real():
async def refusal(headers):
tel = _tel()
await _get(_app(tel), "/api/v1/repos/x/grants", headers)
return [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"][0]["metadata"]["synthetic"]
assert await refusal({"X-Windy-Synthetic": "1"}) is True
assert await refusal({}) is False
def test_synthetic_is_forwarded_downstream_only_for_synthetic_requests():
token = tmod.SYNTHETIC.set(True)
try:
assert tmod.synthetic_headers() == {"X-Windy-Synthetic": "1"}
finally:
tmod.SYNTHETIC.reset(token)
assert tmod.synthetic_headers() == {}
# ---- UPDATE 7: the ledger answers 202 even when it quarantines rows ----------
@pytest.mark.asyncio
async def test_quarantined_rows_are_warned_and_counted_on_the_next_heartbeat(monkeypatch, caplog):
tel = _tel()
tel.boot()
monkeypatch.setattr(
tel, "_post", lambda b: (202, {"accepted": 0, "quarantined": 1, "rejections": ["undeclared key"]})
)
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
await tel.flush()
assert tel.buffer == [] # sent; the ledger dead-lettered it, retrying won't help
assert "QUARANTINED" in caplog.text and "undeclared key" in caplog.text
tel.health()
assert tel.buffer[-1]["metadata"]["telemetry_quarantined"] == 1
assert tel.health_row()["telemetry_quarantined"] == 0 # reset per heartbeat window
@pytest.mark.asyncio
async def test_clean_send_reports_zero_and_logs_nothing(monkeypatch, caplog):
tel = _tel()
tel.boot()
monkeypatch.setattr(tel, "_post", lambda b: (202, {"accepted": 1, "quarantined": 0, "rejections": []}))
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
await tel.flush()
assert caplog.text == ""
row = tel.health_row()
assert row["telemetry_quarantined"] == 0 and row["telemetry_dropped"] == 0
def test_buffer_overflow_is_counted_as_dropped(monkeypatch):
monkeypatch.setattr(tmod, "MAX_BUFFER", 3)
tel = _tel()
for _ in range(5):
tel.boot()
assert len(tel.buffer) == 3
assert tel.health_row()["telemetry_dropped"] == 2

View File

@@ -0,0 +1,106 @@
"""G3.5 — the Eternitas webhook receiver, driven over HTTP (audit 2026-08-13).
The G3.5 invariants in test_invariants.py grep webhooks.py for strings; a
refactor that kept the strings and broke the behaviour would pass them all.
These send real requests through the real route (no DB: every case here stops
before the revocation handler) and assert what the receiver DOES.
"""
from __future__ import annotations
import hashlib
import hmac
import json
import httpx
import pytest
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from api.app.config import Settings
from api.app.errors import RepairPointer
from api.app.routes import webhooks
SECRET = "s" * 64
URL = "/api/v1/webhooks/eternitas"
def _app(secret: str = SECRET) -> FastAPI:
app = FastAPI()
app.include_router(webhooks.router)
app.state.settings = Settings(eternitas_webhook_secret=secret)
@app.exception_handler(RepairPointer)
async def _h(_, exc: RepairPointer) -> JSONResponse:
return JSONResponse(status_code=exc.status_code, content=exc.detail)
return app
async def _post(body: bytes, headers: dict, secret: str = SECRET) -> httpx.Response:
transport = httpx.ASGITransport(app=_app(secret))
async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
return await c.post(
URL, content=body, headers={"content-type": "application/json", **headers}
)
def _sig(raw: bytes, secret: str = SECRET) -> str:
return hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest()
# Deliberately odd spacing/key order: a receiver that re-serialises before
# hashing produces a different digest and must fail.
RAW = b'{"event":"windygit.selftest", "data": {"b": 2, "a": 1}}'
EVENT = {"x-eternitas-event": "windygit.selftest"}
@pytest.mark.asyncio
async def test_prefixed_and_bare_digests_are_both_accepted():
for header in (f"sha256={_sig(RAW)}", _sig(RAW)):
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
assert r.status_code == 200, r.text
assert r.json()["acted"] is False # unknown event: received, nothing done
@pytest.mark.asyncio
async def test_digest_of_reserialised_json_is_refused():
reserialised = json.dumps(json.loads(RAW)).encode()
assert reserialised != RAW
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(reserialised)}"})
assert r.status_code == 401 and r.json()["code"] == "webhook_signature_invalid"
@pytest.mark.asyncio
async def test_forged_or_wrong_key_signature_is_refused():
for header in ("sha256=" + "0" * 64, f"sha256={_sig(RAW, 'other-secret')}", "garbage"):
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
assert r.status_code == 401, header
@pytest.mark.asyncio
async def test_signed_event_without_signature_is_refused():
r = await _post(RAW, EVENT)
assert r.status_code == 401
@pytest.mark.asyncio
async def test_unset_secret_refuses_rather_than_accepts():
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(RAW)}"}, secret="")
assert r.status_code == 503 and r.json()["code"] == "webhook_secret_unset"
@pytest.mark.asyncio
async def test_revocation_with_bad_signature_never_reaches_the_handler():
body = b'{"event":"passport.revoked","passport":"ET26-TEST-GOOD"}'
r = await _post(
body,
{"x-eternitas-event": "passport.revoked", "x-eternitas-signature": "sha256=" + "f" * 64},
)
assert r.status_code == 401 # refused before any DB work
@pytest.mark.asyncio
async def test_reachability_ping_acknowledges_but_never_acts():
r = await _post(b'{"anything": "at all"}', {"x-eternitas-event": "platform.test_ping"})
assert r.status_code == 200 and r.json()["acted"] is False

View File

@@ -0,0 +1,40 @@
# Compute guard allow-list: code that MAY talk to an AI provider directly.
# Windy Mind is the ONLY door to AI compute (Grant, 2026-09-23). Every entry
# here is an exception to that rule and MUST say why. Paths are fnmatch globs
# relative to the repo root. Owner of this file: Windy Git lane (13); changes
# go through the orchestrator. Source of the first entries: COMPUTE_BYPASS_AUDIT.md.
allow:
- repo: windy-mind
paths: ["*"]
reason: "Windy Mind IS the door: provider clients belong here by definition."
- repo: windy-agent
paths: ["*"]
reason: >-
User BYOK: self-hosted agents call providers on the USER's own keys.
Mind stays opt-in there, or every self-hosted user's inference lands on
Grant's bill (no-cloud-cost-liability rule; audit #7).
- repo: windy-code
paths: ["extensions/windy-ai/*"]
reason: "User BYOK AI extension: the user's own provider keys; Mind is one opt-in provider (audit #8)."
- repo: windy-connect
paths: ["*writers/*"]
reason: "Writes client configs that NAME the user's own provider env vars; makes no provider calls (audit #11)."
- repo: windy-pro
paths: ["src/client/desktop/*"]
reason: >-
User BYOK desktop client: cloud STT/translate keys come from what the USER
enters (renderer localStorage -> electron-store; env var only for dev), and
the CSP line allows exactly those user-keyed hosts (audit #10). The
account-server is NOT covered: server-side calls go through Mind.
- repo: windy-pro
paths: ["src/client/web/src/pages/panels/MindPanel.jsx"]
reason: "Validates the USER's own OpenRouter key for BYOK (audit #10); spends no house money."
- repo: windy-git
paths: ["scripts/compute_guard.py", "ci/compute-guard-allow.yml"]
reason: "The guard's own pattern list and this file."

28
deploy/branding/README.md Normal file
View File

@@ -0,0 +1,28 @@
# Windy Git branding (G2.3)
Gitea's **supported** customisation surface: custom templates and public assets.
No Gitea source is modified, so upstream upgrades keep arriving (D-2, I-1).
deploy/branding/ → $GITEA_CUSTOM (/data/gitea) in the container
→ /srv/windygit/git/gitea/ on Veron 1
Apply with `./deploy/branding/apply.sh`.
## Two traps this cost, both worth knowing
**1. `GITEA__DEFAULT__APP_NAME` does not work.** Gitea reads `APP_NAME` from the
*top level* of `app.ini` (before any `[section]`). The env var instead created a
literal `[default]` section, which Gitea ignores — and the installer's stock
`APP_NAME` kept winning, so the site said "Gitea: Git with a cup of tea" while
the config looked correct. Worse, the env-to-ini pass **appended** a second
`APP_NAME` rather than replacing the first. `apply.sh` sets it at the top level
directly.
**2. Cloudflare caches `/assets/*` for 6 hours and no token in this stack can
purge.** Editing a fixed filename leaves the old bytes live for hours — the new
logo and CSS were both invisible while being correct at origin. **Version the
filename** (`theme-windy.v2.css`) on every brand change; a `?query` is not
enough because some caches ignore it.
The nav logo is swapped in CSS rather than by overriding Gitea's navbar
template — a one-line rule instead of a forked template that would drift.

23
deploy/branding/apply.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Apply Windy Git branding to the Gitea custom tree. Idempotent.
set -euo pipefail
CUSTOM="${GITEA_CUSTOM_HOST:-/srv/windygit/git/gitea}"
HERE="$(cd "$(dirname "$0")" && pwd)"
sudo mkdir -p "$CUSTOM"/templates/custom "$CUSTOM"/public/assets/img "$CUSTOM"/public/assets/css
sudo cp -r "$HERE"/templates/. "$CUSTOM"/templates/
sudo cp -r "$HERE"/public/. "$CUSTOM"/public/
sudo chown -R 1000:1000 "$CUSTOM"/templates "$CUSTOM"/public
# APP_NAME must sit at the TOP LEVEL. See README trap #1.
INI="$CUSTOM/conf/app.ini"
sudo cp "$INI" "$INI.bak-brand-$(date +%s)"
sudo python3 - "$INI" <<'PY'
import sys
p = sys.argv[1]
lines = [l for l in open(p).read().splitlines()
if not l.strip().startswith(("APP_NAME", "APP_SLOGAN"))]
lines.insert(0, "APP_NAME = Windy Git")
open(p, "w").write("\n".join(lines) + "\n")
PY
echo "applied. restart gitea to pick it up."

View File

@@ -0,0 +1,27 @@
/* Windy Git brand accent. Layered on top of Gitea's theme rather than
replacing it, so upstream theme fixes keep arriving. */
:root {
--color-primary: #0ea5e9;
--color-primary-dark-1: #0284c7;
--color-primary-dark-2: #0369a1;
--color-primary-light-1: #38bdf8;
--color-primary-light-2: #7dd3fc;
}
.wg-hero { max-width: 780px; margin: 4rem auto 2rem; padding: 0 1.5rem; text-align: center; }
.wg-hero h1 { font-size: 2.6rem; margin: 1.2rem 0 .4rem; letter-spacing: -.02em; }
.wg-hero .wg-sub { font-size: 1.15rem; opacity: .78; margin-bottom: 2.2rem; }
.wg-grid { display: grid; gap: 1.1rem; grid-template-columns: repeat(auto-fit,minmax(230px,1fr));
max-width: 900px; margin: 0 auto 2.5rem; padding: 0 1.5rem; text-align: left; }
.wg-card { border: 1px solid var(--color-secondary); border-radius: 8px; padding: 1.1rem 1.2rem; }
.wg-card h3 { margin: 0 0 .35rem; font-size: 1.02rem; }
.wg-card p { margin: 0; opacity: .74; font-size: .9rem; line-height: 1.5; }
.wg-cta { margin-bottom: 3rem; }
/* Logo swap via CSS.
Cloudflare cached the stock /assets/img/logo.svg for 6h and no available API
token can purge. Pointing at a NEW filename sidesteps the stale object
without touching Gitea's own templates — the supported customisation surface,
per D-2 (membrane, not merge). */
img[src$="/assets/img/logo.svg"] {
content: url("/assets/img/wg-mark.svg");
}

View File

@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" width="32" height="32">
<defs><linearGradient id="w" x1="0" y1="0" x2="1" y2="1">
<stop offset="0%" stop-color="#38bdf8"/><stop offset="100%" stop-color="#0ea5e9"/>
</linearGradient></defs>
<circle cx="16" cy="16" r="15" fill="#0b1220"/>
<path d="M5 11h13a3.2 3.2 0 1 0-3.1-4" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
<path d="M5 16h17a3.6 3.6 0 1 1-3.5 4.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
<path d="M5 21h9a2.8 2.8 0 1 1-2.7 3.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
</svg>

After

Width:  |  Height:  |  Size: 660 B

View File

@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" width="32" height="32">
<defs><linearGradient id="w" x1="0" y1="0" x2="1" y2="1">
<stop offset="0%" stop-color="#38bdf8"/><stop offset="100%" stop-color="#0ea5e9"/>
</linearGradient></defs>
<circle cx="16" cy="16" r="15" fill="#0b1220"/>
<path d="M5 11h13a3.2 3.2 0 1 0-3.1-4" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
<path d="M5 16h17a3.6 3.6 0 1 1-3.5 4.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
<path d="M5 21h9a2.8 2.8 0 1 1-2.7 3.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
</svg>

After

Width:  |  Height:  |  Size: 660 B

View File

@@ -0,0 +1,5 @@
{{/* Windy Git brand layer.
The filename carries a version: Cloudflare caches /assets/* for 6h with no
purge token available to this stack, so editing a fixed filename leaves the
old bytes live for hours. Bump the suffix on every brand change. */}}
<link rel="stylesheet" href="{{AssetUrlPrefix}}/css/theme-windy.v2.css">

View File

@@ -0,0 +1,27 @@
{{template "base/head" .}}
<div role="main" aria-label="{{ctx.Locale.Tr "home"}}" class="page-content home">
<div class="wg-hero">
<img src="{{AssetUrlPrefix}}/img/logo.svg" alt="Windy Git" width="72" height="72">
<h1>Windy Git</h1>
<p class="wg-sub">Your work, every version — and agents as citizens.</p>
<div class="wg-cta">
{{if not .IsSigned}}
<a class="ui primary button" href="{{AppSubUrl}}/user/login">Sign in with Windy</a>
{{else}}
<a class="ui primary button" href="{{AppSubUrl}}/{{.SignedUser.Name}}?tab=repositories">Your repositories</a>
{{end}}
<a class="ui button" href="{{AppSubUrl}}/explore/repos">Explore</a>
</div>
</div>
<div class="wg-grid">
<div class="wg-card"><h3>Code and models, one host</h3>
<p>Git and LFS over Windy Cloud storage. Source, weights and adapters live side by side.</p></div>
<div class="wg-card"><h3>Agents are first-class</h3>
<p>An agent signs in with its own Eternitas passport — not a human's borrowed token — and its work is attributable to it.</p></div>
<div class="wg-card"><h3>Your account, no second password</h3>
<p>Sign in with the Windy account you already have. Nothing new to remember.</p></div>
<div class="wg-card"><h3>Kept, and kept elsewhere</h3>
<p>Every repository is bundled nightly to off-site storage, and the restore is rehearsed rather than assumed.</p></div>
</div>
</div>
{{template "base/footer" .}}

View File

@@ -11,8 +11,9 @@ log:
runner:
file: /data/.runner
capacity: 4 # concurrent jobs; Veron has 24 cores, dind is capped at 12
timeout: 30m
capacity: 1 # per runner; parallelism = number of runner services (6). See docker-compose.yml
timeout: 90m # hard ceiling per job. eternitas's serial pytest is ~50 min; keep
# timeout-minutes in each workflow — a hang still reads as a hang
shutdown_timeout: 3m
insecure: false
fetch_timeout: 5s
@@ -53,6 +54,9 @@ container:
privileged: false
options:
workdir_parent: /workspace
valid_volumes: [] # a job cannot bind-mount anything from the daemon host
# A job may bind-mount exactly ONE daemon path: the read-only windy-pro build
# inputs (mounted :ro into dind itself). Per-runner, not per-repo (act_runner
# limit): any windyadmin repo could mount it; it is non-secret and read-only.
valid_volumes: ["/ci-inputs/windy-pro"]
docker_host: "-" # do NOT expose the runner's own docker socket to jobs
force_pull: false

View File

@@ -0,0 +1,15 @@
# ROLLBACK ONLY: the pre-Sysbox dind (privileged: true), kept one command away.
# Use it if CI breaks under Sysbox:
#
# cd /srv/windygit/src/deploy/runner
# sudo docker compose -f docker-compose.yml -f docker-compose.privileged.yml up -d dind
#
# (then restart the runners while idle). Compose merges `volumes` by container
# path, so this puts back the old `dind-storage` volume with its image cache.
# Going forward again: the same command without the second -f.
services:
dind:
runtime: runc
privileged: true
volumes:
- dind-storage:/var/lib/docker

View File

@@ -26,8 +26,12 @@
# * `dind` and every job container it spawns are UNTRUSTED. They are on a
# private network with no access to the forge, its database, or its .env.
#
# dind itself is privileged — that is the cost, and it is the reason a job
# escape lands in a disposable daemon rather than on Grant's workstation.
# dind is NOT privileged (2026-09-23): it runs under the Sysbox runtime
# (sysbox-ce on Veron, `runtime: sysbox-runc`), a user-namespaced system
# container whose root is an unprivileged host uid. A job that escapes its own
# container lands in dind as a nobody on the host, not as root on Grant's
# workstation. Before Sysbox, dind was `privileged: true`; that config is kept
# as docker-compose.privileged.yml (ROLLBACK ONLY, one command, see that file).
#
# ⚠️ Do NOT "simplify" this by mounting the host docker socket.
@@ -36,21 +40,49 @@ name: windy-git-runner
services:
dind:
image: docker.io/library/docker:27-dind
privileged: true
runtime: sysbox-runc # NOT privileged: see the I-5 note above
environment:
DOCKER_TLS_CERTDIR: "" # plain TCP on an isolated network, no host route
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--tls=false"]
networks: [jobs]
volumes:
- dind-storage:/var/lib/docker
# A fresh volume: Sysbox shifts ownership to its own uid range. The old
# `dind-storage` is kept untouched for the privileged rollback.
- dind-storage-sysbox:/var/lib/docker
# READ-ONLY, non-secret build inputs for windy-pro's desktop jobs (models,
# linux-x64 portable bundle, enter-monitor build), copied from the frozen
# release clone by deploy/runner/refresh-ci-inputs.sh. Jobs may mount ONLY
# this path (config.yaml valid_volumes). Orchestrator-approved 09-23.
- /home/user1-gpu/ci-inputs/windy-pro:/ci-inputs/windy-pro:ro
# G1.5 — bounded so a fork-bomb workflow cannot starve Grant's interactive
# session. Veron 1 is his workstation, not a dedicated build box.
cpus: 12.0 # 12 of 24 cores
mem_limit: 64g
restart: unless-stopped
runner:
image: docker.io/gitea/act_runner:0.2.11
# ── FOUR runners × capacity 1, not one runner × capacity 4 (2026-09-23) ──
#
# act caches every action repo at /root/.cache/act/<hash> INSIDE the runner
# process and re-fetches it at the start of each job. With capacity 4, four
# concurrent jobs share that one directory: one job's refresh rewrites it while
# another is tarring it into its job container, and the job dies with
# `lstat /root/.cache/act/<hash>/…: no such file or directory` on
# `actions/setup-node` / `setup-uv` — a failure that reads like a broken
# workflow. windy-chat (~20 jobs per push) hit it on 3 jobs in its first run.
# `rm -rf /root/.cache/act` only reset the clock. Separate processes get
# separate caches, so the race cannot occur. Same total parallelism, same
# single capped dind — the blast radius is unchanged.
runner: &runner
# 0.2.11 -> 0.6.1 on 2026-08-14. The bundled act in 0.2.11 only knows
# `runs.using: node12|node16|node20`, so ANY repo pinning a current action
# major dies before its first step with "The runs.using key in action.yml
# must be one of: [...], got node24" — Windy-Clone on actions/checkout@v5
# is how this surfaced. Verified: `node24` is absent from the 0.2.11 binary
# and present in 0.6.1. Every key in this directory's config.yaml still
# exists in 0.6.1's schema (0.6.1 only ADDS keys), so the config carries
# over unchanged. Rollback is re-pinning 0.2.11; the registration in the
# runner-data volume survives either way.
image: docker.io/gitea/act_runner:0.6.1
depends_on: [dind]
environment:
# The runner reaches its OWN daemon. Never the host's.
@@ -90,6 +122,47 @@ services:
mem_limit: 4g
restart: unless-stopped
# Each extra runner registers itself on first start (own name, own volume —
# the registration lives in /data/.runner, so volumes must never be shared).
runner-2:
<<: *runner
environment: &env2
DOCKER_HOST: tcp://dind:2375
GITEA_INSTANCE_URL: https://app.windygit.com
GITEA_RUNNER_REGISTRATION_TOKEN: ${RUNNER_TOKEN:?set RUNNER_TOKEN}
GITEA_RUNNER_NAME: veron-1-2
CONFIG_FILE: /config.yaml
volumes: [./config.yaml:/config.yaml:ro, runner-data-2:/data]
runner-3:
<<: *runner
environment:
<<: *env2
GITEA_RUNNER_NAME: veron-1-3
volumes: [./config.yaml:/config.yaml:ro, runner-data-3:/data]
runner-4:
<<: *runner
environment:
<<: *env2
GITEA_RUNNER_NAME: veron-1-4
volumes: [./config.yaml:/config.yaml:ro, runner-data-4:/data]
# 5 and 6 added the same day: with ~11 private repos onboarded (windy-chat
# alone queues ~24 jobs per push) four runners left 50+ jobs waiting. The
# CPU ceiling is dind's (12 of 24 cores, G1.5), not the runner count, so more
# runners add concurrency for I/O-bound jobs (npm ci, uv sync) without
# taking more of Grant's workstation.
runner-5:
<<: *runner
environment:
<<: *env2
GITEA_RUNNER_NAME: veron-1-5
volumes: [./config.yaml:/config.yaml:ro, runner-data-5:/data]
runner-6:
<<: *runner
environment:
<<: *env2
GITEA_RUNNER_NAME: veron-1-6
volumes: [./config.yaml:/config.yaml:ro, runner-data-6:/data]
networks:
jobs:
# Untrusted job containers live here. No route to the forge.
@@ -97,5 +170,11 @@ networks:
volumes:
dind-storage:
dind-storage-sysbox:
runner-data:
runner-data-2:
runner-data-3:
runner-data-4:
runner-data-5:
runner-data-6:

51
deploy/runner/egress.sh Executable file
View File

@@ -0,0 +1,51 @@
#!/usr/bin/env bash
# CI egress filter (2026-09-23) — jobs reach the internet, never Grant's network.
#
# Measured before this existed: an ordinary (unprivileged) job container inside
# the CI dind could open SSH, Ollama, and every dev server on Veron
# (192.168.1.73:22/3000/3300/8080/11434) and anything else on the LAN, WireGuard
# or Tailscale. No container escape needed — a malicious npm/pip dependency in
# any first-party repo's CI could walk straight onto the fleet.
#
# All CI traffic leaves through the `windy-git-runner_jobs` bridge (dind NATs
# its job containers onto it). This script, run at boot and after any runner
# compose change, allows on that bridge:
# * traffic between the runners and dind (same bridge)
# * replies (ESTABLISHED/RELATED)
# * DNS (53) — Docker's embedded resolver forwards to the LAN router
# * everything public
# and drops: RFC1918, CGNAT/Tailscale (100.64/10), link-local, and ANY packet
# addressed to the host itself (INPUT), whatever interface IP it targets.
# Idempotent: owned chains are flushed and rebuilt; hooks are added once.
set -euo pipefail
NET=windy-git-runner_jobs
id=$(docker network inspect "$NET" --format '{{.Id}}')
BR="br-${id:0:12}"
ip link show "$BR" >/dev/null
iptables -N WG-CI-EGRESS 2>/dev/null || iptables -F WG-CI-EGRESS
iptables -A WG-CI-EGRESS -o "$BR" -j RETURN
iptables -A WG-CI-EGRESS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
iptables -A WG-CI-EGRESS -p udp --dport 53 -j RETURN
iptables -A WG-CI-EGRESS -p tcp --dport 53 -j RETURN
for cidr in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16; do
iptables -A WG-CI-EGRESS -d "$cidr" -j DROP
done
iptables -A WG-CI-EGRESS -j RETURN
iptables -N WG-CI-INPUT 2>/dev/null || iptables -F WG-CI-INPUT
iptables -A WG-CI-INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
iptables -A WG-CI-INPUT -j DROP
# Hooks: remove any stale ones (the bridge name changes if the network is
# recreated), then add exactly one of each at the top.
for chain in DOCKER-USER INPUT; do
target=$([ "$chain" = INPUT ] && echo WG-CI-INPUT || echo WG-CI-EGRESS)
while read -r rule; do
iptables -D $chain ${rule#-A $chain }
done < <(iptables -S "$chain" | grep -- "-j $target" || true)
iptables -I "$chain" 1 -i "$BR" -j "$target"
done
echo "ci egress filter active on $BR ($NET)"

28
deploy/runner/prune.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Keep CI storage bounded (2026-09-23).
#
# Kit 0's 09-01 wipe began with CI `_work` dirs (74 GB) + Docker filling the
# disk. Windy Git's runners have no host `_work` dir — every job runs in a
# container inside the CI-only dind — so the thing that grows here is dind's
# image/volume store (38 GB when this was written, never pruned). This prunes
# ONLY that daemon, over its own socket. It never touches the host's Docker.
#
# In-use images/volumes are never removed, so a running job is safe.
set -euo pipefail
CAP_GB="${CI_STORAGE_CAP_GB:-60}"
D=(docker exec windy-git-runner-dind-1 docker -H tcp://127.0.0.1:2375) # dind listens on TCP only
"${D[@]}" container prune -f --filter until=6h >/dev/null
"${D[@]}" volume prune -af >/dev/null # job workspaces of finished jobs
"${D[@]}" image prune -af --filter until=168h >/dev/null
"${D[@]}" builder prune -af --filter until=168h >/dev/null 2>&1 || true
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
if (( used_gb > CAP_GB )); then
# Over the cap even after the age-based pass: drop every unused image. The
# next jobs re-pull (the act image is ~2 GB) — slower, never wrong.
"${D[@]}" image prune -af >/dev/null
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
fi
echo "ci storage ${used_gb}G (cap ${CAP_GB}G)"
(( used_gb <= CAP_GB )) || { echo "STILL OVER CAP"; exit 1; }

View File

@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Refresh the READ-ONLY CI input cache for windy-pro desktop jobs from the FROZEN
# release clone on Veron. Reads ~/windy-pro-release only; never writes to it.
# Run when the release lane says engines / wheels / the portable bundle changed.
# Linux inputs only: 3 models + requirements-bundle.txt, bundled-portable/linux-x64,
# native/enter-monitor/build. Result is chmod a-w and mounted :ro into dind.
set -euo pipefail
SRC=/home/user1-gpu/windy-pro-release
DST=/home/user1-gpu/ci-inputs/windy-pro
models=$(ls "$SRC/extraResources/model" | grep -E '^windy-(nano|lite|core)-ct2$')
[ "$(wc -w <<<"$models")" = 3 ] || { echo "expected 3 models, got: $models"; exit 1; }
mkdir -p "$DST/extraResources/model" "$DST/bundled-portable" "$DST/native-enter-monitor-build"
chmod -R u+w "$DST"
R="ionice -c3 nice -n 19 rsync -a --delete"
for m in $models; do $R "$SRC/extraResources/model/$m/" "$DST/extraResources/model/$m/"; done
$R "$SRC/extraResources/requirements-bundle.txt" "$DST/extraResources/requirements-bundle.txt"
$R "$SRC/bundled-portable/linux-x64/" "$DST/bundled-portable/linux-x64/"
$R "$SRC/native/enter-monitor/build/" "$DST/native-enter-monitor-build/"
date -u +%FT%TZ > "$DST/.refreshed-from-windy-pro-release"
chmod -R a-w "$DST"
du -sh --apparent-size "$DST"

View File

@@ -0,0 +1,13 @@
[Unit]
Description=Windy Git - CI egress filter (jobs reach the internet, never the LAN/host)
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
# The jobs network exists once the runner compose project is up; retry until it does.
ExecStart=/bin/bash -c 'for i in $(seq 1 60); do /srv/windygit/src/deploy/runner/egress.sh && exit 0; sleep 5; done; exit 1'
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,6 @@
[Unit]
Description=Windy Git - prune CI-only dind storage (bounded, never the host daemon)
[Service]
Type=oneshot
ExecStart=/srv/windygit/src/deploy/runner/prune.sh

View File

@@ -0,0 +1,9 @@
[Unit]
Description=Windy Git - prune CI storage every 6 hours
[Timer]
OnCalendar=*-*-* 00/6:37:00
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,13 @@
[Unit]
Description=Windy Git nightly backup (git bundles + windgit schema -> R2)
After=network-online.target docker.service
[Service]
Type=oneshot
WorkingDirectory=/srv/windygit/src
# The .env holds the R2 credentials. The script refuses to run without them
# rather than reporting a backup that did not happen.
EnvironmentFile=/srv/windygit/src/.env
ExecStart=/bin/bash /srv/windygit/src/scripts/backup.sh
Nice=10
IOSchedulingClass=idle

View File

@@ -0,0 +1,3 @@
[Service]
ExecStart=
ExecStart=/usr/local/bin/windy-job windygit-backup 26h --expect "ok — [0-9]+ repos" --owner 13 -- /bin/bash /srv/windygit/src/scripts/backup.sh

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Nightly Windy Git backup
[Timer]
OnCalendar=*-*-* 04:17:00
# Grant's workstation is not always on at 04:17. Without this a missed window
# is simply skipped and the backup silently never runs.
Persistent=true
RandomizedDelaySec=600
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,3 @@
[Service]
ExecStart=
ExecStart=/usr/local/bin/windy-job windygit-ci-prune 7h --expect "ci storage [0-9]+G" --owner 13 -- /srv/windygit/src/deploy/runner/prune.sh

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Sync GitHub -> Windy Git (Phase 1: GitHub is the source of truth)
After=network-online.target docker.service
[Service]
Type=oneshot
WorkingDirectory=/srv/windygit/src
EnvironmentFile=/srv/windygit/src/.env
# GITHUB_TOKEN is set on the host only (root-only unit file / .env) — NEVER commit it.
Environment=GITHUB_OWNER=sneakyfree
ExecStart=/bin/bash /srv/windygit/src/scripts/sync_from_github.sh
Nice=10

View File

@@ -0,0 +1,3 @@
[Service]
ExecStart=
ExecStart=/usr/local/bin/windy-job windygit-sync 20m --expect "all repos in step with GitHub" --owner 13 -- /bin/bash /srv/windygit/src/scripts/sync_from_github.sh

View File

@@ -0,0 +1,10 @@
[Unit]
Description=Keep Windy Git in step with GitHub every 5 minutes
[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,16 @@
[Unit]
Description=Windy Git - Cloudflare Tunnel (the only ingress; no inbound port is opened)
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run
Restart=always
RestartSec=5
# G1.4 - bounded, so a misbehaving ingress can never starve Grant's workstation.
MemoryMax=512M
CPUQuota=100%
[Install]
WantedBy=multi-user.target

View File

@@ -16,7 +16,11 @@ services:
# I-12: baked at build time. A runtime COMMIT_SHA override is ignored.
COMMIT_SHA: ${COMMIT_SHA_BUILD:-}
BUILT_AT: ${BUILT_AT:-}
env_file: [.env]
env_file:
- .env
# WINDYGIT_TELEMETRY_TOKEN (root-only on Veron). Optional: no file = no telemetry.
- path: /etc/windygit/telemetry.env
required: false
environment:
DATABASE_URL: postgresql+asyncpg://windygit:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@db:5432/windygit
GITEA_BASE_URL: http://gitea:3000
@@ -57,12 +61,32 @@ services:
# G2.2 — OIDC only. No local password login, no self-registration.
GITEA__service__DISABLE_REGISTRATION: "true"
GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "true"
# SSO #8 (2026-09-23): the password and passkey forms are OFF. windyadmin
# is site admin with a local password; leaving the form up made that
# password a second, phishable way into the whole forge. Break-glass is
# the CLI on Veron (`docker exec -u git windy-git-gitea-1 gitea admin ...`).
GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
GITEA__service__ENABLE_PASSKEY_AUTHENTICATION: "false"
# G3.1 — a Windy account IS the account. Signing in with Windy provisions
# the Gitea user on first arrival; nobody is asked to invent a second
# identity for the same person, and no local password ever exists.
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "true"
# 🔴 OFF (2026-09-23). With it on, ANY stranger with a Windy Word account
# (public signup, not even email-verified) got a forge account on first
# sign-in — and the CI runners were instance-wide, so their workflows
# would run on Veron beside the R2 god token. Proven with a throwaway
# account, then closed. Opening the forge to non-Grant users is a §7
# Grant decision; until then new accounts are created deliberately.
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "false"
GITEA__oauth2_client__USERNAME: email
GITEA__oauth2_client__ACCOUNT_LINKING: auto
# 🔴 `login`, NOT `auto` (SSO #8). `auto` linked any hub login whose EMAIL
# matched an existing account — and windyadmin (SITE ADMIN) carries Grant's
# email, so the forge's admin rights rested on the hub never letting anyone
# else hold that address. `login` makes an email match prove possession of
# the existing account first. Grant is unaffected: his account is already
# linked by the hub's stable `sub`, which is matched before email.
# ⚠️ env-to-ini SETS but never UNSETS — this value must also be edited in
# /srv/windygit/git/gitea/conf/app.ini if it is ever removed from here.
GITEA__oauth2_client__ACCOUNT_LINKING: login
# The email is asserted by account-server, which is the authority on it.
# Asking the user to re-verify an address their identity provider already
# verified is friction that buys nothing.

View File

@@ -0,0 +1,116 @@
# Second-auditor review of the Windy Git build (Fable, 2026-08-13)
A fresh-eyes trace of Opus's build, **verified against the live system** rather
than against the transcript's own account. One critical finding was fixed during
the review; the rest are recorded here with proportion.
## Fixed during this review
### 🔴 Agent authentication was not authentication (CRITICAL, was live+public)
`auth.py` read the passport out of a bearer token **without verifying the EPT
signature**, asked Eternitas *"is this passport reputable?"*, and seated the
caller on a yes. That answers reputation, not possession.
**Proven by exploit:** a forged `alg:none` token naming a passport lifted from
the logs returned **HTTP 200** as that agent on the public API. Anyone who knows
a passport number (they appear in logs, the lockbox, and revocation messages)
could impersonate that agent.
The human path already failed closed for exactly this reason
(`require_verified_jwt`) — but the gate sat *after* the agent branch returned, so
it protected the safe path and skipped the exploitable one.
**Fix:** the agent path now fails closed in production, *before* the trust
lookup, mirroring the human path. Reopens automatically when the ES256/JWKS
verifier (G3.2/G9.1) is built. Re-tested live: forged token now `503`. Added a
**behavioral** test (forged token through real `get_caller` must raise) and a
**canary probe** (forged token must stay refused; a 2xx pages).
## Open findings (recorded, not yet fixed)
### 🟠 The EI throttle table is dead code (MEDIUM)
`BAND_MULTIPLIER` and `rate_*_per_day` (G3.4) are defined and **read by
nothing** — verified by grep. An authenticated agent has no rate limit at all.
When the agent path reopens with real verification, wire the throttle before it
does, or a single agent can hammer repo-create/token-mint unbounded.
### 🟠 The test suite is mostly source-string assertions (MEDIUM)
56 invariant tests carried **~86 "does the source contain this string"**
assertions and **zero** that exercised the auth decision. `make check` green
means the code still *contains* the patterns, not that it *behaves*. The auth
bypass passed every test. **Direction:** convert the top ~10 guards into
behavioral tests against an ephemeral instance (this review added the first
two). The live-curl proofs Opus ran were excellent but were never captured, so
they don't defend against regression.
### 🟠 Privileged dind sits beside broad-scoped tokens (MEDIUM — Grant-aware)
`dind privileged=true`, and the host `.env` holds the account-wide R2 token and
a GitHub PAT, on the same box running untrusted CI. Escape from privileged dind
→ host fs → both tokens. Grant waived *minting a new token* for the sandbox; the
specific escalation path (privileged-dind-next-to-god-token) is a separate
decision. Lowest-effort mitigations: rootless/sysbox runner, or move the two
tokens out of the API container's env into a path the API reads but the runner
host does not share.
### ✅ Revocation — was WORSE than flagged, now fixed (was CRITICAL once agent auth reopened)
Re-examined after real agent auth went live, and the finding grew teeth. A
revoked passport returns **HTTP 200, `status: revoked`, `band: unproven`,
`allowed_actions: []`** (verified live). `resolve_passport` keyed refusal only on
HTTP 4xx and `band=="untrusted"`, so it returned band `unproven` and **seated the
revoked agent** — revocation was not enforced on the live path at all. The
webhook everyone worried about was only ever cache-invalidation; the live trust
lookup was the real gate, and it wasn't checking.
**Fixed:** `decide_trust` now allows only `status=="active"`; revoked / suspended
/ frozen / unknown all refuse, fail-closed. Revocation now takes effect on the
next request, no webhook required. Eternitas additionally refuses to mint EPTs
for revoked bots, so the residual window was a pre-existing ~365-day token —
exactly what the live check now stops. 79 tests green including the full wiring.
The webhook (`webhook_secret` lost to the 500) is now genuinely LOW: it only
matters for locally-issued credentials/grants (G6.3, not built), and it is no
longer the thing standing between a revoked agent and access.
## What is genuinely strong (and worth protecting)
- **Honesty engineering is real:** fail-closed providers, `/health` refusing to
claim green it can't prove, I-12's baked-sha proven live against a hostile env
var. This directly cures the parent ecosystem's #1 root cause.
- **Incident response was first-rate:** the login outage was root-caused to a
510-deep accept queue and a per-query node fork, with the "one function, not
468 call sites" reframe that is correct and valuable.
- **It caught its own mistakes** — the mirror direction and the push-triggered
deploy workflows, the latter *before* they fired.
- **The DR posture is right:** 131 read-only mirrors (no CI, zero deploy risk) +
a rehearsed restore. Rehearsed restore is rare in this ecosystem; keep it.
## The one-line lesson
Opus aimed its considerable discipline at **honesty and documentation**
(excellent) more than at **adversarial correctness** — so the property that
mattered most, *is an agent really that agent*, shipped inverted and untested.
The remedy is not more process; it is **behavioral tests and canary probes for
the security-critical paths**, so verification persists instead of living in a
transcript.
## Disposition update — 2026-09-23 (lane 13)
**Privileged dind beside the tokens — materially reduced, not closed.**
- The account-wide R2 token is **gone from Veron**. `.env` now carries a token scoped
to Workers R2 Bucket Item Read/Write on `windy-git-lfs` + `windy-git-backups` only,
minted by API (verified: works on both buckets, refused on any other). A CI escape
now reaches Windy Git's own two buckets, not every bucket and zone in the account.
- Runners take jobs **only from windyadmin-owned repos** (`action_runner.owner_id`), and
forge self-registration is off, so no stranger's workflow can run here.
- A host egress filter (`deploy/runner/egress.sh`) stops job containers reaching Veron,
the LAN, WireGuard or Tailscale.
- Still open: dind runs `--privileged` (next: Sysbox); the host still holds a GitHub
token and the Gitea admin token.
**Revocation / webhook secret** — `ETERNITAS_WEBHOOK_SECRET` recovered from the
Eternitas platform row and set; signed deliveries verify.
**Tests are string asserts** — the security paths now have behavioural suites
(`test_hub_jwt.py`, `test_webhooks_behavior.py`, `test_pr_status_bridge.py`);
a mutation check showed the old grep invariant passing a broken HMAC prefix strip.

View File

@@ -87,6 +87,50 @@ Per repo, deliberately, when that repo is quiet:
4. later, when it flips to Windy-Git-first: remove it from `REPOS` *first*,
repoint its sessions, add a push-mirror back to GitHub
## Private repos: Windy Git IS their CI (permanent, 2026-09-23)
The platform repos stay **private** on GitHub (Grant, 2026-09-23), and private
repos cannot run GitHub Actions on this account at all. Windy Git is therefore
their CI permanently, not a stopgap:
GitHub push ──sync (15 min)──▶ Windy Git ──runner──▶ Veron 1
▲ │
└──── commit status windy-git/<workflow>/<job> ◀───┘ scripts/pr_status_bridge.py
- `pr_status_bridge.py` runs at the end of every sync. It opens a `[GH#N]`
mirror PR in Windy Git for every open **same-repo** GitHub PR (so
`pull_request` workflows fire), closes it when the GitHub PR closes, and posts
each job's result back to GitHub on PR heads and the default-branch head.
**Never merge a `[GH#N]` PR here** — merge on GitHub.
- Fork PRs are never run: their branch is never synced, and untrusted code
beside the privileged dind is the open audit finding.
- Covered repos: `BRIDGE_REPOS` in the script. Public repos are left out on
purpose; they run real GitHub Actions and two verdicts per commit is noise.
- `skipped` jobs post nothing — no green for a job nobody ran.
- **Image-build jobs** (name matches `docker`) post nothing: job containers
have no Docker daemon by design (I-5), so they are red on every commit. A
rootless builder (BuildKit rootless / buildx in the capped dind) is the open
decision that would bring them back.
**Onboarding another private repo** — the promotion steps below, then:
# on Veron 1, as root
set -a; . /srv/windygit/src/.env; set +a
python3 scripts/import_from_github.py <repo> # writable; aborts if the repo exists
# disable EVERY deploying workflow before anything is pushed:
curl -X PUT -H "Authorization: token $GITEA_ADMIN_TOKEN" \
http://localhost:3080/api/v1/repos/windyadmin/<repo>/actions/workflows/deploy.yml/disable
# add <repo> to REPOS in sync_from_github.sh AND BRIDGE_REPOS in pr_status_bridge.py
An import fires no push event, so `main` has no verdict until its next commit.
To get one now: force Windy Git's `main` back one commit, then
`systemctl start windygit-sync` — the sync pushes it forward and CI fires.
⚠️ **`/actions/tasks` lists only jobs a runner has PICKED UP.** Queued runs are
invisible there, so a repo can read "0 runs" while work is waiting. The truth is
`action_run` in the `gitea` database (status 1 success, 2 failure, 5 waiting,
6 running).
## ⚠️ Deploy workflows are DISABLED on Windy Git, deliberately
Six workflows fire on `push:` and deploy to production:

View File

@@ -15,6 +15,7 @@ Mirrored into `windy-cloud` and `eternitas` on change.
| eternitas | `GET /api/v1/trust/{passport}` | band + allowed_actions |
| eternitas | `GET /api/v1/registry/{passport}/integrity` | ⚠️ note the path — `windy-registry` calls `/api/v1/passports/{p}/status`, which 404s, which is why the integrity index has never been populated |
| account-server | OIDC discovery + JWKS | human identity (G3.1) |
| windy-admin ledger | `POST /v1/events` (admin.windyword.ai) | field telemetry: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed`. Shapes are declared with the ledger owner BEFORE shipping (the server quarantines undeclared keys). Codes, counts, route templates only |
| windy-cloud-sites | `POST /api/v1/sites/{id}/versions` | publish docs from a repo |
## Calls IN

View File

@@ -1,6 +1,10 @@
# RUNBOOK — Windy Git on Veron 1 (rung R0)
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron`. Passwordless sudo.
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron` (or `ts-veron`). Passwordless sudo.
**Checkouts (one-repo doctrine):** the ONE standing dev checkout is **OC5
`~/windy-git`** (platform repos live on OC5). `/srv/windygit/src` on Veron is the
*deploy* copy — it holds no local work. Nothing else should exist.
⛔ **Kit 0 is never a host for this service** (D-4). `api/app/main.py` refuses to
boot in production if it finds itself on `72.60.118.54`.
@@ -15,6 +19,9 @@ boot in production if it finds itself on `72.60.118.54`.
| `/etc/cloudflared/config.yml` | tunnel ingress |
| `/etc/cloudflared/windy-git.json` | tunnel credentials, mode 600 |
| `/srv/windygit/src/.env` | secrets, mode 600, **never committed** |
| `/srv/windygit/git/gitea/conf/app.ini` | Gitea's persisted config — env-to-ini SETS but never UNSETS; edit here when removing a `GITEA__*` var |
| `/srv/windygit/sync/*.git` | bare staging copies the GitHub→Windy Git sync pushes from |
| `/srv/windygit/src/deploy/runner/.env` | `RUNNER_TOKEN` — a **windyadmin user-level** registration token (not instance-level; see CI) |
## Ports — all loopback, on purpose
@@ -22,7 +29,7 @@ boot in production if it finds itself on `72.60.118.54`.
|---|---|
| `127.0.0.1:3080` | Gitea (host 3000 is a resident node dev server; 3300 is nginx — **do not fight them for a port**) |
| `127.0.0.1:8600` | windy-git API |
| `127.0.0.1:2000` | cloudflared metrics |
| `127.0.0.1:2001` | cloudflared metrics (`metrics:` in `/etc/cloudflared/config.yml`) — **not 2000**, see Troubleshooting |
**No inbound port is opened.** cloudflared dials out, so the dynamic residential
IP is irrelevant and there is no firewall hole to maintain.
@@ -41,12 +48,26 @@ sudo systemctl status windygit-tunnel
```bash
ssh wg-veron
cd /srv/windygit/src && git pull
cd /srv/windygit/src && git fetch origin && git merge --ff-only origin/main # READ the output
export COMMIT_SHA_BUILD=$(git rev-parse HEAD) BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
sudo -E docker compose up -d --build
sudo -E docker compose up -d --build --no-deps api # API only: no forge restart
curl -s https://api.windygit.com/version # MUST equal git rev-parse HEAD
```
A Gitea config change (compose `GITEA__*`) needs `sudo docker compose up -d --no-deps gitea`
— a ~6 s forge outage; running CI jobs survive it. Check `app.ini` afterwards.
⚠️ **Never `git pull -q` in a deploy script.** `-q` hides *errors*, not just
noise. On 2026-08-14 a divergent branch made `pull -q` fail silently and the
"deploy" ran for 20 minutes against stale code while reporting success. Use
`git fetch && git merge --ff-only` (or `reset --hard origin/main` on THIS
checkout only, which holds no local work) and read the output.
⚠️ **Never force-push a branch a deploy checkout tracks.** An earlier
`git commit --amend` + `--force-with-lease` rewrote history `/srv/windygit/src`
was already sitting on, orphaning it. If you must amend, re-point the deploy
checkout in the same breath.
⚠️ **Never put `COMMIT_SHA` in `.env`.** It does nothing here — the sha is baked
into the image and a runtime override is ignored with a warning (I-12). That env
pin is the documented root cause of nine sibling services misreporting their
@@ -61,11 +82,54 @@ curl -sI https://app.windygit.com/ | head -1 # Gitea, 200
sudo ss -tlnp | grep -E "3080|8600" # both must be 127.0.0.1
```
## Timers (host systemd units — the sync timer is NOT in the repo)
| Unit | Cadence | Does |
|---|---|---|
| `windygit-sync.timer` | every 5 min (`OnUnitActiveSec`) | GitHub → Windy Git for `REPOS` in `scripts/sync_from_github.sh`, then `scripts/pr_status_bridge.py` (mirror PRs + GitHub commit statuses). A manual `systemctl start` RESETS the 5-min clock. |
| `windygit-backup.timer` | nightly | `git bundle` + pg_dump → R2, 30-day retention |
| `windygit-ci-prune.timer` | every 6 h | `deploy/runner/prune.sh` — CI dind storage, 60 GB cap |
| `windygit-tunnel.service` | always | the only ingress |
## CI (Gitea Actions) — see `docs/CUTOVER.md` for onboarding a repo
- **Six runners × capacity 1** (`deploy/runner/docker-compose.yml`), one shared
dind capped at 12 cores / 64 GB. Capacity >1 in one runner shares
`/root/.cache/act` between jobs and races (`lstat …: no such file`).
- **Runners are scoped to the `windyadmin` user** (`action_runner.owner_id=1`),
so only first-party repos run. A repo owned by anyone else — a plane-created
agent or `u-system` repo — gets NO runner. Re-registrations inherit this
because `RUNNER_TOKEN` is user-level.
- Job ceiling 90 min (`config.yaml` `runner.timeout`); a `config.yaml` change
needs each runner restarted **while idle** — `compose up -d` won't recreate it.
- `/actions/tasks` lists only PICKED-UP jobs. Queue truth is `action_run_job`
in the `gitea` DB: `sudo docker exec -i windy-git-db-1 psql -U windygit -d gitea`
(status 1 ok · 2 fail · 3 cancelled · 4 skipped · 5 waiting · 6 running).
- Job logs are in R2, not on disk. `GET /api/v1/repos/{o}/{r}/actions/jobs/{JOB_ID}/logs`
takes the `action_run_job` id, not the task id.
## Sign-in posture
- Windy SSO only: password + passkey forms OFF, `ACCOUNT_LINKING=login`,
**auto-registration OFF** — opening the forge to non-Grant users is a §7
Grant decision.
- **Break-glass:** `sudo docker exec -u git windy-git-gitea-1 gitea admin user generate-access-token --username windyadmin --token-name <name> --scopes <scopes> --raw`
(delete it after: `delete from access_token where name='<name>'` in the gitea DB —
Gitea refuses token management over token auth).
## Troubleshooting
**A hostname returns 530 or won't resolve** — the tunnel is down. `sudo systemctl
restart windygit-tunnel`, then `journalctl -u windygit-tunnel -n 50`.
**`windygit-tunnel` crash-loops with `bind: address already in use` on the metrics
port** — cloudflared exits if it cannot bind `metrics:`, taking ingress with it.
Until 2026-09-23 this unit restarted ~91,000 times because another project's
`cornercall-tunnel` held 127.0.0.1:2000; ingress only survived because a stray
generic `cloudflared.service` ran the same config (now disabled). Windy Git's
metrics port is **2001**. `sudo ss -ltnp | grep :2001` names any squatter.
Keep exactly ONE unit running `/etc/cloudflared/config.yml`: `windygit-tunnel`.
**TLS handshake fails with `curl` exit 35 and no HTTP status at all** — someone
added a **two-level** hostname. Free Universal SSL covers `windygit.com` and
`*.windygit.com` only. The request dies before the tunnel is consulted, so it
@@ -88,3 +152,16 @@ disqualifying the moment a stranger depends on it. **The trigger is not a date
it is the first external push.** Move the control plane to a dedicated VPS (not
Kit 0), keep Veron 1 as the runner. It is an rsync, a Postgres dump and three
DNS record edits.
## Re-run a PR's CI (Gitea 1.24 has no rerun API)
```bash
ssh wg-veron
cd /srv/windygit/src && bash scripts/rerun_ci.sh <repo> <branch> <github-head-sha-prefix>
```
Moves the Windy Git branch back one commit; the next sync force-pushes the
GitHub head again and Gitea re-fires every workflow for that event on the same
commit. Guarded: refuses unless the branch is at the given sha, waits for a
sync that starts AFTER the rewind, restores the branch itself on timeout.
Don't use the web "Re-run" button: it needs a hub-SSO session as windyadmin,
which is Grant's identity.

262
docs/TURNOVER-2026-08-14.md Normal file
View File

@@ -0,0 +1,262 @@
# Windy Git — turnover, 2026-08-14
Paste the block at the bottom into a fresh terminal. Everything above is context
for whoever reads this file directly.
## Where things stand
Windy Git is **live and in use**: `app.windygit.com` (forge), `api.windygit.com`
(our plane), on **Veron 1** behind a Cloudflare Tunnel, zero inbound ports, $0/mo.
143 repos, 85 tests green, health `ok` on all four checks.
Grant signs in with his existing Windy Word credentials — no second account.
Agents authenticate with real Eternitas EPT signature verification and are
rate-limited by integrity band.
## SOLVED — the CI failures were never about Postgres
The `localhost` → `postgres` fix was correct and is worth keeping, but it was
**not** what was failing these jobs. They died at step 2, before Postgres was
ever contacted.
**Root cause: `astral-sh/setup-uv@v4` asks the forge for uv's latest release.**
setup-uv v4 added "resolve latest version instead of downloading latest release"
(astral-sh/setup-uv#178). Resolution goes through `@actions/github`, whose
octokit reads **`GITHUB_API_URL`** — which act_runner points at *our forge*. So
the action requested:
```
GET https://app.windygit.com/api/v1/repos/astral-sh/uv/releases/latest → 404
```
Gitea has no `astral-sh/uv`, so it answered its standard 404 body, *"The target
couldn't be found."* setup-uv threw that string, act printed it as `::error::`,
and every later step was skipped by `success()`.
**The fix (merged to GitHub, 11 repos):** pin an explicit `version:` on every
`setup-uv@v4`/`@v5` step. `resolveVersion()` short-circuits on an explicit
version *before* any API call, and the download URL is hardcoded to github.com —
so the forge round-trip disappears. Pinned to `0.12.5`, which is what `latest`
already resolved to.
windy-mind #101, WindyCloud #90, eternitas #150, then the sweep: Windy-Clone #77,
windy-agent #355, windy-call #34, windy-cell #31, windy-hand #6, windy-mail #105,
windy-search #77, windy-text #29. All merged and synced.
### Two things that made this hard to see, both worth keeping
- **act attributes the error to the wrong step.** `::error::The target couldn't
be found.` is printed immediately after `actions/checkout`'s `::remove-matcher`,
so it reads exactly like a checkout failure. It is not. What settled it was the
**Gitea access log** — `sudo docker logs windy-git-gitea-1 | grep " 404 "` — which
named the real URL at the same millisecond as the job error. When a job fails
with an opaque forge-shaped message, go to the forge's access log, not the job log.
- **The natural experiment was sitting right there.** windy-registry and
windy-drops use `setup-uv@v3` and always passed; every v4/v5 caller failed. A
version skew across otherwise-identical repos is a diagnosis, not a coincidence.
### The jobs API "job not found" that blocked the last session
Not a bug. `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs` requires the
job id to belong to **the repo in the path** — a valid id under the wrong owner/repo
404s. The API works fine; the URLs were mismatched. Logs are readable this way and
you do **not** need the web UI.
Note logs are **not on disk** — `[storage] STORAGE_TYPE = minio` sends action logs
to R2, so `actions_log/` on the host is empty. Read them through the API.
## SOLVED — second root cause: the runner was four majors behind
Windy-Clone failed for a completely different reason: `The runs.using key in
action.yml must be one of: [composite docker node12 node16 node20 go], got
**node24**`. act_runner **0.2.11**'s bundled act predates node24, so any repo
pinning a current action major (`actions/checkout@v5`, `actions/setup-python@v6`)
died before its first step.
**Bumped to `gitea/act_runner:0.6.1`** (`cd7dd9b`). Verified beforehand that
`node24` is absent from the 0.2.11 binary and present in 0.6.1, and that every
key in `deploy/runner/config.yaml` still exists in 0.6.1's schema — 0.6.1 only
*adds* keys, so the config carried over untouched. The runner re-declared with
the same id and labels `[veron-1 linux-x64 self-hosted linux x64]`; the
registration in the `runner-data` volume survived. Windy-Clone went 4/4 red →
4/4 green. Rollback is re-pinning 0.2.11; registration is backed up at
`/srv/windygit/runner-registration.bak`.
## What is still red, and why each one is real
The CI plane is healthy. windy-mind is fully green (`742 passed, 1 skipped`).
What remains are genuine repo defects that were **invisible before**, because
every job died at step 2:
| repo | job | cause |
|---|---|---|
| WindyCloud | `lint` | `ruff format --check` — 7 files would be reformatted |
| eternitas | `py-sdk` | `uv run pytest` → `Failed to spawn: pytest`; pytest isn't a declared dep of that project |
| eternitas | `test` | needs its own look |
| windy-agent | `test (3.12/3.13/3.14)` | all three died **together** at 22:50:19 after ~43 min, mid-suite at 64%, with no verdict in the log. Not the 30m `runner.timeout` (that would have fired at 22:36) and not the runner bump (that was 23:00). Something bulk-killed them; unexplained. |
| WindyCloud | `docker`, windy-search `Docker build` | **architectural** — see below |
`WindyCloud`'s `docker` job wants to build an image and gets `failed to connect
to the docker API at unix:///var/run/docker.sock`. Job containers deliberately
have **no** docker socket (I-5, and `deploy/runner/docker-compose.yml` says in
so many words not to mount it). Mounting the host socket would hand every
workflow root on Veron 1. This needs a decision — buildx-in-dind, a rootless
builder, or "this job does not run on Windy Git" — not a quiet socket mount.
The WindyCloud `lint` and eternitas `py-sdk` rows are small fixes in their own
repos, left alone on purpose: they are product defects, not forge defects.
## Second, smaller finding — act's action cache rots
act caches action repos at `/root/.cache/act/<hash>` inside the runner container
and refreshes them with a go-git mirror fetch of `refs/*:refs/*`, unforced. That
includes `refs/pull/*`, which GitHub **recomputes** whenever a base branch moves.
Reproduced directly:
```
! [rejected] refs/pull/1015/merge -> refs/pull/1015/merge (non-fast-forward)
```
which surfaces as `Non-terminating error while running 'git clone': some refs
were not updated`, after which the action does not report `Checked out <ref>`.
It **has** now failed a job on its own. After the runner bump, `windy-mind tests`
died with:
```
❌ Failure - Main Install uv
lstat /root/.cache/act/d3e6…/.git-blame-ignore-revs: no such file or directory
```
act tars the cached action directory into the job container, and a file vanished
mid-walk. The cache dir had been created at 23:01 and mutated again at 23:03,
with `.gitignore` showing as deleted — act removes it before `docker cp`. The
likely mechanism is **concurrent jobs sharing one cache dir**: `capacity: 4`, and
windy-mind fires four jobs at once that all use setup-uv. Wiping the cache and
re-running the job alone made it pass (`742 passed, 1 skipped`). *Mechanism not
isolated* — the wipe alone may have been sufficient.
One dead end worth not repeating: the cached worktree sits at `38f3f104` while
`git rev-parse v4` says `e4db8464`. That is **not** a wrong checkout — `v4` is an
*annotated tag*, and `v4^{commit}` is `38f3f104`. Don't chase it.
Wipe with `sudo docker exec windy-git-runner-runner-1 rm -rf /root/.cache/act`
(safe — container layer, not a volume). It will rot again. A real fix is either
lowering `capacity` or upstream act; neither was attempted.
## Traps that will waste your time
- **Gitea status codes are not what they look like.** `1 = success, 2 = failure`,
3 cancelled, 4 skipped, 5 waiting, 6 running, 7 blocked. Reading 1/2 as
waiting/running inverts every conclusion you draw from `action_run_job`.
- **Gitea sets `Secure` cookies** (ROOT_URL is https), so a `curl` login against
`http://127.0.0.1:3080` silently keeps no session — it 303s to `/` and you
still get "Sign In". Log in through `https://app.windygit.com`.
- **There is no rerun API in 1.24.6.** `POST /api/v1/.../runs/{n}/rerun` 404s.
Use the web route `POST /{owner}/{repo}/actions/runs/{n}/rerun` with the session
cookie plus an `X-Csrf-Token` header taken from the `_csrf` cookie.
- **`git pull -q` hides errors.** A divergent branch once made a "deploy" run 20
minutes against stale code while reporting success. Use `git fetch && git
merge --ff-only` and read the output.
- **Never force-push a branch a deploy checkout tracks** (`--amend` orphaned
`/srv/windygit/src` once).
- **Gitea's env-to-ini SETS but never UNSETS**, and sometimes *appends a
duplicate*. `GITEA__DEFAULT__APP_NAME` does not work at all — Gitea reads
`APP_NAME` from the **top level** of `app.ini`; the env var creates a literal
`[default]` section it ignores. Edit `app.ini` on the host.
- **Cloudflare caches `/assets/*` for 6h and no token in this stack can purge.**
Version brand asset **filenames** (`theme-windy.v2.css`), not query strings.
- **`base64` wraps at 76 chars** and corrupts long tokens in test commands →
`curl (43)`, phantom HTTP 000. Use `base64 -w0`.
- **Kit 0 is fragile.** 54 containers on 4 vCPU. Two production incidents in two
days, both from *non-production* workloads. Check `uptime` before deploying
anything there, and build before recreating so the swap is seconds.
- **Service containers**: use the service NAME and its INTERNAL port (5432),
never the mapped host port. The three repos did NOT share one pattern — a naive
`localhost` → `postgres` swap would have left WindyCloud on port 15432 (it maps
`15432:5432`) and windy-registry on a `job.services.postgres.ports[…]` expression.
## Open items, roughly by value
1. **Decide what the image-building jobs should do on Windy Git** — WindyCloud
`docker` and windy-search `Docker build` (above). The only remaining *forge*
question; it needs a decision, not code.
2. **windy-agent's three `test` jobs were bulk-killed at 22:50:19** after ~43
minutes, mid-suite, with no verdict. Unexplained and not the runner bump.
3. The product-level test failures in the table above.
4. **act's action cache race** — lower `capacity` below 4, or accept re-runs.
5. **Get non-prod work off Kit 0.** 12 dev/demo containers on the box running
identity, the CA, mail, Matrix and the broker. Cost two incidents already;
the postgres-adapter fix would not have prevented either.
6. **Login is ~4–6s** — `postgres-adapter.ts:114` forks a `node -e` process per
query. Measured: node startup alone is 1.7s on Kit 0 vs 0.01s on Veron. The
fix is **one function** (persistent worker + `pg.Pool`), not the "468 call
sites" the SOTU scoped. See `docs/incidents/2026-08-12-login-latency-analysis.md`.
7. **Privileged dind sits beside broad-scoped tokens** on the CI host — Grant's
call, needs a decision not a code change.
8. Push-velocity throttling is declared but unenforceable from our plane (git
push never touches the API); needs a Gitea pre-receive hook.
## Read these first
- `~/.claude/.../memory/project_windy_git.md` — the full record, densest source
- `DNA_STRAND_MASTER_PLAN.md` — D-1…D-9 locked decisions, I-1…I-13 invariants
- `docs/AUDIT-fable-2026-08-13.md` — second-auditor findings and dispositions
- `docs/CUTOVER.md` — the GitHub↔Windy Git migration plan and its one rule
---
## Copy-paste prompt
```
Picking up Windy Git (agent-native code+model host on Veron 1, live at
app.windygit.com). Read these before doing anything:
1. ~/.claude/projects/-home-grantwhitmer/memory/project_windy_git.md
2. ~/windy-git/docs/TURNOVER-2026-08-14.md
3. ~/windy-git/DNA_STRAND_MASTER_PLAN.md (D-1..D-9, I-1..I-13)
State: live and in use. Grant signs in with his existing Windy account. Agents
authenticate with real EPT signature verification. 143 repos, 85 tests green.
TWO CI root causes are SOLVED and verified:
(a) setup-uv v4+ resolved uv's "latest" through GITHUB_API_URL, which
act_runner points at our own forge, so it 404'd ("The target couldn't be
found.") and every job died at step 2. Fixed by pinning an explicit uv
version across 11 repos.
(b) act_runner 0.2.11 predates `runs.using: node24`, so any repo on
actions/checkout@v5 died before its first step. Bumped to 0.6.1.
windy-mind is fully green (742 passed). Windy-Clone went 4/4 red to 4/4 green.
TASK: one decision, then cleanup.
1. DECIDE what the image-building CI jobs should do here — WindyCloud `docker`
and windy-search `Docker build`. They need a Docker daemon; job containers
deliberately have no socket (I-5 — mounting the host socket hands every
workflow root on Veron 1). Options: buildx inside the existing dind, a
rootless builder, or exclude the job. Do NOT mount the host socket.
2. windy-agent's three `test` jobs were bulk-killed together at 22:50:19 after
~43 min, mid-suite, with no verdict in the log. Not the 30m runner.timeout,
not the runner bump. Unexplained — worth a look.
3. Small product defects: WindyCloud `lint` (ruff format, 7 files), eternitas
`py-sdk` (pytest not a declared dep), eternitas `test`.
Ground rules already paid for the hard way:
- Gitea job status: 1=SUCCESS, 2=FAILURE, 5=waiting, 6=running. Not what you'd guess.
- When a job fails with an opaque forge-shaped error, read the FORGE access log
(`docker logs windy-git-gitea-1 | grep " 404 "`) — act misattributes the error
to the previous step.
- Job logs: `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs`. The job id
must belong to the repo in the path or you get a misleading "job not found".
Logs are in R2, not on disk.
- Log into the forge over https://app.windygit.com — Gitea's cookies are Secure,
so a curl login to http://127.0.0.1:3080 silently keeps no session.
- verify the WHOLE flow, not the half that curls easily
- never `git pull -q` in a deploy path; it hides errors
- fixes go to GitHub, not Windy Git (sync is GitHub -> Windy Git, force-push)
- if a job fails with `lstat .../<file>: no such file or directory` on an
action, act's cache rotted: `docker exec windy-git-runner-runner-1 rm -rf
/root/.cache/act`, then re-run. Safe; it is a container layer, not a volume.
- check Kit 0's `uptime` before deploying there; two incidents in two days
```

View File

@@ -25,10 +25,13 @@ dependencies = [
"alembic>=1.14",
"httpx>=0.27",
"boto3>=1.35",
# ES256 verification of Eternitas EPTs (G3.2/G9.1). Without crypto extras
# PyJWT cannot verify EC signatures and silently offers no protection.
"pyjwt[crypto]>=2.9",
]
[project.optional-dependencies]
dev = ["pytest>=8.3", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
dev = ["pytest>=8.3", "pyyaml>=6.0", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
[tool.ruff]
line-length = 100

View File

@@ -23,6 +23,23 @@ BUCKET="${R2_BUCKET_BACKUPS:-windy-git-backups}"
KEEP_DAYS="${BACKUP_KEEP_DAYS:-30}"
FAILED=0
# NEVER bundle these to R2 (orchestrator decision 2026-09-23). They carry
# credentials in plaintext — kit-army-config IS the lockbox, and the soul repos
# hold agent memory with keys in it — and these bundles are unencrypted, so
# anyone holding the R2 key could read every secret in the fleet. They are
# backed up ENCRYPTED elsewhere (Windy Drops lane, restic, restore-tested) and
# stay mirrored on Veron's own disk in Gitea. Extended globs, matched on name.
EXCLUDE="${BACKUP_EXCLUDE:-kit-army-config anima *-soul}"
excluded() {
local n=$1 pat pats
read -ra pats <<< "$EXCLUDE" # read never glob-expands; `for p in $EXCLUDE` would
for pat in "${pats[@]}"; do
# shellcheck disable=SC2053 # unquoted RHS: glob match is the point
[[ "$n" == $pat ]] && return 0
done
return 1
}
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
@@ -44,6 +61,10 @@ count=0
for repo in "$GIT_ROOT"/*/*.git; do
owner="$(basename "$(dirname "$repo")")"
name="$(basename "$repo" .git)"
if excluded "$name"; then
log "skip ${owner}/${name} (credential-bearing: never bundled to R2 in plaintext)"
continue
fi
out="$WORK/${owner}__${name}.bundle"
# --all captures every ref, not just the default branch. A bundle of one

View File

@@ -33,6 +33,7 @@ import sys
import time
import urllib.error
import urllib.request
from collections.abc import Callable
from dataclasses import dataclass, field
STATE_PATH = os.environ.get("CANARY_STATE", "canary-state.json")
@@ -46,6 +47,16 @@ ALERT_FROM = os.environ.get("CANARY_ALERT_FROM", "office@thewindstorm.uk")
LOGIN_WARN_SECONDS = float(os.environ.get("CANARY_LOGIN_WARN_S", "35"))
TIMEOUT = float(os.environ.get("CANARY_TIMEOUT_S", "60"))
# Journey cleanup rule (orchestrator, 2026-09-23). The login probe creates a hub
# session (access + refresh token) every run, so it must end it. The hub's
# /auth/logout revokes the token AND every refresh token of the account
# (verified live: access 401, refresh 401 after it). So the next successful
# logout also heals anything a failed run left behind; no ledger needed.
LOGOUT_URL = "https://account.windyword.ai/api/v1/auth/logout"
LOGOUT_ATTEMPTS = 8 # retried on 5xx / no response only
LOGOUT_GAP_S = 15.0
LOGOUT_GONE = (401, 404, 410) # the session is already over = done
@dataclass
class Result:
@@ -54,6 +65,7 @@ class Result:
detail: str
seconds: float = 0.0
user_visible: str = ""
followups: list[Result] = field(default_factory=list)
@dataclass
@@ -65,11 +77,19 @@ class Check:
body: dict | None = None
headers: dict = field(default_factory=dict)
warn_seconds: float | None = None
# When True this check INVERTS: a 2xx is a critical failure (a security
# control opened) and a 401/403/503 is the healthy, expected outcome.
must_refuse: bool = False
# Runs on a 2xx with the response body; returns follow-up results (cleanup).
after: Callable[[bytes], list[Result]] | None = None
def _probe(c: Check) -> Result:
data = json.dumps(c.body).encode() if c.body else None
headers = {"User-Agent": "windy-git-canary/1.0", **c.headers}
# Our own probes are synthetic traffic (ecosystem convention, Telemetry
# UPDATE 4): every service they touch labels the resulting rows.
headers["X-Windy-Synthetic"] = "1"
if data:
headers["Content-Type"] = "application/json"
req = urllib.request.Request(c.url, data=data, method=c.method, headers=headers)
@@ -77,16 +97,30 @@ def _probe(c: Check) -> Result:
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
elapsed = time.monotonic() - start
if c.must_refuse:
# A 2xx here means a control that should reject accepted. That is
# the alarm, not the absence of one.
return Result(c.name, "down",
f"ACCEPTED (HTTP {r.status}) — this MUST be refused",
elapsed, c.what_it_proves)
if r.status >= 400:
return Result(c.name, "down", f"HTTP {r.status}", elapsed, c.what_it_proves)
raw = r.read()
warn = c.warn_seconds
if warn and elapsed > warn:
return Result(
res = Result(
c.name, "slow", f"HTTP {r.status} in {elapsed:.1f}s (warn >{warn:.0f}s)",
elapsed, c.what_it_proves,
)
return Result(c.name, "ok", f"HTTP {r.status} in {elapsed:.1f}s", elapsed, c.what_it_proves)
else:
res = Result(c.name, "ok", f"HTTP {r.status} in {elapsed:.1f}s", elapsed, c.what_it_proves)
if c.after:
res.followups = c.after(raw)
return res
except urllib.error.HTTPError as e:
if c.must_refuse and e.code in (401, 403, 503):
return Result(c.name, "ok", f"correctly refused (HTTP {e.code})",
time.monotonic() - start, c.what_it_proves)
return Result(c.name, "down", f"HTTP {e.code}", time.monotonic() - start, c.what_it_proves)
except Exception as e: # noqa: BLE001 — a probe must never raise upward
return Result(
@@ -95,6 +129,52 @@ def _probe(c: Check) -> Result:
)
def logout(token: str, *, attempts: int = LOGOUT_ATTEMPTS, gap: float = LOGOUT_GAP_S,
sleep: Callable[[float], None] = time.sleep) -> Result:
"""End the session the login probe opened. Honest: never ok unless proven."""
what = "the canary leaves no live session behind (journey cleanup rule)"
headers = {
"User-Agent": "windy-git-canary/1.0",
"X-Windy-Synthetic": "1",
"Authorization": f"Bearer {token}",
}
start = time.monotonic()
last = "no attempt"
for i in range(attempts):
if i:
sleep(gap)
req = urllib.request.Request(LOGOUT_URL, data=b"", method="POST", headers=headers)
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
return Result("identity.logout", "ok", f"session ended (HTTP {r.status})",
time.monotonic() - start, what)
except urllib.error.HTTPError as e:
if e.code in LOGOUT_GONE:
return Result("identity.logout", "ok", f"session already over (HTTP {e.code})",
time.monotonic() - start, what)
if e.code < 500: # a 4xx won't change on retry: fail fast
return Result("identity.logout", "down", f"CLEANUP FAILED: HTTP {e.code}",
time.monotonic() - start, what)
last = f"HTTP {e.code}"
except Exception as e: # noqa: BLE001 — no response / timeout: retry
last = f"{type(e).__name__}"
return Result("identity.logout", "down",
f"CLEANUP FAILED after {attempts} tries: {last} (next run's logout heals it)",
time.monotonic() - start, what)
def _logout_after_login(raw: bytes) -> list[Result]:
try:
token = (json.loads(raw or b"{}") or {}).get("token")
except ValueError:
token = None
if not token:
return [Result("identity.logout", "down",
"CLEANUP FAILED: login returned no token to log out with",
0.0, "the canary leaves no live session behind (journey cleanup rule)")]
return [logout(token)]
def build_checks() -> list[Check]:
checks = [
Check(
@@ -129,6 +209,36 @@ def build_checks() -> list[Check]:
),
]
# SECURITY REGRESSION GUARD. A forged, unsigned token naming a real passport
# must be refused. On 2026-08-13 this returned HTTP 200 (full agent
# impersonation). If it ever returns 2xx again, the bypass is back.
import base64 as _b64
import json as _j
def _seg(d: dict) -> str:
return _b64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
# Two shapes, because they exercise two different gates. The EPT-shaped one
# is the important one now: it is what real signature verification guards.
for _label, _hdr in (
("security.forged_agent_token", {"alg": "none", "typ": "JWT"}),
("security.forged_ept", {"alg": "none", "typ": "EPT"}),
):
_tok = (
f"{_seg(_hdr)}."
f"{_seg({'sub': 'ET26-1EF9-VJAN', 'passport': 'ET26-1EF9-VJAN', 'iss': 'eternitas.ai', 'exp': 9999999999})}"
".not-a-real-signature"
)
checks.append(
Check(
_label,
"https://api.windygit.com/api/v1/repos",
"an unsigned token cannot impersonate an agent",
headers={"Authorization": f"Bearer {_tok}"},
must_refuse=True,
)
)
# THE important one. /health was 200 for the entire 2026-08-12 outage while
# this was timing out. A canary that skips it is decorative.
pw = os.environ.get("CANARY_LOGIN_PASSWORD", "")
@@ -142,6 +252,7 @@ def build_checks() -> list[Check]:
method="POST",
body={"email": email, "password": pw},
warn_seconds=LOGIN_WARN_SECONDS,
after=_logout_after_login,
)
)
return checks
@@ -218,7 +329,10 @@ def main() -> int:
args = ap.parse_args()
previous = load_state()
results = [_probe(c) for c in build_checks()]
results = []
for c in build_checks():
r = _probe(c)
results += [r, *r.followups]
print(f"windy canary — {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}\n")
for r in results:

10
scripts/cancel_unrunnable.sh Executable file
View File

@@ -0,0 +1,10 @@
#!/usr/bin/env bash
# Cancel jobs no runner can ever take (see cancel_unrunnable.sql). Run on Veron as root.
set -euo pipefail
SPOOL="${JANITOR_SPOOL:-/var/lib/windy-git/janitor-cancelled.jsonl}"
mkdir -p "$(dirname "$SPOOL")"
out=$(docker exec -i windy-git-db-1 sh -c 'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1' \
< "$(dirname "$0")/cancel_unrunnable.sql")
printf '%s\n' "$out" | grep '^{' >> "$SPOOL" || true
n=$(printf '%s\n' "$out" | grep -c '^{' || true)
echo "[janitor] cancelled ${n} unrunnable job(s)"

View File

@@ -0,0 +1,53 @@
-- Cancel CI jobs that can never run (called by scripts/cancel_unrunnable.sh).
--
-- A job whose runs-on names a label no Windy Git runner offers (ubuntu-latest,
-- macos-latest, windows-latest …) waits forever: Gitea evaluates a job's `if:`
-- only when a runner picks it, so even `if: false` / tag-only jobs sit in the
-- queue, invisible to /actions/tasks, and keep their run "waiting" for good.
-- After 30 minutes they are cancelled here; the run's status is then recomputed
-- (failure > still-active > cancelled > success), the same precedence Gitea uses.
-- Keep RUNNER_LABELS in step with deploy/runner/config.yaml.
BEGIN;
WITH dead AS (
UPDATE action_run_job j
SET status = 3, stopped = extract(epoch from now())::bigint, updated = extract(epoch from now())::bigint
WHERE j.status IN (5, 7)
AND to_timestamp(j.created) < now() - interval '30 minutes'
AND EXISTS (SELECT 1 FROM jsonb_array_elements_text(j.runs_on::jsonb) l
WHERE l NOT IN ('veron-1', 'linux-x64', 'self-hosted', 'linux', 'x64'))
RETURNING j.id, j.run_id, j.name, j.runs_on, j.created
), runs AS (
UPDATE action_run r
SET status = CASE
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status = 2) THEN 2
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status IN (5, 6, 7)
AND x.id NOT IN (SELECT id FROM dead)) THEN r.status
ELSE 3 END,
stopped = CASE WHEN r.stopped = 0 THEN extract(epoch from now())::bigint ELSE r.stopped END
WHERE r.id IN (SELECT DISTINCT run_id FROM dead)
RETURNING r.id
)
-- One JSON line per cancelled job: the telemetry emitter ships these as
-- ci.job_cancelled (declared with Telemetry Boss, 2026-09-23).
SELECT json_build_object(
'repo', p.lower_name,
'workflow', regexp_replace(r.workflow_id, '\.ya?ml$', ''),
'job', d.name,
'reason', 'unrunnable_label',
'runs_on', (SELECT string_agg(l, ',') FROM jsonb_array_elements_text(d.runs_on::jsonb) l),
'waited_s', (extract(epoch from now())::bigint - d.created))::text
FROM dead d JOIN action_run r ON r.id = d.run_id JOIN repository p ON p.id = r.repo_id
WHERE (SELECT count(*) FROM runs) >= 0;
-- Jobs BLOCKED on `needs:` inside a run that has already finished (a needed job
-- failed): Gitea leaves them status 7 forever. They were never going to run;
-- mark them skipped (4), which is what GitHub shows for the same situation.
UPDATE action_run_job j
SET status = 4, updated = extract(epoch from now())::bigint
FROM action_run r
WHERE r.id = j.run_id
AND j.status = 7
AND r.status IN (1, 2, 3)
AND to_timestamp(j.created) < now() - interval '30 minutes'
RETURNING j.run_id;
COMMIT;

270
scripts/compute_guard.py Normal file
View File

@@ -0,0 +1,270 @@
#!/usr/bin/env python3
"""Compute guard: Windy Mind is the ONLY door to AI compute (Grant, 2026-09-23).
Flags code that talks to an AI provider directly instead of through Windy Mind:
a provider API host, a provider SDK import or dependency, or a raw provider key
name. Direct calls skip Mind's metering, caps and live-model routing, and they
spend whichever key happens to be lying around (the audit found Grant's personal
Max OAuth token inside a platform container).
WARN-ONLY for now: the bridge posts `windy-git/compute-guard` as success with a
"⚠ WARN" description, so nothing turns red. `COMPUTE_GUARD_MODE=block` flips
findings to failure once the repos are clean (orchestrator's call).
- PR heads: only lines the PR ADDS (vs its merge-base with the default branch).
- Default-branch head: the whole tree (the baseline, and what `report` prints).
Exceptions live in ONE file, ci/compute-guard-allow.yml, each with a reason.
Tests, docs, lockfiles, vendored code and CI config are never scanned.
Reads the sync's bare GitHub clones on Veron (no docker exec: IO-stall lesson).
python3 scripts/compute_guard.py report [repo ...] # whole-tree findings on each default branch
"""
from __future__ import annotations
import fnmatch
import hashlib
import json
import os
import re
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
ALLOW_FILE = Path(os.environ.get("COMPUTE_GUARD_ALLOW", ROOT / "ci" / "compute-guard-allow.yml"))
WORK = Path(os.environ.get("SYNC_WORK", "/srv/windygit/sync"))
CACHE = Path(os.environ.get("COMPUTE_GUARD_CACHE", "/var/lib/windy-git/compute-guard-cache.json"))
MODE = os.environ.get("COMPUTE_GUARD_MODE", "warn") # warn | block
HOSTS = [
"api.anthropic.com", "api.openai.com", "api.groq.com",
"generativelanguage.googleapis.com", "api.mistral.ai", "api.perplexity.ai",
"openrouter.ai", "api.together.xyz", "api.together.ai", "api.cerebras.ai",
"api.sambanova.ai", "api.deepseek.com", "api.x.ai", "api.cohere.ai",
"api.cohere.com", "api.fireworks.ai", "api.replicate.com",
"api-inference.huggingface.co",
]
KEYS = [
"ANTHROPIC_API_KEY", "ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_AUTH_TOKEN",
"OPENAI_API_KEY", "GROQ_API_KEY", "GEMINI_API_KEY", "GOOGLE_GENERATIVE_AI_API_KEY",
"GOOGLE_AI_API_KEY", "MISTRAL_API_KEY", "PERPLEXITY_API_KEY", "PPLX_API_KEY",
"OPENROUTER_API_KEY", "TOGETHER_API_KEY", "CEREBRAS_API_KEY", "SAMBANOVA_API_KEY",
"DEEPSEEK_API_KEY", "XAI_API_KEY", "COHERE_API_KEY", "FIREWORKS_API_KEY",
"REPLICATE_API_TOKEN",
]
PY_SDKS = r"anthropic|openai|groq|mistralai|cohere|google\.generativeai|google\.genai|together|cerebras|litellm"
JS_SDKS = (r"@anthropic-ai/sdk|openai|groq-sdk|@google/generative-ai|@google/genai|@mistralai/mistralai"
r"|cohere-ai|together-ai|@ai-sdk/(?:anthropic|openai|groq|google|mistral)")
RULES: list[tuple[str, re.Pattern]] = [
("provider host", re.compile("|".join(re.escape(h) for h in HOSTS))),
("provider key", re.compile(r"\b(?:" + "|".join(KEYS) + r")\b")),
("provider SDK", re.compile(rf"^\s*(?:from|import)\s+(?:{PY_SDKS})(?:\s|\.|$|,)")),
("provider SDK", re.compile(rf"""(?:from\s+|require\(\s*|import\(\s*)['"](?:{JS_SDKS})(?:/[^'"]*)?['"]""")),
# dependency manifests: package.json keys, requirements / pyproject lines
("provider SDK dep", re.compile(rf'''^\s*"(?:{JS_SDKS})"\s*:''')),
("provider SDK dep", re.compile(rf'''^\s*["']?(?:{PY_SDKS.replace(chr(92) + ".", "-")})(?:\[[^\]]*\])?\s*(?:[<>=~!]=?|["',]|$)''')),
]
DEP_FILES = re.compile(r"(^|/)(package\.json|requirements[^/]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)$")
# Never scanned: tests, docs, lockfiles, vendored/built code, CI config.
SKIP = re.compile(
r"(^|/)(tests?|__tests__|spec|docs?|node_modules|vendor|dist|build|\.github|\.gitea)/"
r"|(^|/)(test_[^/]*|[^/]*_test\.py|conftest\.py|[^/]*\.(test|spec)\.[cm]?[jt]sx?)$"
r"|\.(md|mdx|rst|txt|lock|snap|svg|png|jpg|pdf)$"
r"|(^|/)(package-lock\.json|pnpm-lock\.yaml|yarn\.lock|uv\.lock|poetry\.lock|Cargo\.lock)$"
)
@dataclass(frozen=True)
class Finding:
path: str
line: int
kind: str
match: str
def load_allow(path: Path = ALLOW_FILE) -> list[dict]:
data = yaml.safe_load(path.read_text()) or {}
entries = data.get("allow") or []
for e in entries: # a reason per entry is the whole point of the file
if not (e.get("repo") and e.get("paths") and str(e.get("reason", "")).strip()):
raise ValueError(f"allow entry needs repo, paths and a reason: {e}")
return entries
def allowed(repo: str, path: str, allow: list[dict]) -> bool:
for e in allow:
if e["repo"] == repo and any(fnmatch.fnmatch(path, g) for g in e["paths"]):
return True
return False
COMMENT = re.compile(r"^\s*(?:#|//|/\*|\*|<!--)")
def scan_line(path: str, text: str) -> list[tuple[str, str]]:
# A comment is not a call: "the ANTHROPIC_OAUTH_TOKEN setting was removed"
# (windy-search) must not count, nor a commented-out `# OPENAI_API_KEY=`.
if COMMENT.match(text):
return []
hits = []
for kind, rx in RULES:
if kind == "provider SDK dep" and not DEP_FILES.search(path):
continue
m = rx.search(text)
if m:
hits.append((kind, m.group(0).strip()[:60]))
return hits
def _git(bare: Path, *args: str) -> str:
return subprocess.run(
["git", "--git-dir", str(bare), *args],
capture_output=True, text=True, check=True, timeout=120,
).stdout
def scan_tree(repo: str, bare: Path, sha: str, allow: list[dict]) -> list[Finding]:
"""Every line in the tree at `sha` (default branch: the baseline)."""
# A cheap prefilter by git, then the real rules in Python.
pre = "|".join([re.escape(h) for h in HOSTS] + KEYS + ["anthropic", "openai", "groq", "mistral",
"generativeai", "genai", "cohere", "together", "cerebras", "litellm"])
try:
out = _git(bare, "grep", "-nIE", "-e", pre, sha, "--", ".")
except subprocess.CalledProcessError as e:
if e.returncode == 1: # no matches
return []
raise
found = []
for raw in out.splitlines():
# <sha>:<path>:<line>:<text>
try:
_, path, line, text = raw.split(":", 3)
except ValueError:
continue
if SKIP.search(path) or allowed(repo, path, allow):
continue
for kind, match in scan_line(path, text):
found.append(Finding(path, int(line), kind, match))
return found
def scan_added(repo: str, bare: Path, base_ref: str, sha: str, allow: list[dict]) -> list[Finding]:
"""Only the lines a PR adds, vs its merge-base with the default branch."""
mb = _git(bare, "merge-base", base_ref, sha).strip()
diff = _git(bare, "diff", "-U0", "--no-color", "--no-ext-diff", mb, sha)
return parse_added(repo, diff, allow)
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")
def parse_added(repo: str, diff: str, allow: list[dict]) -> list[Finding]:
found, path, line = [], None, 0
for raw in diff.splitlines():
if raw.startswith("+++ "):
p = raw[4:]
path = None if p == "/dev/null" else p[2:] if p.startswith("b/") else p
continue
m = HUNK.match(raw)
if m:
line = int(m.group(1))
continue
if path is None or raw.startswith("--- "):
continue
if raw.startswith("+"):
if not (SKIP.search(path) or allowed(repo, path, allow)):
for kind, match in scan_line(path, raw[1:]):
found.append(Finding(path, line, kind, match))
line += 1
return found
# ---- cache: a tree scan runs once per (repo, sha, rules+allow) --------------
def _fingerprint(allow: list[dict]) -> str:
return hashlib.sha256(
json.dumps([HOSTS, KEYS, PY_SDKS, JS_SDKS, SKIP.pattern, allow], sort_keys=True).encode()
).hexdigest()[:16]
def cached_scan(key: str, fn) -> list[Finding]:
try:
cache = json.loads(CACHE.read_text())
except (OSError, ValueError):
cache = {}
if key in cache:
return [Finding(**f) for f in cache[key]]
result = fn()
cache[key] = [f.__dict__ for f in result]
if len(cache) > 2000: # keep it small: newest entries win
cache = dict(list(cache.items())[-1000:])
try:
CACHE.parent.mkdir(parents=True, exist_ok=True)
tmp = CACHE.with_suffix(".tmp")
tmp.write_text(json.dumps(cache))
tmp.replace(CACHE)
except OSError:
pass
return result
def check(repo: str, sha: str, default_branch: str, is_default_head: bool) -> list[Finding] | None:
"""Findings for one commit, or None when the guard can't run (never a fake OK)."""
bare = WORK / f"{repo}.git"
if not bare.is_dir():
return None
allow = load_allow()
fp = _fingerprint(allow)
if is_default_head:
return cached_scan(f"tree:{repo}:{sha}:{fp}", lambda: scan_tree(repo, bare, sha, allow))
return cached_scan(
f"pr:{repo}:{sha}:{fp}",
lambda: scan_added(repo, bare, f"refs/heads/{default_branch}", sha, allow),
)
def status_for(findings: list[Finding], whole_tree: bool) -> tuple[str, str, Finding | None]:
"""(state, description, first finding) for the GitHub commit status."""
scope = "in tree" if whole_tree else "added"
if not findings:
what = "no direct AI-provider use in tree" if whole_tree else "no direct AI-provider use added"
return "success", f"OK: {what} (Windy Mind is the only door)", None
f = findings[0]
n = len(findings)
state = "failure" if MODE == "block" else "success"
lead = "BLOCKED" if MODE == "block" else "⚠ WARN (not blocking)"
desc = f"{lead}: {n} direct AI-provider use{'s' if n > 1 else ''} {scope}, e.g. {f.path}:{f.line} {f.match}"
return state, desc[:140], f
def report(repos: list[str]) -> int:
allow = load_allow()
total = 0
for repo in repos:
bare = WORK / f"{repo}.git"
if not bare.is_dir():
print(f"## {repo}: no sync clone, skipped")
continue
head = _git(bare, "symbolic-ref", "--short", "HEAD").strip()
sha = _git(bare, "rev-parse", head).strip()
fs = scan_tree(repo, bare, sha, allow)
total += len(fs)
print(f"## {repo} ({head} {sha[:7]}): {len(fs)} finding(s)")
for f in fs:
print(f" {f.path}:{f.line} [{f.kind}] {f.match}")
print(f"TOTAL {total}")
return 0
if __name__ == "__main__":
if len(sys.argv) >= 2 and sys.argv[1] == "report":
default = os.environ.get("BRIDGE_REPOS", "").split() or sorted(
p.name.removesuffix(".git") for p in WORK.glob("*.git"))
sys.exit(report(sys.argv[2:] or default))
sys.exit(__doc__)

View File

@@ -43,7 +43,12 @@ import urllib.request
#
# Bulk import belongs on the host anyway: no hairpin through the edge, no
# Cloudflare ~100s proxy ceiling (G4A.5) on a large clone. Run this on Veron 1.
GITEA = os.environ.get("GITEA_BASE_URL", "http://localhost:3080")
#
# 🔴 Deliberately NOT `GITEA_BASE_URL`: the deploy `.env` sets that to
# `http://gitea:3000` for the API container, and sourcing `.env` on the host
# made this script die on DNS *after* a caller had already deleted the mirror it
# was meant to replace (2026-09-23).
GITEA = os.environ.get("IMPORT_GITEA_URL", "http://localhost:3080")
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
GITHUB_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
@@ -224,13 +229,12 @@ def main() -> int:
if not targets:
ap.error("name a repo, or pass --safe-batch / --list-candidates")
if "windy-pro" in targets:
sys.exit(
"REFUSING windy-pro. Six checkouts exist, the build counter has forked "
"three ways (main 12 / overnight 34 / wave-44 56), and two sessions "
"recorded different HEADs hours apart. Resolve which is current and "
"write it down BEFORE importing (G11.5)."
)
# G11.5 RESOLVED 2026-09-23 (lane 8c, ~/windy-orchestra/WINDYPRO_CHECKOUTS.md):
# a read-only audit of all 14 windy-pro checkouts on 5 machines found GitHub
# main is canonical (Kit 0 prod and Windy 0 sit exactly on it; the others are
# stale, not divergent). Phase 1 keeps GitHub the source of truth anyway, so a
# writable Windy Git copy is CI only. Its six deploy/release workflows must be
# disabled on import — see docs/CUTOVER.md.
if args.mirror:
print("mirror mode: repos will be read-only and will NOT run CI.\n")

402
scripts/pr_status_bridge.py Executable file
View File

@@ -0,0 +1,402 @@
#!/usr/bin/env python3
"""Give private GitHub repos a CI signal from Windy Git (P1, 2026-09-23).
GitHub Actions cannot run on private repos on this account — not even on
self-hosted runners ([[reference-github-actions-billing-lock]]). The code is
already synced into Windy Git every 15 min and CI runs there, so the only thing
missing is the *signal on GitHub*, where people and agents actually read PRs.
Two jobs, run after every sync:
1. **Mirror open PRs.** The sync carries branches, not PRs, and the workflows
trigger on `pull_request` — so a PR branch alone fires nothing. For each open
same-repo GitHub PR we keep one open Windy Git PR with the same head/base.
Gitea then fires `pull_request` on open and `synchronize` whenever the sync
moves the branch. Windy Git PRs whose GitHub PR closed are closed here too.
Fork PRs are ignored: their head branch is never synced, and untrusted fork
code on this runner is exactly the blast radius the audit warned about.
2. **Post results back** as GitHub commit statuses (context
`windy-git/<workflow>/<job>`) on each PR head and on the default-branch head.
Only posts when a context's state changed, so a 15-min loop doesn't pile
hundreds of identical statuses onto one commit.
Runs ON Veron 1 (localhost Gitea; no Cloudflare hairpin). Needs
GITEA_ADMIN_TOKEN and a GITHUB_TOKEN with `repo` scope. Nothing here executes
repo code, and no secret is handed to any repo.
"""
from __future__ import annotations
import base64
import json
import os
import re
import subprocess
import sys
import time
import urllib.error
import urllib.request
import yaml
GITEA = os.environ.get("BRIDGE_GITEA_URL", "http://localhost:3080").rstrip("/")
PUBLIC = "https://app.windygit.com"
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
GH_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
WG_OWNER = os.environ.get("WINDYGIT_OWNER", "windyadmin")
# Private repos only. Public repos run real GitHub Actions on veron1's GitHub
# runner; bridging those too would put two competing verdicts on every commit.
REPOS = os.environ.get(
"BRIDGE_REPOS",
"windy-chat windy-mail windy-calendar Windy-Clone WindyCloud windy-search windy-connect"
" windy-drops windy-code-web windy-code windy-traveler windy-registry eternitas"
" windy-translate windytranslate-site windytraveler-site windy-hand"
" windy-cloud-sites windy-cloud-domains windy-cloud-vps windytalk windy-pro windy-mind",
).split()
# Gitea run status -> GitHub status state. `skipped` is deliberately absent: a
# job skipped by its own `if:` (e.g. substrate-drift's no-secrets path) has no
# verdict, and painting it green would be a claim nobody tested.
STATE = {
"success": "success",
"failure": "failure",
"cancelled": "error",
"running": "pending",
"waiting": "pending",
"blocked": "pending",
}
MIRROR_TAG = "[GH#"
# Image-build jobs cannot pass here BY DESIGN: job containers get no Docker
# daemon (I-5 — the host socket would hand every workflow root on Veron 1).
# Posting them would put a permanent red X on every commit, and a signal that is
# always red trains everyone to ignore red. Not posted until a rootless builder
# exists; that is a decision, recorded in docs/CUTOVER.md, not a failure.
NO_DAEMON_JOB = re.compile(r"docker", re.IGNORECASE)
# Jobs Grant ruled NON-BLOCKING (GRANT_DECISIONS_2026-09-23): still run on
# Windy Git and visible there, but not posted to GitHub, so they cannot turn a
# commit's combined status red. Format: "repo:workflow/job,workflow/job;repo2:..."
# windy-pro's desktop/installer jobs belong to Grant's desktop side (fixed from
# his Mac mini), not to any lane's merge gate.
NON_BLOCKING: dict[str, set[str]] = {}
for _entry in os.environ.get(
"BRIDGE_NON_BLOCKING", "windy-pro:ci/build-desktop,ci/test-installer,ci/reality-check"
).split(";"):
if ":" in _entry:
_repo, _jobs = _entry.split(":", 1)
NON_BLOCKING[_repo.strip()] = {j.strip() for j in _jobs.split(",") if j.strip()}
# Gitea reads the FIRST of these dirs that has workflow files at a commit (1.24).
WORKFLOW_DIRS = (".gitea/workflows", ".github/workflows")
def workflow_problem(text: str) -> str | None:
"""Why Gitea would drop this workflow file, or None if it looks runnable.
Gitea skips an invalid workflow with one log line and fires no run at all,
so on GitHub the PR just shows nothing, and people wait for CI that is never
coming. These are the shapes we have actually hit, not a full schema.
"""
try:
doc = yaml.safe_load(text)
except yaml.YAMLError as e:
mark = getattr(e, "problem_mark", None)
return f"invalid YAML at line {mark.line + 1}" if mark else "invalid YAML"
if not isinstance(doc, dict):
return "not a YAML mapping"
if "on" not in doc and True not in doc: # YAML 1.1 reads a bare `on` as True
return "no `on:` trigger"
jobs = doc.get("jobs")
if not isinstance(jobs, dict) or not jobs:
return "no `jobs:`"
for name, job in jobs.items():
if not isinstance(job, dict):
return f"job `{name}` is not a mapping"
if "runs-on" not in job and "uses" not in job:
return f"job `{name}` has no `runs-on:`"
return None
def invalid_workflows(repo: str, sha: str) -> dict[str, tuple[str, str]]:
"""{context: (path, problem)} for each workflow file at `sha` that won't run."""
for d in WORKFLOW_DIRS:
st, entries = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{d}?ref={sha}")
if st == 404:
continue
if st != 200:
raise RuntimeError(f"{repo}: Windy Git {d}@{sha[:7]} -> {st}")
files = [e for e in entries or [] if e.get("type") == "file"
and e["name"].endswith((".yml", ".yaml"))]
if not files:
continue
bad = {}
for e in files:
st, f = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{e['path']}?ref={sha}")
if st != 200:
raise RuntimeError(f"{repo}: Windy Git {e['path']}@{sha[:7]} -> {st}")
problem = workflow_problem(base64.b64decode(f["content"]).decode("utf-8", "replace"))
if problem:
stem = re.sub(r"\.ya?ml$", "", e["name"])
bad[f"windy-git/{stem}/workflow"] = (e["path"], problem)
return bad
return {}
def _call(base: str, token_header: str, method: str, path: str, body=None):
req = urllib.request.Request(
base + path,
data=json.dumps(body).encode() if body is not None else None,
method=method,
headers={
"Authorization": token_header,
"Content-Type": "application/json",
"Accept": "application/json",
# urllib's default UA is 403'd as a bot by GitHub's edge and CF.
"User-Agent": "windy-git-pr-bridge/1",
},
)
# Transport errors (TLS handshake timeout, reset) are retried: one GitHub
# blip used to fail the whole sync, flip its heartbeat to ok:false and page
# someone for nothing. HTTP errors are answers, not blips — never retried.
for attempt in range(3):
try:
with urllib.request.urlopen(req, timeout=60) as r:
raw = r.read()
return r.status, (json.loads(raw) if raw else None)
except urllib.error.HTTPError as e:
return e.code, None
except (urllib.error.URLError, TimeoutError, ConnectionError):
if attempt == 2:
raise
time.sleep(2 * (attempt + 1))
raise AssertionError("unreachable")
def gitea(method, path, body=None):
return _call(GITEA + "/api/v1", f"token {GITEA_TOKEN}", method, path, body)
def github(method, path, body=None):
return _call("https://api.github.com", f"Bearer {GITHUB_TOKEN}", method, path, body)
def sync_prs(repo: str) -> list[str]:
"""Mirror open same-repo GitHub PRs into Windy Git. Returns their head shas."""
st, gh_prs = github("GET", f"/repos/{GH_OWNER}/{repo}/pulls?state=open&per_page=100")
if st != 200:
raise RuntimeError(f"{repo}: GitHub PR list -> {st}")
st, wg_prs = gitea("GET", f"/repos/{WG_OWNER}/{repo}/pulls?state=open&limit=50")
if st != 200:
raise RuntimeError(f"{repo}: Windy Git PR list -> {st}")
ours = {p["title"].split("]")[0] + "]": p for p in wg_prs if p["title"].startswith(MIRROR_TAG)}
heads, wanted = [], set()
for pr in gh_prs:
if pr["head"]["repo"] is None or pr["head"]["repo"]["full_name"] != f"{GH_OWNER}/{repo}":
continue # fork PR — never synced, never run here
tag = f"{MIRROR_TAG}{pr['number']}]"
wanted.add(tag)
heads.append(pr["head"]["sha"])
if tag in ours:
continue
st, _ = gitea(
"POST",
f"/repos/{WG_OWNER}/{repo}/pulls",
{
"head": pr["head"]["ref"],
"base": pr["base"]["ref"],
"title": f"{tag} {pr['title']}"[:250],
"body": f"Mirror of {pr['html_url']} so CI runs here. Do not merge in Windy Git — "
"GitHub is the source of truth; merge there.",
},
)
print(f" {repo}: opened mirror PR for GH#{pr['number']} -> {st}")
for tag, p in ours.items():
if tag not in wanted:
gitea("PATCH", f"/repos/{WG_OWNER}/{repo}/pulls/{p['number']}", {"state": "closed"})
print(f" {repo}: closed mirror PR {tag} (closed on GitHub)")
return heads
SAFE_NAME = re.compile(r"^[A-Za-z0-9._-]+$")
SAFE_SHA = re.compile(r"^[0-9a-f]{40}$")
def queued_jobs(repo: str, sha: str) -> list[dict]:
"""Jobs at `sha` that are waiting for a runner and that a runner CAN take.
Gitea 1.24's API lists only PICKED-UP jobs (/actions/tasks), so a queued PR
showed nothing on GitHub and people asked whether the push was lost. The
truth is in the gitea DB. Bounded + non-fatal: during the 09-23 IO stall
`docker exec` hung for an hour and must never wedge the bridge again.
Only status 5 (waiting) with labels some live runner has. A `pending` we
post must end in a verdict we will also see, or it sits yellow forever:
blocked jobs (7) often end SKIPPED, and jobs for labels no runner has
(macos-/windows-/ubuntu-latest) are cancelled by the janitor unpicked;
neither ever appears in /actions/tasks.
"""
if not (SAFE_NAME.match(repo) and SAFE_NAME.match(WG_OWNER) and SAFE_SHA.match(sha)):
return []
query = (
"select json_build_object("
" 'jobs', (select coalesce(json_agg(t), '[]'::json) from ("
" select ar.index as run_number, ar.workflow_id, j.name, j.runs_on"
" from action_run_job j join action_run ar on ar.id = j.run_id"
" join repository r on r.id = j.repo_id join \"user\" o on o.id = r.owner_id"
f" where o.lower_name = '{WG_OWNER.lower()}' and r.lower_name = '{repo.lower()}'"
f" and ar.commit_sha = '{sha}' and j.status = 5) t),"
" 'labels', (select coalesce(json_agg(agent_labels), '[]'::json)"
" from action_runner where coalesce(deleted, 0) = 0));"
)
try:
out = subprocess.run(
["docker", "exec", "-i", "windy-git-db-1", "sh", "-c",
'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1'],
input=query, capture_output=True, text=True, check=True, timeout=30,
).stdout.strip()
got = json.loads(out or "{}")
runners = [set(json.loads(x or "[]")) for x in got.get("labels") or []]
return [
j for j in got.get("jobs") or []
if any(set(json.loads(j.get("runs_on") or "[]")) <= r for r in runners)
]
except (subprocess.SubprocessError, OSError, ValueError) as e:
print(f" {repo}: queued-job lookup skipped ({type(e).__name__})")
return []
def post_statuses(repo: str, sha: str) -> None:
# Gitea caps a page at 50 (MAX_RESPONSE_ITEMS) whatever `limit` says, and a
# daily scheduled workflow can push a quiet main's runs off page 1.
runs = []
for page in range(1, 6):
st, body = gitea("GET", f"/repos/{WG_OWNER}/{repo}/actions/tasks?limit=50&page={page}")
if st != 200:
raise RuntimeError(f"{repo}: Windy Git runs -> {st}")
runs += body.get("workflow_runs", [])
if len(body.get("workflow_runs", [])) < 50:
break
latest: dict[str, dict] = {}
for r in runs:
if r["head_sha"] != sha or NO_DAEMON_JOB.search(r["name"]):
continue
if f"{r['workflow_id'].removesuffix('.yml')}/{r['name']}" in NON_BLOCKING.get(repo, ()):
continue
ctx = f"windy-git/{r['workflow_id'].removesuffix('.yml')}/{r['name']}"
if ctx not in latest or r["id"] > latest[ctx]["id"]:
latest[ctx] = r
# Queued jobs: `pending` where nothing newer has been picked up. A re-run
# queued behind an old failure must read pending, not the stale red.
for q in queued_jobs(repo, sha):
if NO_DAEMON_JOB.search(q["name"]):
continue
wf = q["workflow_id"].removesuffix(".yml")
if f"{wf}/{q['name']}" in NON_BLOCKING.get(repo, ()):
continue
ctx = f"windy-git/{wf}/{q['name']}"
if ctx not in latest or q["run_number"] > latest[ctx]["run_number"]:
latest[ctx] = {"id": 0, "status": "waiting", "run_number": q["run_number"]}
bad = invalid_workflows(repo, sha)
if not (latest or bad):
return
st, existing = github("GET", f"/repos/{GH_OWNER}/{repo}/commits/{sha}/statuses?per_page=100")
current: dict[str, str] = {}
for s in existing or []: # newest first
current.setdefault(s["context"], s["state"])
for ctx, (path, problem) in sorted(bad.items()):
if current.get(ctx) == "error":
continue
st, _ = github(
"POST",
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
{
"state": "error",
"context": ctx,
"description": f"Windy Git ignored this workflow, no CI ran: {problem}"[:140],
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}/{path}",
},
)
print(f" {repo}@{sha[:7]} {ctx} = error ({problem}) -> {st}")
for ctx, r in sorted(latest.items()):
state = STATE.get(r["status"])
if state is None or current.get(ctx) == state:
continue
st, _ = github(
"POST",
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
{
"state": state,
"context": ctx,
"description": f"Windy Git CI on Veron 1: {r['status']}"[:140],
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/actions/runs/{r['run_number']}",
},
)
print(f" {repo}@{sha[:7]} {ctx} = {state} -> {st}")
GUARD_CTX = "windy-git/compute-guard"
def post_compute_guard(repo: str, sha: str, default_branch: str, is_default_head: bool) -> None:
"""Windy Mind is the only door to AI compute: flag direct provider use (warn-only).
Non-fatal and never a fake OK: if the guard can't run, nothing is posted.
"""
try:
import compute_guard as cg # same directory; loaded lazily so the bridge never depends on it
findings = cg.check(repo, sha, default_branch, is_default_head)
except Exception as e: # noqa: BLE001 — the guard must never break CI signals
print(f" {repo}@{sha[:7]} compute-guard skipped ({type(e).__name__}: {str(e)[:80]})")
return
if findings is None:
return
state, desc, first = cg.status_for(findings, whole_tree=is_default_head)
st, existing = github("GET", f"/repos/{GH_OWNER}/{repo}/commits/{sha}/statuses?per_page=100")
for s in existing or []: # newest first: compare the latest guard status only
if s["context"] == GUARD_CTX:
if (s["state"], s.get("description")) == (state, desc):
return
break
url = (f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}/{first.path}#L{first.line}"
if first else f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}")
st, _ = github("POST", f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
{"state": state, "context": GUARD_CTX, "description": desc, "target_url": url})
print(f" {repo}@{sha[:7]} {GUARD_CTX} = {state} ({len(findings)} finding(s)) -> {st}")
def main() -> int:
if not (GITEA_TOKEN and GITHUB_TOKEN):
sys.exit("GITEA_ADMIN_TOKEN and GITHUB_TOKEN are required")
failed = 0
for repo in REPOS:
try:
shas = sync_prs(repo)
default_branch, default_head = "main", None
st, br = github("GET", f"/repos/{GH_OWNER}/{repo}")
if st == 200:
default_branch = br["default_branch"]
st, b = github("GET", f"/repos/{GH_OWNER}/{repo}/branches/{default_branch}")
if st == 200:
default_head = b["commit"]["sha"]
shas.append(default_head)
for sha in dict.fromkeys(shas):
post_statuses(repo, sha)
post_compute_guard(repo, sha, default_branch, sha == default_head)
except Exception as e: # one repo's failure must not hide the others'
print(f" FAILED {repo}: {e}")
failed = 1
return failed
if __name__ == "__main__":
sys.exit(main())

24
scripts/promote_to_ci.sh Executable file
View File

@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# promote_to_ci.sh <repo> [workflow-to-disable ...] — pull mirror -> writable CI repo.
# Run ON Veron as root. See docs/CUTOVER.md "Onboarding another private repo".
#
# ⚠️ It DELETES the mirror before importing (Gitea's migrate refuses an existing
# name). If the import then fails, the Windy Git copy is gone until you re-run —
# GitHub and the nightly R2 bundles still hold everything, but check first that
# scripts/import_from_github.py will accept the repo. (2026-09-23: windy-pro was
# deleted this way while the importer still refused it by name.)
set -euo pipefail
set -a; . /srv/windygit/src/.env; set +a
export IMPORT_GITEA_URL=http://localhost:3080
A=http://localhost:3080/api/v1; H="Authorization: token $GITEA_ADMIN_TOKEN"; r=$1; shift
info=$(curl -s -H "$H" $A/repos/windyadmin/$r)
m=$(echo "$info" | python3 -c 'import json,sys;print(json.load(sys.stdin).get("mirror"))')
if [ "$m" = True ]; then
curl -sf -o /dev/null -X DELETE -H "$H" $A/repos/windyadmin/$r
(cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1)
elif [ "$m" = False ]; then echo "$r already writable"; else echo "$r absent -> importing"; (cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1); fi
db=$(curl -s -H "$H" $A/repos/windyadmin/$r | python3 -c 'import json,sys;print(json.load(sys.stdin).get("default_branch","main"))')
for i in $(seq 1 120); do curl -sf -o /dev/null -H "$H" $A/repos/windyadmin/$r/branches/$db && break; sleep 5; done
for w in "$@"; do printf " disable %s: " "$w"; curl -s -o /dev/null -w '%{http_code}\n' -X PUT -H "$H" $A/repos/windyadmin/$r/actions/workflows/$w/disable; done
curl -s -H "$H" $A/repos/windyadmin/$r/actions/workflows | python3 -c 'import json,sys,os;print(" "+os.environ.get("R",""),[(w["path"].split("/")[-1],w["state"]) for w in json.load(sys.stdin).get("workflows",[])])'
echo " default=$db"

49
scripts/rerun_ci.sh Executable file
View File

@@ -0,0 +1,49 @@
#!/usr/bin/env bash
# Re-run a PR's (or branch's) CI on Windy Git. Runs ON Veron 1.
#
# bash scripts/rerun_ci.sh <repo> <branch> <sha-prefix>
#
# Gitea 1.24 has NO rerun API; the web button needs a hub-SSO session as
# windyadmin, which is Grant's identity, so we don't use it. Instead: move the
# Windy Git branch back one commit, let the next sync force-push the GitHub head
# again, and Gitea fires an ordinary push / pull_request_sync event on the SAME
# commit. Every workflow on that event re-runs, not only the failed one.
#
# Safety: refuses unless the branch is exactly at <sha-prefix> (GitHub's head),
# never rewinds while a sync is running (a run already past this repo would
# not push it back), waits for a sync that STARTS after the rewind, and if the
# branch is not verifiably back at <sha-prefix> by the deadline, restores it
# itself, so Windy Git is never left behind GitHub.
set -euo pipefail
repo="${1:?repo}"; branch="${2:?branch}"; want="${3:?sha prefix}"
G="sudo docker exec -u git windy-git-gitea-1 git -C /data/git/repositories/windyadmin/${repo}.git"
head=$($G rev-parse "refs/heads/${branch}")
[[ "$head" == "$want"* ]] || { echo "refusing: ${branch} is at ${head:0:7}, not ${want}"; exit 1; }
parent=$($G rev-parse "${head}^")
# NOT `systemctl is-active`: the sync is Type=oneshot, which reads "activating"
# (exit 3) for its whole run, so is-active says "idle" mid-run.
busy() { case "$(systemctl show windygit-sync -p ActiveState --value)" in
activating|active|deactivating|reloading) return 0;; esac; return 1; }
while busy; do sleep 5; done
$G update-ref "refs/heads/${branch}" "$parent" "$head"
mark=$(awk '{print int($1*1000000)}' /proc/uptime)
echo "rewound ${repo}:${branch} ${head:0:7} -> ${parent:0:7}"
deadline=$(( $(date +%s) + 900 ))
until [ "$(systemctl show windygit-sync -p ExecMainStartTimestampMonotonic --value)" -gt "$mark" ] \
&& [ "$($G rev-parse "refs/heads/${branch}")" = "$head" ]; do
if [ "$(date +%s)" -ge "$deadline" ]; then
$G update-ref "refs/heads/${branch}" "$head" "$($G rev-parse "refs/heads/${branch}")" || true
echo "TIMEOUT: restored ${branch} to ${head:0:7} by hand; NO new run fired"; exit 1
fi
sleep 10
done
echo "restored by sync: ${branch} = ${head:0:7}"
sleep 5
~/bin/wg-q <<SQL
select ar.index, ar.workflow_id, ar.event, ar.status, to_char(to_timestamp(ar.created),'HH24:MI:SS')
from action_run ar join repository r on r.id = ar.repo_id
where r.name = '${repo}' and ar.commit_sha = '${head}' order by ar.id desc limit 6;
SQL

View File

@@ -38,7 +38,17 @@ FAILED=0
# Repos Windy Git tracks FROM GitHub. Remove a repo from this list at the moment
# it flips to Windy-Git-first, or the sync will fight its authors and win.
REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent}"
REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent windy-git windy-chat windy-mail windy-connect windy-drops windy-code-web windy-code windy-traveler windy-translate windytranslate-site windytraveler-site windy-hand windy-cloud-domains windy-cloud-vps windytalk windy-pro}"
# Repos whose TAGS must not reach Windy Git. A tag push fires `on: push: tags`
# workflows; windy-pro's build-electron is a matrix over ubuntu/macos/windows-
# latest, labels no runner here has, so every leg would queue forever (and
# queued jobs are invisible in /actions/tasks). Releases are built elsewhere.
NO_TAGS="${SYNC_NO_TAGS:-windy-pro}"
# `archive/*` branches never reach Windy Git (negative refspec, git >= 2.29).
# They are off-machine safety copies of unpushed work (one-repo doctrine), not
# work in progress: GitHub holds them, and CI time on them is waste.
mkdir -p "$WORK"
log() { printf '[sync %s] %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }
@@ -63,12 +73,30 @@ for r in $REPOS; do
if git --git-dir="$bare" push --quiet --force \
"https://${WG_OWNER}:${GITEA_ADMIN_TOKEN}@${WG}/${WG_OWNER}/${r}.git" \
'+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*' 2>/dev/null; then
'+refs/heads/*:refs/heads/*' '^refs/heads/archive/*' $([[ " $NO_TAGS " == *" $r "* ]] || echo '+refs/tags/*:refs/tags/*') 2>/dev/null; then
log "$r ok (${before:0:7})"
else
log "FAILED push $r -> windy git"; FAILED=1
fi
done
# Jobs that name labels no runner has (ubuntu/macos/windows-latest) would wait
# forever and invisibly; cancel them after 30 min. Never fails the sync.
# Both DB steps go through `docker exec`, which hangs outright while the host
# is in an IO stall (09-23: data2 SMR cliff wedged this sync for 10+ min and
# stopped mirroring + the bridge for every lane). They are optional; mirroring
# and the bridge are not. Bound them so a stuck exec costs one step, not the run.
timeout -k 10 120 bash "$(dirname "$0")/cancel_unrunnable.sh" || log "janitor failed or timed out (non-fatal)"
# Private repos can't run GitHub Actions; mirror their open PRs here so CI
# fires, and post the verdicts back to GitHub as commit statuses.
if ! python3 "$(dirname "$0")/pr_status_bridge.py"; then
log "FAILED pr status bridge"; FAILED=1
fi
# CI telemetry -> admin.windyword.ai (shapes declared with Windy Telemetry 40).
# Sends nothing until WINDYGIT_TELEMETRY_TOKEN is set; never fails the sync.
timeout -k 10 180 python3 "$(dirname "$0")/telemetry_emit.py" || log "telemetry emit failed or timed out (non-fatal)"
[[ "$FAILED" -ne 0 ]] && { log "COMPLETED WITH FAILURES"; exit 1; }
log "all repos in step with GitHub"

385
scripts/telemetry_emit.py Normal file
View File

@@ -0,0 +1,385 @@
#!/usr/bin/env python3
"""Emit Windy Git CI telemetry to admin.windyword.ai (Windy Telemetry 40's ledger).
Runs on Veron after every sync (root; reads the gitea DB via `docker exec`).
Shapes are declared with Telemetry 40 (2026-09-23) — do not add keys or enum
values without re-declaring: a declared family quarantines any row that
doesn't match.
ci.run one row per FINISHED job, exactly once — cursor on
(finish time, job id) in STATE; jobs finish out of id order
service.health one row per invocation: CI plane counts for the interval
Privacy: ids, names of repos/jobs, codes, counts, durations. No commit
messages, no logs, no author names.
--dry-run print the batch instead of posting (and don't advance STATE)
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
import urllib.error
import urllib.request
from datetime import UTC, datetime
INGEST = os.environ.get("TELEMETRY_INGEST_URL", "https://admin.windyword.ai/v1/events")
TOKEN = os.environ.get("WINDYGIT_TELEMETRY_TOKEN", "")
STATE = os.environ.get("TELEMETRY_STATE", "/var/lib/windy-git/telemetry-state.json")
PLATFORM, SERVICE = "windy-git", "ci"
OUTCOME = {1: "success", 2: "failure", 3: "cancelled", 4: "skipped"}
EVENTS = {"push", "pull_request", "pull_request_sync", "schedule", "workflow_dispatch"}
RUNNERS_EXPECTED = 6
def sql(query: str) -> list[dict]:
"""Rows as dicts, via psql's json_agg — no driver needed on the host."""
wrapped = f"select coalesce(json_agg(t), '[]'::json) from ({query}) t;"
out = subprocess.run(
[
"docker",
"exec",
"-i",
"windy-git-db-1",
"sh",
"-c",
'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1',
],
input=wrapped,
capture_output=True,
text=True,
check=True,
).stdout.strip()
return json.loads(out or "[]")
def load_state() -> dict:
try:
with open(STATE) as f:
return json.load(f)
except (OSError, ValueError):
return {}
def iso(epoch: float) -> str:
return datetime.fromtimestamp(epoch, UTC).isoformat().replace("+00:00", "Z")
# ---- push velocity: DETECT + ALERT ONLY (G3.4, 2026-09-23) -----------------
# `git push` goes straight to Gitea and never touches our API, so throttle.py
# cannot see it (NOT_ENFORCED_HERE). Gitea's own `action` table does record
# every push, so we read it here, emit `forge.push_velocity` when an account
# crosses a threshold, and let Telemetry Boss's detector page. Nothing here sits
# in the push path; nothing is ever refused (orchestrator, 09-23).
#
# Thresholds = the STANDARD-band bases from config.py (500 pushes/day; the
# force-push base of 10/day is used for ref deletes, the closest thing we can
# see). EI band multipliers are NOT applied: a platinum agent over 500/day is
# still flagged, for a human to look at, not blocked. Gitea records no
# "forced" flag, so force pushes cannot be told apart from pushes: named, not
# guessed.
PV_RULES = ( # (rule, row key, window_s, threshold)
("pushes_1h", "p1h", 3600, 60),
("pushes_24h", "p24h", 86400, 500),
("ref_deletes_24h", "d24h", 86400, 10),
)
# The GitHub -> Windy Git sync pushes as windyadmin every 5 min, by design.
PV_EXEMPT = {"windyadmin"}
# Gitea op_type: 5 commit push, 9 tag push, 16 tag delete, 17 branch delete.
# One action row per WATCHER is written for each push; user_id = act_user_id
# keeps exactly the actor's own copy.
PV_QUERY = """
select a.act_user_id as uid, u.lower_name as login,
(select el.external_id from external_login_user el
where el.user_id = a.act_user_id order by el.external_id limit 1) as wid,
count(*) filter (where a.op_type in (5, 9) and a.created_unix > {h1}) as p1h,
count(*) filter (where a.op_type in (5, 9)) as p24h,
count(*) filter (where a.op_type in (16, 17)) as d24h,
count(distinct a.repo_id) as repos
from action a join "user" u on u.id = a.act_user_id
where a.created_unix > {h24} and a.user_id = a.act_user_id
and a.op_type in (5, 9, 16, 17)
group by 1, 2"""
def passport_from_login(login: str) -> str | None:
"""agent-et26abcd1234 -> ET26-ABCD-1234 (repos.py _owner_login, reversed)."""
m = re.fullmatch(r"agent-([a-z0-9]{4})([a-z0-9]{4})([a-z0-9]{4})", login)
return "-".join(g.upper() for g in m.groups()) if m else None
def push_velocity_events(rows: list[dict], now: float, alerted: dict) -> tuple[list[dict], dict]:
"""(events, alerted') — one row per account per rule per window while over.
`alerted` maps "<uid>:<rule>" -> epoch of the last row. An account still over
the line is re-reported once per window, not every 5 minutes; one that drops
back under is forgotten, so a later burst reports again.
"""
events, keep = [], {}
for r in rows:
login = str(r["login"])
if login in PV_EXEMPT:
continue
agent = login.startswith("agent-")
for rule, key, window, limit in PV_RULES:
n = int(r[key])
if n <= limit:
continue
k = f"{r['uid']}:{rule}"
last = alerted.get(k)
if last is not None and now - float(last) < window:
keep[k] = last
continue
keep[k] = now
ev = {
"ts": iso(now),
"platform": PLATFORM,
"service": "forge",
"event_type": "forge.push_velocity",
"metadata": {
"rule": rule,
"window_s": window,
"count": n,
"threshold": limit,
"repos": int(r["repos"]),
"gitea_user_id": int(r["uid"]),
},
}
# Actor rule (telemetry UPDATE 2): agent/human rows MUST carry an
# actor_id. Humans sign in to the forge only via Windy SSO, so the
# external login id IS their windy_identity_id. No id we can prove
# -> actor_type system + metadata.caller, never an invented id (I-12).
actor_id = passport_from_login(login) if agent else (r.get("wid") or None)
if actor_id:
ev["actor_type"], ev["actor_id"] = ("agent" if agent else "human"), str(actor_id)
else:
ev["actor_type"] = "system"
ev["metadata"]["caller"] = "unknown"
events.append(ev)
return events, keep
def main() -> int:
dry = "--dry-run" in sys.argv
state = load_state()
now = time.time()
since = float(state.get("last_ts", now - 300))
# Cursor = (finish time, job id), NOT job id alone: jobs finish out of id
# order, so an id high-water mark silently drops every long job that started
# before the mark and finished after it (Telemetry Boss caught this: 43
# finished vs 8 ci.run rows). Finish time = stopped, or updated for jobs
# Gitea/the janitor skipped without a stop time.
if "last_fin" in state:
last_fin, last_id = int(state["last_fin"]), int(state["last_id"])
else: # first run or pre-cursor state: start now, never replay history
last_fin, last_id = int(state.get("last_ts", now)), 0
cutoff = int(now) - 5 # leave the current second alone; late writers land next run
FIN = "coalesce(nullif(j.stopped, 0), j.updated)"
jobs = sql(f"""
select j.id, j.name as job, j.status, j.started, j.stopped, {FIN} as fin,
p.lower_name as repo, p.default_branch, r.workflow_id, r.event,
r.ref, r.index as run, left(r.commit_sha, 7) as sha
from action_run_job j
join action_run r on r.id = j.run_id
join repository p on p.id = r.repo_id
where j.status in (1, 2, 3, 4)
and ({FIN}, j.id) > ({last_fin}, {last_id})
and {FIN} <= {cutoff}
order by {FIN}, j.id
limit 2000""")
try: # posted_to_github: the bridge's own rules, from the same checkout
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import pr_status_bridge as bridge
except Exception: # noqa: BLE001
bridge = None
events = []
for j in jobs:
ref = j["ref"] or ""
if ref.startswith("refs/pull/"):
kind = "pr"
elif ref == f"refs/heads/{j['default_branch']}":
kind = "default"
else:
kind = "other"
# Gitea stores whole seconds; duration_ms is seconds*1000 (so 10000 = 10 s).
dur = (j["stopped"] - j["started"]) * 1000 if j["started"] and j["stopped"] else None
ev = {
"ts": iso(j["stopped"]),
"platform": PLATFORM,
"service": SERVICE,
"event_type": "ci.run",
"actor_type": "system",
"metadata": {
"repo": j["repo"],
"workflow": (j["workflow_id"] or "").removesuffix(".yml").removesuffix(".yaml"),
"job": j["job"],
"outcome": OUTCOME[j["status"]],
"event": j["event"] if j["event"] in EVENTS else "other",
"branch_kind": kind,
"run": j["run"],
"sha": j["sha"],
},
}
if dur is not None and dur >= 0:
ev["duration_ms"] = int(dur)
if bridge is not None:
wf = ev["metadata"]["workflow"]
ev["metadata"]["posted_to_github"] = bool(
j["repo"] in {r.lower() for r in bridge.REPOS}
and kind in ("default", "pr")
and j["status"] != 4
and not bridge.NO_DAEMON_JOB.search(j["job"])
and f"{wf}/{j['job']}" not in bridge.NON_BLOCKING.get(j["repo"], set())
)
events.append(ev)
# --- heartbeat: counts since the previous invocation --------------------
# Interval counts come from EXACTLY the rows emitted above, so
# sum(jobs_finished) over any window == count(ci.run) in it, by construction.
h = sql(f"""
select
(select count(*) from action_run_job where status in (5, 7)) as jobs_waiting,
(select count(*) from action_run_job where status = 6) as jobs_running,
(select count(*) from action_runner where deleted is null and last_online >= {int(now) - 120}) as runners_online,
(select coalesce(extract(epoch from now())::bigint - min(created), 0)
from action_run_job where status in (5, 7)) as oldest_waiting_s""")[0]
h["jobs_finished"] = len(jobs)
h["jobs_failed"] = sum(1 for j in jobs if j["status"] == 2)
h["jobs_cancelled"] = sum(1 for j in jobs if j["status"] == 3)
meta = {k: int(v) for k, v in h.items()}
meta["interval_s"] = int(now - since) # ecosystem-standard key
# UPDATE 7. Quarantines seen on earlier sends (the ledger answers 202 anyway)
# are carried in the state file until a heartbeat reports them. Dropped is 0
# by construction: a failed send keeps the cursor and the spool, so every row
# is re-sent next run (a partial failure can duplicate, never lose).
meta["telemetry_quarantined"] = int(state.get("quarantined_unreported", 0))
meta["telemetry_dropped"] = 0
for k in ("repos_synced", "repos_sync_failed", "statuses_posted", "bridge_errors"):
v = os.environ.get(f"TELEMETRY_{k.upper()}")
if v is not None and v.isdigit(): # absent = couldn't count; never invent 0
meta[k] = int(v)
events.append(
{
"ts": iso(now),
"platform": PLATFORM,
"service": SERVICE,
"event_type": "service.health",
"actor_type": "system",
"metadata": meta,
}
)
# Isolated: a failing push-velocity query must never cost the ci.run rows.
pv_alerted = state.get("pv_alerted", {})
try:
pv_rows = sql(PV_QUERY.format(h1=int(now) - 3600, h24=int(now) - 86400))
pv_events, pv_alerted = push_velocity_events(pv_rows, now, pv_alerted)
except (subprocess.CalledProcessError, ValueError, KeyError) as e:
print(f"[telemetry] push velocity check FAILED (non-fatal): {type(e).__name__}")
pv_events = []
for e in pv_events:
m = e["metadata"]
print(f"[telemetry] WARNING push velocity: gitea user {m['gitea_user_id']} "
f"{m['rule']} = {m['count']} > {m['threshold']}")
events += pv_events
# ci.job_cancelled: spooled by the janitor (cancel_unrunnable.sh), one JSON per job.
spool = os.environ.get("JANITOR_SPOOL", "/var/lib/windy-git/janitor-cancelled.jsonl")
spooled = 0
try:
with open(spool) as f:
for line in f:
try:
m = json.loads(line)
except ValueError:
continue
events.append(
{
"ts": iso(now),
"platform": PLATFORM,
"service": SERVICE,
"event_type": "ci.job_cancelled",
"actor_type": "system",
"metadata": {
k: m[k]
for k in ("repo", "workflow", "job", "reason", "runs_on", "waited_s")
},
}
)
spooled += 1
except OSError:
pass
if dry:
out = os.environ.get("TELEMETRY_DRY_OUT")
if out:
with open(out, "w") as f:
json.dump({"events": events}, f)
else:
print(json.dumps({"events": events}, indent=1)[:4000])
print(f"[telemetry] DRY RUN: {len(events)} events ({len(jobs)} ci.run)")
return 0
if not TOKEN:
print("[telemetry] WINDYGIT_TELEMETRY_TOKEN unset — not sending (not a failure)")
return 0
quarantined = 0
for i in range(0, len(events), 500):
req = urllib.request.Request(
INGEST,
data=json.dumps({"events": events[i : i + 500]}).encode(),
method="POST",
headers={
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
"User-Agent": "windy-git-telemetry/1",
},
)
try:
with urllib.request.urlopen(req, timeout=30) as r:
body = r.read()
if r.status >= 300:
raise urllib.error.HTTPError(
INGEST, r.status, body[:300].decode(errors="replace"), None, None
)
try:
resp = json.loads(body or b"{}")
except ValueError:
resp = {}
q = resp.get("quarantined") if isinstance(resp, dict) else None
if isinstance(q, int) and q > 0:
quarantined += q
reasons = "; ".join(map(str, resp.get("rejections") or [])) or "no reason given"
print(f"[telemetry] WARNING {q} row(s) QUARANTINED by the ledger: {reasons}")
except urllib.error.HTTPError as e:
print(f"[telemetry] FAILED ingest HTTP {e.code}: {e.read()[:200]!r}")
return 1 # state NOT advanced: the same rows retry next run
except urllib.error.URLError as e:
print(f"[telemetry] FAILED ingest: {e.reason}")
return 1
if spooled:
open(spool, "w").close() # only after every batch was accepted
os.makedirs(os.path.dirname(STATE), exist_ok=True)
new_fin, new_id = (jobs[-1]["fin"], jobs[-1]["id"]) if jobs else (last_fin, last_id)
with open(STATE + ".tmp", "w") as f:
json.dump(
{"last_fin": new_fin, "last_id": new_id, "last_ts": now,
"quarantined_unreported": quarantined, "pv_alerted": pv_alerted},
f,
)
os.replace(STATE + ".tmp", STATE)
print(f"[telemetry] sent {len(events)} events ({len(jobs)} ci.run)")
return 0
if __name__ == "__main__":
sys.exit(main())