ci.run (one row per finished job, exactly once via a high-water mark;
branch_kind default|pr|other so the dashboard can show "main is red") and
service.health (interval counts: finished/failed/cancelled, waiting,
running, runners online, oldest wait). Shapes declared with Windy
Telemetry 40; sends nothing until WINDYGIT_TELEMETRY_TOKEN exists.
State is only advanced after a 2xx, so a failed post retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a needed job fails, Gitea leaves dependants BLOCKED (7) even after
the run finishes; eternitas build jobs sat there 8h. Mark them skipped
(what GitHub shows) once the run is done and 30 min have passed.
Found by the new telemetry dry run (oldest_waiting_s = 29160).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
windy-pro alone left ~4 jobs per run waiting forever (build-electron on
macos/windows/ubuntu-latest, deploy if:false): Gitea evaluates job if:
only at pick time, the labels do not exist here, and waiting jobs are
invisible in /actions/tasks. 37 such jobs across 10 runs today. After
30 min they are cancelled and the run status recomputed. Runs each sync,
non-fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build-desktop, test-installer and reality-check still run on Windy Git
and stay visible there, but the bridge no longer posts them to GitHub, so
they cannot turn windy-pro's combined status red. Windy Git side only;
the desktop code is Grant's to fix. BRIDGE_NON_BLOCKING, per repo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kit-army-config (the lockbox) and every *-soul / anima repo carry
credentials; the nightly R2 bundles are unencrypted, so the R2 key was a
key to every secret. Excluded by name (BACKUP_EXCLUDE); they are backed up
encrypted by the Windy Drops lane (restic) and stay mirrored on Veron.
Behavioural test runs the script's own exclusion function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
windy-mind has been writable + CI on Windy Git since 08-13 (deploy.yml
disabled, uv pinned); it only lacked GitHub commit statuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
archive/<machine>-<date>/<branch> are off-machine safety copies of
unpushed work (one-repo doctrine). GitHub holds them; running CI on them
is waste. Negative refspec ^refs/heads/archive/* on the push (git 2.43
on Veron). Requested by 8c for windy-pro's Mac mini archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- deploy/systemd/: sync/backup timers+services, tunnel, and the windy-job
heartbeat drop-ins (silent-failure audit). They existed only on Veron,
the same drift that left the runbook wrong. GITHUB_TOKEN is stripped
(repo is public); it stays in the root-only unit on the host.
- sync: SYNC_NO_TAGS (default windy-pro). build-electron fires on v* tags
and targets ubuntu/macos/windows-latest, labels no runner has, so it
would queue forever, invisibly. Desktop releases are built elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- import_from_github.py no longer refuses windy-pro: lane 8c audited all
14 checkouts (WINDYPRO_CHECKOUTS.md), GitHub main is canonical.
- sync + bridge: windy-cloud-domains, windy-cloud-vps, windytalk (default
branch master), windy-pro; windy-cloud-sites added to the bridge (it was
synced but never bridged).
- scripts/promote_to_ci.sh: the mirror->writable procedure as one script,
with the delete-before-import hazard documented.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven HTTP-level tests through the real route: sha256= prefix and bare
digests accepted, digest of re-serialised JSON refused, forged/wrong-key/
missing signatures refused, unset secret -> 503, a revocation with a bad
signature never reaches the handler, the reachability ping never acts.
Mutation-checked: dropping the prefix strip fails the behavioral test
while the old string-grep invariant still passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
windy-hand promoted to writable + bridged. windy-agent is PUBLIC and its
GitHub Actions already run on Veron's GitHub runner; running its 3-version
pytest matrix here too was pure duplicate load (3 x ~4.5 cores for 25+
min, host load 64 on 24 cores). Actions are now off for windy-agent on
Windy Git; it stays in REPOS as a synced copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured: an unprivileged job container inside the CI dind could open SSH,
Ollama and dev servers on Veron (192.168.1.73) and the rest of the LAN,
WireGuard and Tailscale — lateral movement for any malicious dependency,
no escape needed. egress.sh (idempotent; windygit-ci-egress.service at
boot) hooks the jobs bridge: runner<->dind, replies, DNS and public
egress allowed; RFC1918, CGNAT, link-local and the host itself dropped.
Verified from a job container: 6/6 private targets blocked, DNS,
internet and the public forge OK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RUNBOOK-VERON: deploy uses fetch + merge --ff-only and api-only rebuilds
(the old text used git pull, contradicting its own warning); new sections
for host timers, CI (6 runners x1, windyadmin-scoped, 90m ceiling, queue
truth in the gitea DB, logs in R2), sign-in posture and break-glass;
standing checkout = OC5. AGENTS.md no longer says GENESIS / no code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promoted to writable; the two sites' CF deploy.yml disabled (they would
need the CF god token in a job container). All three only have
ubuntu-latest workflows today, so nothing runs until their lanes switch
runs-on to [self-hosted, linux, x64].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any Windy Word account (public signup, unverified email) auto-registered a
forge account on first sign-in — reproduced with a throwaway account —
and the act runners are instance-wide, so a stranger's workflow would run
on Veron's privileged dind beside the R2 god token. §7 makes opening the
forge to non-Grant users Grant's call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hub_jwt: aud list is ["windy_git"] (contract v1 array). Dropped
"windy-git": that is Gitea's OIDC client_id, so a forge id_token would
have passed the aud check. `type: human` is now REQUIRED (id_tokens have
none), which makes accepting the discovery-URL issuer safe.
- runner job ceiling 30m -> 90m: eternitas's serial pytest is ~50 min and
would have been killed mid-suite.
- eternitas (private) added to the GitHub status bridge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The human path refused every token in production (503
human_signin_not_ready) because no verifier existed. api/app/hub_jwt.py
verifies hub access tokens against account.windyword.ai's JWKS:
- RS256 only (closes alg:none and HS256-with-public-key confusion)
- iss must be "windy-identity" — what hub ACCESS tokens carry (observed
live); id_tokens (discovery-URL issuer) are not accepted as bearers
- aud optional today, must name Windy Git when present; hub_require_aud
flips it mandatory once the hub emits it. PyJWT's own aud check is off
on purpose: it rejects ANY aud-bearing token when no audience is given.
- type must be human; identity = windy_identity_id, never sub (per-row id)
- production verifies even if require_verified_jwt is off
11 behavioral tests sign real RS256 tokens with a local key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- runner-5/6: 50+ jobs were queued with ~11 private repos onboarded. dind
keeps the 12-core ceiling, so this adds concurrency, not CPU.
- Gitea: password + passkey sign-in forms off (break-glass = CLI), and
ACCOUNT_LINKING auto -> login. auto linked any hub login whose email
matched an existing account, and SITE ADMIN windyadmin carries Grant's
email. Grant is linked by the hub's stable sub, which matches first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All four promoted from pull mirrors to writable. windy-code keeps only
canonical-domains-lint active: its other 15 workflows target hosted
macOS/Windows/ubuntu-latest runners and would queue forever here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- deploy/runner/prune.sh + windygit-ci-prune.timer (6h): age-based prune
of the CI-only dind (containers, finished-job volumes, images/builder
cache >7d) plus a hard 60 GB cap. Only that daemon, over its own TCP
socket; never the host's Docker. It was 38 GB and unbounded — the same
class of growth that filled Kit 0 on 09-01.
- pr_status_bridge: jobs named *docker* are not posted. Job containers
have no daemon by design (I-5), so they are red on every commit; a
permanent red X teaches everyone to ignore red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
windy-connect promoted from pull mirror to writable (release.yml, which
publishes to PyPI on tag push, disabled — the sync pushes tags).
windy-search was already writable; its scheduled drift-check is disabled
because it now runs as cron on Kit 0. Both added to BRIDGE_REPOS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Four runners x capacity 1 instead of one x capacity 4. Concurrent jobs in
one act_runner share /root/.cache/act; a refresh racing a copy killed 3 of
windy-chat's ~20 jobs at setup-node (lstat ... no such file). Separate
processes have separate caches. Same parallelism, same capped dind.
- Behavioral tests for pr_status_bridge (latest verdict wins, no reposting,
skipped never painted green, fork PRs never run, pagination, PR lifecycle).
- import_from_github.py reads IMPORT_GITEA_URL, not GITEA_BASE_URL: sourcing
the deploy .env pointed it at http://gitea:3000 and it died on DNS after the
mirror it replaces had already been deleted.
- CUTOVER.md: the private-repo CI path, onboarding steps, and the
/actions/tasks-hides-queued-runs trap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub Actions can't run on the private platform repos. Windy Git already
has their code and a working runner, so:
- windy-chat and windy-mail were read-only pull mirrors (which can never
run Actions); they are now writable, deploy.yml/build-image.yml disabled,
and synced from GitHub like the others.
- scripts/pr_status_bridge.py mirrors open same-repo GitHub PRs into Windy
Git (so pull_request workflows fire) and posts each job's result back as
a GitHub commit status (windy-git/<workflow>/<job>) on PR heads and the
default-branch head. Fork PRs are never run. Runs after every sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
windygit-tunnel had crash-looped ~91k times: another project's
cornercall-tunnel holds 127.0.0.1:2000, and cloudflared exits when it
cannot bind its metrics port. Ingress only survived because a stray
cloudflared.service ran the same config. That unit is now disabled and
/etc/cloudflared/config.yml uses metrics 127.0.0.1:2001.
Also add windy-git to the GitHub->Windy Git sync list; its self-hosted
copy was stuck 3 commits behind (only check + canary workflows, no
deploys, so syncing is safe).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second root cause found and fixed: act_runner 0.2.11 predates
`runs.using: node24`, so any repo on actions/checkout@v5 died before its first
step. Bumped to 0.6.1; Windy-Clone went 4/4 red to 4/4 green.
Also upgrades the act-cache note from "watch item" to a confirmed job failure
(lstat on a vanished file mid-tar), with the wipe command and the annotated-tag
dead end that looks like a wrong checkout but isn't.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0.2.11's bundled act only knows runs.using node12/node16/node20, so any repo
pinning a current action major (actions/checkout@v5, actions/setup-python@v6)
fails before its first step with "The runs.using key in action.yml must be one
of: [...], got node24". Windy-Clone is how this surfaced.
Verified node24 is absent from the 0.2.11 binary and present in 0.6.1, and that
every key in deploy/runner/config.yaml still exists in 0.6.1's schema.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The localhost->postgres fix was correct but was never what failed these jobs;
they died at step 2. setup-uv v4+ resolves "latest" through GITHUB_API_URL,
which act_runner points at our own forge, so it 404s and every later step is
skipped by success(). Pinned an explicit uv version across 11 repos.
Also records the diagnosis traps that cost the most time: Gitea's job status
enum (1=success, 2=failure), act misattributing the error to the previous step,
the jobs-log API needing a repo-matched id, and Secure cookies defeating a
loopback curl login.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
windy-registry's postgres integration went failure -> success, proving the fix.
windy-mind and WindyCloud still fail for a cause I could not determine: the
jobs API returns 'job not found' for the ids the runs report, so logs were not
retrievable that way. Next session should read them from the Gitea web UI.
Records the trap that the three repos did NOT share one pattern — a naive
localhost->postgres swap would have left WindyCloud on port 15432.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
State, the immediate task (three-repo localhost->service-name CI fix), the traps
already paid for, and a copy-paste prompt.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
The probe's own log was never retrievable through the jobs API, but the
question it asked was answered better by a direct comparison of two real
workflows on the same runner and image:
windy-git gate @postgres:5432 -> passes its migration round-trip
eternitas migrations @localhost:5432 -> failed
Also scopes test_g73 to workflows that actually run Python. It failed the probe
for not pinning a version when the probe only shelled out to psql — the test
being wrong rather than the workflow.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
Several migrated workflows hardcode postgres at localhost:5432, which is
correct on GitHub-hosted runners (services are port-mapped to the VM) and
suspect on Gitea Actions (the job runs IN a container, so localhost is the job).
Prove which form works before rewriting anyone's workflow.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
The front end was 100% stock Gitea: green teacup, "Gitea: Git with a cup of
tea", "A painless, self-hosted Git service". G2.3 was specified in the plan with
an acceptance test and never executed, and nothing enforced it.
Now: Windy Git name, wind-mark logo, brand-blue accent, and a landing page that
says what this actually is. Uses Gitea's SUPPORTED surface (custom templates +
public assets) so upstream upgrades keep arriving — no source modified (D-2/I-1).
Two traps this cost, both now documented and tested:
1. GITEA__DEFAULT__APP_NAME does not work. Gitea reads APP_NAME from the TOP
LEVEL of app.ini; the env var created a literal [default] section that Gitea
ignores, so the installer's stock APP_NAME kept winning while the config
looked correct. The env-to-ini pass also APPENDED a second APP_NAME rather
than replacing the first — a new variant of the documented G4A.3 trap.
2. Cloudflare caches /assets/* for 6h and no token in this stack can purge, so
the new logo and CSS were invisible while being correct at origin. Brand
assets now carry a VERSION IN THE FILENAME; bump it on every change.
Committed with an idempotent apply.sh, because applying it straight to Veron's
disk first was itself the config-drift trap this project documents: a rebuild
would have silently reverted to stock Gitea.
85 tests green.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
I reintroduced the exact defect I had just criticised. ACTION_BASE listed
"push" and "push.force", but git push goes straight to Gitea over HTTPS and
never touches this API — so nothing records a push, a count would be zero
forever, and enforce() would look up a limit, count nothing, and allow
everything. A silent no-op wearing the costume of a control, made worse by a
config name that implies the protection exists.
Split into ACTION_BASE (actually enforced: repo.create, grant.create) and
NOT_ENFORCED_HERE (push, push.force) with the reason and the remedy written
down: enforcing push velocity needs a Gitea-side pre-receive or push webhook
reporting into agent_actions.
enforce("push") now raises rather than silently allowing, and a test asserts the
two sets stay disjoint.
Found by auditing whether the auth fix could be walked around — every
/api/v1/repos/* route does require a caller, and the only unauthenticated
endpoints are /health, /version and the HMAC-verified webhook.
83 tests green.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
Two deploy traps paid for on 2026-08-14: 'git pull -q' hid a divergent-branch
error so a deploy ran against stale code while reporting success, and the
divergence came from amending a commit a deploy checkout already tracked.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
/version went null after a deploy — the exact "service cannot name its own
commit" defect this project was built to prevent, caught by its own honesty
check.
Cause: the sed replaced "" with "" (a no-op when COMMIT_SHA is empty) and the
grep then matched that same empty string, so the guard verified nothing. A build
with no COMMIT_SHA passed and shipped a container reporting commit_sha: null.
Now the build fails loudly instead.
Second cause of the stale deploy, and it was mine: an earlier `git commit
--amend` + force-push rewrote history the Veron deploy checkout was already
sitting on, leaving it divergent so `git pull -q` failed SILENTLY (-q hid
"Need to specify how to reconcile divergent branches"). Two lessons: do not
force-push a branch a deploy checkout tracks, and do not pull with -q in a
deploy script.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
looks_like_ept used jwt.get_unverified_header, which validates the WHOLE token
and therefore rejects anything with a malformed signature segment. Routing
consequently depended on signature well-formedness: an EPT-shaped token with a
bad signature fell through to the HUMAN path, where it was refused for the wrong
reason and — with require_verified_jwt off (dev) — could have been read as a
human identity via its `sub` claim.
Now the header segment is decoded directly, so routing depends only on what the
token CLAIMS to be; whether it is authentic remains verify_ept's job.
Also routes alg:none to the EPT verifier regardless of typ, since a `none`
token is never valid for any caller. Both forged shapes now return 401
ept_invalid — the honest code — instead of 503 "feature not ready".
Found by noticing a forged EPT returned 503 where the verifier should have
answered 401, rather than accepting "it was refused, close enough".
80 tests green.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
Monkeypatched httpx so resolve_passport sees a real revoked trust body
(status=revoked, band=unproven, allowed=[]) and must raise
PassportNotInGoodStanding — proving the WIRING, not just the decision. This is
the path that stops a validly-signed EPT that outlived its passport's
revocation (~365-day tokens).
Live-confirmed alongside: Eternitas refuses to mint EPTs for revoked bots
("credentials are not issued for non-active bots"), so the only exposure was a
pre-existing token — exactly what this now catches. The active agent's EPT still
returns 200. A fully-live revoked test would require revoking a real fleet
passport (destructive), so the wiring is proven deterministically instead.
79 tests green.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
A revoked passport returns HTTP 200, status=revoked, band=unproven,
allowed_actions=[] (verified live 2026-08-13). resolve_passport keyed refusal
only on HTTP 4xx and band=="untrusted", so it returned band 'unproven' and the
agent was seated. Revocation was NOT enforced on the live auth path at all — and
now that agent auth actually works, a revoked agent could authenticate and act.
Extracts decide_trust(body) -> (band, actions) | raise. Only status=="active"
is allowed; revoked/suspended/frozen/unknown all refuse, fail-closed on the
field that carries the most consequential fact about an identity. The agent call
site turns that into a clean 403 passport_revoked.
This is the REAL revocation gate — the token cannot be un-issued, but its
standing is re-checked on every request, so revocation takes effect on the next
call with no webhook required. The Eternitas webhook remains useful for
invalidating locally-issued credentials/grants (G6.3, not built yet), but it was
never the primary gate and its being unwired is no longer a live exposure.
Behavioral tests: revoked body refused, active accepted, unknown/missing status
fails closed.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
The EPT-shaped forgery is the one that matters after G3.2 — it is what
signature verification actually guards. The JWT-shaped one still exercises the
human gate. Both must_refuse; a 2xx on either pages.
Verified live: forged EPT -> 401 ept_invalid, forged JWT -> 503,
genuine EPT -> 200.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
REOPENS the agent path — but only because possession is now actually proven.
EPT verification (api/app/ept.py): ES256 against Eternitas's published key set
at /.well-known/eternitas-keys. algorithms=["ES256"] makes alg:none and
algorithm confusion unrepresentable rather than merely unlikely; issuer and exp
are enforced by the library; an unknown kid is refused.
Order is deliberate: signature FIRST, trust lookup second. These EPTs live ~365
days and carry rev/tru baked in at issuance, so a year-old "rev: false" proves
nothing — revocation and band still come from a live lookup on every request.
Found while building it: real EPTs put the passport in the "sub" claim. The old
code read "passport"/"sub_passport", which no genuine EPT carries — so real
agents were never recognised and ONLY forged tokens ever authenticated. The
bypass was not just a hole, it was the only thing that worked.
Throttle (api/app/throttle.py): BAND_MULTIPLIER and rate_*_per_day were defined
and read by nothing. Now enforced on repo.create and grant.create, counted
against agent_actions (one source of truth, not a private counter that drifts
from the audit log). Fails CLOSED — a limiter that fails open protects you until
the moment something is wrong. Untrusted band is 403 read-only, not 429, because
"slow down" would be a lie.
Tests: 14 behavioral, signing real ES256 tokens with a locally-generated key so
they exercise the crypto path with no network dependency — genuine tokens
accepted, and alg:none / foreign key / tampered payload / expired / wrong issuer
/ unknown kid / missing claims all refused. 74 green.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
Adds must_refuse checks: a 2xx is the alarm, a 401/403/503 is health. The
forged alg:none agent token is probed every 10 min; if it ever returns 2xx the
canary goes red and pages. Proven both ways — 503 reads ok, a 200 endpoint
alarms 'ACCEPTED — this MUST be refused'. The security property is now enforced
by a running check, not assumed.
Co-Authored-By: Claude (Fable 5) <noreply@anthropic.com>
Verified live 2026-08-13: a forged 'alg:none' token naming a passport lifted
from the logs returned HTTP 200 as that agent. The agent path read the passport
without verifying the EPT signature, asked Eternitas 'is this passport
reputable?', and seated the caller on a yes. That answers reputation, not
possession — anyone who knows a passport number could impersonate that agent on
the public API.
The human path already failed closed for exactly this reason
(require_verified_jwt). The gate was on the wrong path: it sat AFTER the agent
branch returned. The agent path now fails closed too, BEFORE the trust lookup,
so a forged token never even reaches Eternitas. Reopens automatically when the
ES256/JWKS verifier (G3.2/G9.1) exists and this gate consults it.
Adds BEHAVIORAL tests (not string-grep): a forged alg:none token exercised
through the real get_caller must raise, not authenticate. This is the test that
would have caught the bypass; the suite had 86 source-string assertions and
zero that ran the auth decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
131 read-only mirrors (cannot run Actions, zero deploy risk) + 12 writable with
CI and deploys disabled. Total size matches the measured GitHub archive exactly,
which is the confirmation the copy is complete.
Splits the two concerns cleanly: having a copy is safe and should cover
everything now; running code needs judgement and happens per repo.
windy-pro IS included as a mirror — the G11.5 caution is about making it
writable while six checkouts disagree on HEAD, not about holding a read-only
copy. The DR copy is now complete.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
