From dfe5543edaeb9457d3a1da687c309995c0eb1546 Mon Sep 17 00:00:00 2001 From: Grant Whitmer Date: Wed, 23 Sep 2026 03:19:12 -0400 Subject: [PATCH] docs: runbook + AGENTS match reality (item 5 of the launch bar) RUNBOOK-VERON: deploy uses fetch + merge --ff-only and api-only rebuilds (the old text used git pull, contradicting its own warning); new sections for host timers, CI (6 runners x1, windyadmin-scoped, 90m ceiling, queue truth in the gitea DB, logs in R2), sign-in posture and break-glass; standing checkout = OC5. AGENTS.md no longer says GENESIS / no code. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 14 ++++++++---- docs/RUNBOOK-VERON.md | 51 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0ee9747..4076e62 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,11 +2,17 @@ Read this before touching anything. Then read `DNA_STRAND_MASTER_PLAN.md`, which is the source of truth. -## Current state +## Current state (2026-09-23) -**GENESIS.** No code. No `make dev` yet — building it is codon **G0.8**. +**LIVE on Veron 1** — `app.windygit.com` (Gitea 1.24.6, Windy SSO only), +`api.windygit.com` (our plane: humans via hub JWKS, agents via Eternitas EPT), +`models.windygit.com`. Strands G0–G5, G7, G11 done; see the plan for the rest. -The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Cloudflare Tunnel), then **G2** (Gitea, stock and branded), then **G3** (identity), then **G4** (storage). G0–G4 are sequential. G5–G12 are concurrent once G4 lands. +It is also **the permanent CI for the private platform repos** (GitHub Actions +cannot run on them): `scripts/sync_from_github.sh` + `scripts/pr_status_bridge.py`, +onboarding in `docs/CUTOVER.md`, operations in `docs/RUNBOOK-VERON.md`. + +Standing dev checkout: **OC5 `~/windy-git`**. Deploy copy: Veron `/srv/windygit/src`. ## The rules that will get you reverted if you break them @@ -28,7 +34,7 @@ The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Clo - Errors are 4-field repair pointers: `{code, speak, machine_cause, remediation_tool}`. No exceptions, including validation errors. - Every tool response carries `state_proof` + `next_actions`. - Telemetry `actor_type` comes from the enum `{human, agent, system}`. **`'service'` is not legal** — it 422s and silently drops the whole batch. A sibling service is losing telemetry to exactly this today. -- Runner labels are explicit and pinned. **`ubuntu-latest` is banned** — all four `windy-registry` workflows use it and every run fails. +- Runner labels are explicit and pinned: `[self-hosted, linux, x64]` or `veron-1`. **`ubuntu-latest` is banned** — no runner here has it, so the job queues forever. ## Membrane diff --git a/docs/RUNBOOK-VERON.md b/docs/RUNBOOK-VERON.md index ca0b17d..4cc97a2 100644 --- a/docs/RUNBOOK-VERON.md +++ b/docs/RUNBOOK-VERON.md @@ -1,6 +1,10 @@ # RUNBOOK — Windy Git on Veron 1 (rung R0) -Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron`. Passwordless sudo. +Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron` (or `ts-veron`). Passwordless sudo. + +**Checkouts (one-repo doctrine):** the ONE standing dev checkout is **OC5 +`~/windy-git`** (platform repos live on OC5). `/srv/windygit/src` on Veron is the +*deploy* copy — it holds no local work. Nothing else should exist. ⛔ **Kit 0 is never a host for this service** (D-4). `api/app/main.py` refuses to boot in production if it finds itself on `72.60.118.54`. @@ -15,6 +19,9 @@ boot in production if it finds itself on `72.60.118.54`. | `/etc/cloudflared/config.yml` | tunnel ingress | | `/etc/cloudflared/windy-git.json` | tunnel credentials, mode 600 | | `/srv/windygit/src/.env` | secrets, mode 600, **never committed** | +| `/srv/windygit/git/gitea/conf/app.ini` | Gitea's persisted config — env-to-ini SETS but never UNSETS; edit here when removing a `GITEA__*` var | +| `/srv/windygit/sync/*.git` | bare staging copies the GitHub→Windy Git sync pushes from | +| `/srv/windygit/src/deploy/runner/.env` | `RUNNER_TOKEN` — a **windyadmin user-level** registration token (not instance-level; see CI) | ## Ports — all loopback, on purpose @@ -41,12 +48,15 @@ sudo systemctl status windygit-tunnel ```bash ssh wg-veron -cd /srv/windygit/src && git pull +cd /srv/windygit/src && git fetch origin && git merge --ff-only origin/main # READ the output export COMMIT_SHA_BUILD=$(git rev-parse HEAD) BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) -sudo -E docker compose up -d --build +sudo -E docker compose up -d --build --no-deps api # API only: no forge restart curl -s https://api.windygit.com/version # MUST equal git rev-parse HEAD ``` +A Gitea config change (compose `GITEA__*`) needs `sudo docker compose up -d --no-deps gitea` +— a ~6 s forge outage; running CI jobs survive it. Check `app.ini` afterwards. + ⚠️ **Never `git pull -q` in a deploy script.** `-q` hides *errors*, not just noise. On 2026-08-14 a divergent branch made `pull -q` fail silently and the "deploy" ran for 20 minutes against stale code while reporting success. Use @@ -72,6 +82,41 @@ curl -sI https://app.windygit.com/ | head -1 # Gitea, 200 sudo ss -tlnp | grep -E "3080|8600" # both must be 127.0.0.1 ``` +## Timers (host systemd units — the sync timer is NOT in the repo) + +| Unit | Cadence | Does | +|---|---|---| +| `windygit-sync.timer` | every 5 min (`OnUnitActiveSec`) | GitHub → Windy Git for `REPOS` in `scripts/sync_from_github.sh`, then `scripts/pr_status_bridge.py` (mirror PRs + GitHub commit statuses). A manual `systemctl start` RESETS the 5-min clock. | +| `windygit-backup.timer` | nightly | `git bundle` + pg_dump → R2, 30-day retention | +| `windygit-ci-prune.timer` | every 6 h | `deploy/runner/prune.sh` — CI dind storage, 60 GB cap | +| `windygit-tunnel.service` | always | the only ingress | + +## CI (Gitea Actions) — see `docs/CUTOVER.md` for onboarding a repo + +- **Six runners × capacity 1** (`deploy/runner/docker-compose.yml`), one shared + dind capped at 12 cores / 64 GB. Capacity >1 in one runner shares + `/root/.cache/act` between jobs and races (`lstat …: no such file`). +- **Runners are scoped to the `windyadmin` user** (`action_runner.owner_id=1`), + so only first-party repos run. A repo owned by anyone else — a plane-created + agent or `u-system` repo — gets NO runner. Re-registrations inherit this + because `RUNNER_TOKEN` is user-level. +- Job ceiling 90 min (`config.yaml` `runner.timeout`); a `config.yaml` change + needs each runner restarted **while idle** — `compose up -d` won't recreate it. +- `/actions/tasks` lists only PICKED-UP jobs. Queue truth is `action_run_job` + in the `gitea` DB: `sudo docker exec -i windy-git-db-1 psql -U windygit -d gitea` + (status 1 ok · 2 fail · 3 cancelled · 4 skipped · 5 waiting · 6 running). +- Job logs are in R2, not on disk. `GET /api/v1/repos/{o}/{r}/actions/jobs/{JOB_ID}/logs` + takes the `action_run_job` id, not the task id. + +## Sign-in posture + +- Windy SSO only: password + passkey forms OFF, `ACCOUNT_LINKING=login`, + **auto-registration OFF** — opening the forge to non-Grant users is a §7 + Grant decision. +- **Break-glass:** `sudo docker exec -u git windy-git-gitea-1 gitea admin user generate-access-token --username windyadmin --token-name --scopes --raw` + (delete it after: `delete from access_token where name=''` in the gitea DB — + Gitea refuses token management over token auth). + ## Troubleshooting **A hostname returns 530 or won't resolve** — the tunnel is down. `sudo systemctl