Compare commits
125 Commits
7711adcb97
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6f19e23eaf | ||
| ef96051d6f | |||
|
|
a0dc6f8568 | ||
|
|
51e0b9d30b | ||
|
|
51777b0ad0 | ||
|
|
cbcb4c206b | ||
|
|
2c62e2834a | ||
| 3a9b7ecab7 | |||
|
|
cd0400c371 | ||
|
|
4e7ab667ed | ||
|
|
f10db19316 | ||
|
|
3503894a1e | ||
|
|
fbab5c4669 | ||
|
|
1b552c0882 | ||
|
|
8ebc18c3bc | ||
|
|
9566826ce9 | ||
|
|
160a3d69ba | ||
|
|
7e28a23c22 | ||
|
|
1da4d39c0b | ||
|
|
53fbcfe78f | ||
|
|
1c552e94de | ||
|
|
8690c4f8d3 | ||
|
|
313f41b49c | ||
|
|
ea02ade0cb | ||
|
|
497222094f | ||
|
|
d28da26000 | ||
|
|
04c857654a | ||
|
|
32512a32fc | ||
|
|
b52e6b4784 | ||
|
|
6440c7d5f4 | ||
|
|
17acae6af6 | ||
|
|
790d794900 | ||
|
|
1b8cebbfe1 | ||
|
|
83fbf1f992 | ||
|
|
9b5334fee7 | ||
|
|
f71a5aab39 | ||
|
|
f219437282 | ||
|
|
b15584d33a | ||
|
|
0a57d96f2e | ||
|
|
8b1ae4ac60 | ||
|
|
14ed23a9fe | ||
|
|
c4799dcc13 | ||
|
|
683c01ec7d | ||
|
|
4bb4893131 | ||
|
|
9e9eb08d96 | ||
|
|
dece301bf0 | ||
|
|
87dcddb87d | ||
|
|
a5f7b036a8 | ||
|
|
255aa58b35 | ||
|
|
e64a1b5fcb | ||
|
|
1bc55eaec9 | ||
|
|
fdb0f5989e | ||
|
|
9be2952ff8 | ||
|
|
fe3bce39ff | ||
|
|
6b0b60eb4a | ||
|
|
60708dd8db | ||
|
|
aedc772d29 | ||
|
|
acb8ec3a16 | ||
|
|
0051c72037 | ||
|
|
4c76b1b12a | ||
|
|
2d4fadb090 | ||
|
|
e3b69fa759 | ||
|
|
4acf50d9ef | ||
|
|
b7a7e94df0 | ||
| c83f808a60 | |||
| eb27e63db3 | |||
| 1c3b5b0638 | |||
| 90643fe48e | |||
| 00ec963f82 | |||
| b5e4eaf57a | |||
| baaa542bae | |||
| 0634a6cb1b | |||
| 1a171eabd5 | |||
| 28c31236b8 | |||
| cd5967031b | |||
| f246417095 | |||
| 50c1464043 | |||
| 7a63f90da3 | |||
| b8f97f0731 | |||
| 95c33c8004 | |||
| d5181f1c6d | |||
| e6530d3171 | |||
| 419443573a | |||
| 4a34b35441 | |||
| dfe5543eda | |||
| 40cb455d0d | |||
| 8c404eb410 | |||
| 5b16114b98 | |||
| 18ea9a4686 | |||
| 32e8ac8474 | |||
| 8e9fa3116c | |||
| 74ad4950b2 | |||
| e7bbf9af51 | |||
| 45686283be | |||
| e4a15869c0 | |||
| dcf9286f16 | |||
| 1b09b9b0d3 | |||
| 390c1e7479 | |||
| 2b30b0ac99 | |||
| cd7dd9b7ae | |||
| 9fc27eabd6 | |||
|
|
db055a1922 | ||
|
|
86326ca6a7 | ||
|
|
e0d5d118be | ||
|
|
01e36155a3 | ||
|
|
fe8f84bbdf | ||
|
|
eae1bff50b | ||
|
|
e7dee39151 | ||
|
|
25368547cb | ||
|
|
c60bfb2b89 | ||
|
|
a0ed4a5ec0 | ||
|
|
830ca48705 | ||
|
|
deb3a8fafc | ||
|
|
587fb05265 | ||
|
|
c96d1d3102 | ||
|
|
79dcea4d3e | ||
|
|
c257cc55c6 | ||
|
|
85fa65a52b | ||
|
|
d8deffe4db | ||
|
|
83047def94 | ||
|
|
711c47d8c9 | ||
|
|
ae6ca8b9d2 | ||
|
|
89723c6ebd | ||
|
|
b2f00821d7 | ||
|
|
51acf9f86e |
@@ -51,9 +51,10 @@ jobs:
|
||||
- name: install
|
||||
run: |
|
||||
python3 --version
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -q --upgrade pip
|
||||
.venv/bin/pip install -e ".[dev]"
|
||||
# From uv.lock, never floating: CI tests exactly what the image ships.
|
||||
# --locked also FAILS if pyproject.toml changed without re-locking.
|
||||
python3 -m pip install -q uv==0.12.5
|
||||
uv sync --locked --extra dev
|
||||
|
||||
- name: lint
|
||||
run: .venv/bin/ruff check api scripts
|
||||
|
||||
14
AGENTS.md
14
AGENTS.md
@@ -2,11 +2,17 @@
|
||||
|
||||
Read this before touching anything. Then read `DNA_STRAND_MASTER_PLAN.md`, which is the source of truth.
|
||||
|
||||
## Current state
|
||||
## Current state (2026-09-23)
|
||||
|
||||
**GENESIS.** No code. No `make dev` yet — building it is codon **G0.8**.
|
||||
**LIVE on Veron 1** — `app.windygit.com` (Gitea 1.24.6, Windy SSO only),
|
||||
`api.windygit.com` (our plane: humans via hub JWKS, agents via Eternitas EPT),
|
||||
`models.windygit.com`. Strands G0–G5, G7, G11 done; see the plan for the rest.
|
||||
|
||||
The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Cloudflare Tunnel), then **G2** (Gitea, stock and branded), then **G3** (identity), then **G4** (storage). G0–G4 are sequential. G5–G12 are concurrent once G4 lands.
|
||||
It is also **the permanent CI for the private platform repos** (GitHub Actions
|
||||
cannot run on them): `scripts/sync_from_github.sh` + `scripts/pr_status_bridge.py`,
|
||||
onboarding in `docs/CUTOVER.md`, operations in `docs/RUNBOOK-VERON.md`.
|
||||
|
||||
Standing dev checkout: **OC5 `~/windy-git`**. Deploy copy: Veron `/srv/windygit/src`.
|
||||
|
||||
## The rules that will get you reverted if you break them
|
||||
|
||||
@@ -28,7 +34,7 @@ The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Clo
|
||||
- Errors are 4-field repair pointers: `{code, speak, machine_cause, remediation_tool}`. No exceptions, including validation errors.
|
||||
- Every tool response carries `state_proof` + `next_actions`.
|
||||
- Telemetry `actor_type` comes from the enum `{human, agent, system}`. **`'service'` is not legal** — it 422s and silently drops the whole batch. A sibling service is losing telemetry to exactly this today.
|
||||
- Runner labels are explicit and pinned. **`ubuntu-latest` is banned** — all four `windy-registry` workflows use it and every run fails.
|
||||
- Runner labels are explicit and pinned: `[self-hosted, linux, x64]` or `veron-1`. **`ubuntu-latest` is banned** — no runner here has it, so the job queues forever.
|
||||
|
||||
## Membrane
|
||||
|
||||
|
||||
@@ -81,7 +81,7 @@ Numbered because code cites them. Changing one requires an ADR that names it.
|
||||
|
||||
1. **I-1 · Gitea is a component, never a merged tree.** Our code lives in our services and calls Gitea's REST API. Any patch to Gitea source lives in `patches/` as a numbered, rebasable diff with a one-line justification, and `make check` fails if `patches/` grows past **3** files without an ADR.
|
||||
2. **I-2 · The membrane is ENUMERATED.**
|
||||
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo).
|
||||
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo). · windy-admin ledger `POST https://admin.windyword.ai/v1/events` (field telemetry, 2026-09-23: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed` — shapes declared with the ledger owner first; no content, no passports, no emails)
|
||||
**Calls in:** `POST /internal/repo-from-folder` (Cloud portal: git-enable a folder) · `POST /internal/mirror-status` (ops).
|
||||
**Events out:** `repo.created`, `repo.pushed`, `release.published`, `model.published`, `ci.completed`.
|
||||
**Events in:** `passport.revoked` (fail-closed), `storage.quota.exceeded`, `identity.created`.
|
||||
|
||||
24
Dockerfile
24
Dockerfile
@@ -10,15 +10,31 @@ WORKDIR /app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends git curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY pyproject.toml ./
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Dependencies come from uv.lock, hash-pinned, never "latest at build time".
|
||||
# Floating installs meant a rebuild could ship different fastapi/starlette/
|
||||
# pydantic than CI tested (Windy Cloud's OpenAPI drift, 09-23). The lock was
|
||||
# cut to exactly what prod ran then. uv only exports; pip installs, so the
|
||||
# image layout (system python, uvicorn on PATH) is unchanged.
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.12.5 /uv /usr/local/bin/uv
|
||||
COPY pyproject.toml uv.lock ./
|
||||
RUN uv export --frozen --no-dev --no-emit-project -o /tmp/requirements.txt \
|
||||
&& pip install --no-cache-dir --require-hashes -r /tmp/requirements.txt \
|
||||
&& rm /tmp/requirements.txt
|
||||
COPY api ./api
|
||||
RUN pip install --no-cache-dir --no-deps -e .
|
||||
|
||||
COPY alembic ./alembic
|
||||
COPY alembic.ini ./
|
||||
COPY scripts ./scripts
|
||||
|
||||
RUN sed -i "s|^BAKED_COMMIT_SHA: str = \"\"|BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"|" api/app/buildinfo.py \
|
||||
# I-12: an EMPTY COMMIT_SHA must fail the build, not sail through it.
|
||||
# Previously the sed replaced "" with "" (a no-op) and the grep then matched
|
||||
# that same empty string, so a build with no COMMIT_SHA passed and shipped a
|
||||
# container reporting commit_sha: null — exactly the "service cannot name its
|
||||
# own commit" defect this project exists to prevent. Caught 2026-08-14 when
|
||||
# /version went null after a deploy.
|
||||
RUN test -n "${COMMIT_SHA}" || (echo "FATAL: COMMIT_SHA build arg is empty (I-12)" && false) \
|
||||
&& sed -i "s|^BAKED_COMMIT_SHA: str = \"\"|BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"|" api/app/buildinfo.py \
|
||||
&& sed -i "s|^BAKED_BUILT_AT: str = \"\"|BAKED_BUILT_AT: str = \"${BUILT_AT}\"|" api/app/buildinfo.py \
|
||||
&& grep -q "BAKED_COMMIT_SHA: str = \"${COMMIT_SHA}\"" api/app/buildinfo.py
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ boot guard in `api/app/main.py` that refuses to start there in production.
|
||||
| 8600 | `windy-git-api` — our plane |
|
||||
| **3080** | Gitea — host 3000 and 3300 are taken by resident projects on Veron 1 |
|
||||
| 5432 | Postgres |
|
||||
| 2000 | cloudflared metrics (probe target) |
|
||||
| 2001 | cloudflared metrics — NOT 2000: `cornercall-tunnel` (another project) takes 2000, and a metrics bind failure kills the whole tunnel |
|
||||
|
||||
## Ingress — Cloudflare Tunnel `windy-git`
|
||||
|
||||
|
||||
125
api/app/auth.py
125
api/app/auth.py
@@ -25,7 +25,10 @@ import httpx
|
||||
from fastapi import Header, Request
|
||||
|
||||
from api.app.config import Settings
|
||||
from api.app.ept import EptInvalid, looks_like_ept, verify_ept
|
||||
from api.app.errors import RepairPointer, passport_unresolvable
|
||||
from api.app.hub_jwt import HubTokenInvalid, verify_hub_token
|
||||
from api.app.telemetry import synthetic_headers
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -72,6 +75,36 @@ BAND_MULTIPLIER: dict[str, float] = {
|
||||
}
|
||||
|
||||
|
||||
class PassportNotInGoodStanding(Exception):
|
||||
"""The passport resolved, but Eternitas does not list it as active
|
||||
(revoked / suspended / frozen / unknown status)."""
|
||||
|
||||
def __init__(self, passport: str, status: str) -> None:
|
||||
self.passport = passport
|
||||
self.status = status
|
||||
super().__init__(f"{passport} status={status!r}")
|
||||
|
||||
|
||||
def decide_trust(body: dict, passport: str) -> tuple[str, tuple[str, ...]]:
|
||||
"""The trust body -> (band, allowed_actions), or refuse.
|
||||
|
||||
THE GATE THAT WAS MISSING. A revoked passport returns HTTP 200 with
|
||||
`status: revoked`, `band: unproven`, `allowed_actions: []` — verified live
|
||||
2026-08-13. The previous code keyed refusal only on HTTP 4xx and on
|
||||
band=="untrusted", so a revoked agent (200, band unproven) authenticated and
|
||||
acted normally. Revocation was not enforced on the live path at all; the
|
||||
webhook that was supposed to be the backup was never the primary gate.
|
||||
|
||||
Only `status == "active"` is allowed. Anything else — including a status
|
||||
Eternitas invents tomorrow — refuses. Fail-closed on the field that carries
|
||||
the most consequential fact about an identity.
|
||||
"""
|
||||
status = str(body.get("status", "")).lower()
|
||||
if status != "active":
|
||||
raise PassportNotInGoodStanding(passport, status or "missing")
|
||||
return body.get("band", "unproven"), tuple(body.get("allowed_actions", ()))
|
||||
|
||||
|
||||
async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tuple[str, ...]]:
|
||||
"""G3.6 — THE STATUS-CODE LAW.
|
||||
|
||||
@@ -93,7 +126,7 @@ async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tupl
|
||||
)
|
||||
|
||||
url = f"{settings.eternitas_base_url}/api/v1/trust/{passport}"
|
||||
headers = {"X-API-Key": settings.eternitas_platform_api_key}
|
||||
headers = {"X-API-Key": settings.eternitas_platform_api_key, **synthetic_headers()}
|
||||
last_status = 0
|
||||
for attempt in range(3):
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(8.0, connect=3.0)) as client:
|
||||
@@ -105,8 +138,9 @@ async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tupl
|
||||
continue
|
||||
last_status = r.status_code
|
||||
if r.status_code == 200:
|
||||
body = r.json()
|
||||
return body.get("band", "unproven"), tuple(body.get("allowed_actions", []))
|
||||
# decide_trust raises PassportNotInGoodStanding on a non-active
|
||||
# status; that propagates past the retry loop as a hard refusal.
|
||||
return decide_trust(r.json(), passport)
|
||||
if r.status_code in (400, 404):
|
||||
# Malformed or not-issued. Refuse immediately — retrying cannot help
|
||||
# and pretending it might is how a soft-allow gets written.
|
||||
@@ -159,42 +193,77 @@ async def get_caller(
|
||||
token = authorization.split(" ", 1)[1].strip()
|
||||
|
||||
# --- agent (Eternitas EPT) --------------------------------------------
|
||||
# An EPT names its passport; the trust API is the authority on whether that
|
||||
# passport may act. We never read a band out of the token itself.
|
||||
passport = _unverified_claim(token, "passport") or _unverified_claim(token, "sub_passport")
|
||||
if passport:
|
||||
band, actions = await resolve_passport(settings, passport)
|
||||
# Possession FIRST, reputation second. The signature proves the caller holds
|
||||
# this passport; the trust lookup then says what it may do. Doing only the
|
||||
# second was the 2026-08-13 impersonation bypass.
|
||||
if looks_like_ept(token):
|
||||
try:
|
||||
verified = verify_ept(token, settings.eternitas_base_url)
|
||||
except EptInvalid as exc:
|
||||
raise RepairPointer(
|
||||
status_code=401,
|
||||
code="ept_invalid",
|
||||
speak="We couldn't confirm that helper's ID, so we didn't let it in.",
|
||||
machine_cause=f"EPT verification failed: {exc}",
|
||||
remediation_tool="windy_git.reissue_agent_token",
|
||||
) from exc
|
||||
|
||||
# The token is authentic. It is NOT evidence of current standing: these
|
||||
# EPTs live ~365 days and carry `rev`/`tru` baked in at issuance, so a
|
||||
# year-old `rev: false` proves nothing. Revocation and band come from a
|
||||
# live lookup, every time.
|
||||
try:
|
||||
band, actions = await resolve_passport(settings, verified.passport)
|
||||
except PassportNotInGoodStanding as exc:
|
||||
# The signature is authentic, but the identity is no longer good.
|
||||
# Revocation takes effect here, live, on the next request — no
|
||||
# webhook required. That is the honest place for it: the token can't
|
||||
# be un-issued, but its standing is checked every time.
|
||||
raise RepairPointer(
|
||||
status_code=403,
|
||||
code="passport_revoked",
|
||||
speak="That helper's access has been turned off.",
|
||||
machine_cause=f"eternitas status for {verified.passport} is {exc.status!r}, not active",
|
||||
remediation_tool=None,
|
||||
) from exc
|
||||
if band.lower() == "untrusted":
|
||||
raise RepairPointer(
|
||||
status_code=403,
|
||||
code="agent_read_only",
|
||||
speak="That helper can look, but it isn't allowed to make changes yet.",
|
||||
machine_cause=f"passport {passport} band=untrusted is read-only",
|
||||
machine_cause=f"passport {verified.passport} band=untrusted is read-only",
|
||||
remediation_tool=None,
|
||||
)
|
||||
return Caller(
|
||||
actor_type=ActorType.agent,
|
||||
passport=passport,
|
||||
passport=verified.passport,
|
||||
band=band,
|
||||
allowed_actions=actions,
|
||||
)
|
||||
|
||||
# --- human (account-server RS256) -------------------------------------
|
||||
if settings.is_production and settings.require_verified_jwt:
|
||||
# I-8, applied to ourselves. G3.2's JWKS verifier is not written yet, and
|
||||
# an unverified JWT is an authentication bypass rather than a shortcut.
|
||||
# Refusing is the only honest answer until the verifier exists.
|
||||
raise RepairPointer(
|
||||
status_code=503,
|
||||
code="human_signin_not_ready",
|
||||
speak="Signing in isn't switched on yet. Nothing you have is affected.",
|
||||
machine_cause=(
|
||||
"JWKS verification (G3.2) is not implemented; refusing to accept "
|
||||
"an unverified human token in production"
|
||||
),
|
||||
remediation_tool=None,
|
||||
)
|
||||
# --- human (hub RS256 access token, G3.2) ------------------------------
|
||||
# Production ALWAYS verifies, whatever require_verified_jwt says: the flag
|
||||
# only exists to let local dev run against unsigned fixture tokens.
|
||||
if settings.require_verified_jwt or settings.is_production:
|
||||
try:
|
||||
human = verify_hub_token(
|
||||
token,
|
||||
settings.account_server_base_url,
|
||||
issuers=tuple(settings.hub_issuers),
|
||||
audiences=tuple(settings.hub_audiences),
|
||||
require_aud=settings.hub_require_aud,
|
||||
)
|
||||
except HubTokenInvalid as exc:
|
||||
raise RepairPointer(
|
||||
status_code=401,
|
||||
code="token_invalid",
|
||||
speak="We couldn't confirm that sign-in. Try signing in again.",
|
||||
machine_cause=f"hub token verification failed: {exc}",
|
||||
remediation_tool=None,
|
||||
) from exc
|
||||
return Caller(actor_type=ActorType.human, identity_id=human.identity_id)
|
||||
|
||||
# Local dev only (require_verified_jwt=False outside production).
|
||||
identity_id = _unverified_claim(token, "windy_identity_id") or _unverified_claim(token, "sub")
|
||||
if not identity_id:
|
||||
raise RepairPointer(
|
||||
@@ -214,10 +283,8 @@ def _unverified_claim(token: str, claim: str) -> str | None:
|
||||
on the result re-establishes trust independently: an agent's authority comes
|
||||
from a live Eternitas trust lookup, never from the token's own assertions.
|
||||
|
||||
⚠️ Full RS256/ES256 JWKS verification for the human path lands in G3.2's
|
||||
verifier and MUST be in place before `api.windygit.com` accepts a human
|
||||
token from outside. Until then the human path is reachable only from inside
|
||||
the tunnel, and `settings.require_verified_jwt` refuses it in production.
|
||||
Humans are verified by hub_jwt.verify_hub_token (G3.2); this reader backs
|
||||
only the local-dev path, which production never takes.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
|
||||
@@ -53,16 +53,33 @@ class Settings(BaseSettings):
|
||||
|
||||
# ---- account-server OIDC (human identity) -----------------------------
|
||||
account_server_base_url: str = "https://account.windyword.ai"
|
||||
# G3.2 — what a hub ACCESS token must say about itself (see hub_jwt.py).
|
||||
# Token contract v1 (lane 8c, 2026-09-23): access tokens may carry either
|
||||
# issuer. id_tokens are kept out by `type` + `windy_identity_id` + aud, not
|
||||
# by issuer.
|
||||
hub_issuers: list[str] = ["windy-identity", "https://account.windyword.ai"]
|
||||
# Contract v1: aud is an ARRAY; first-party tokens list every product, and
|
||||
# Windy Git's entry is `windy_git` (underscore). ⚠️ NEVER add "windy-git"
|
||||
# (hyphen): that is Gitea's OIDC client_id, so an id_token minted for the
|
||||
# forge would carry it and pass as a bearer here.
|
||||
hub_audiences: list[str] = ["windy_git"]
|
||||
# Flip to True once the hub emits aud on every access token.
|
||||
hub_require_aud: bool = False
|
||||
|
||||
# ---- field telemetry (admin.windyword.ai ledger) ----------------------
|
||||
# Unset token = nothing sent, nothing buffered. The token lives in the
|
||||
# root-only /etc/windygit/telemetry.env on Veron, never in the repo.
|
||||
windygit_telemetry_token: str = ""
|
||||
telemetry_ingest_url: str = "https://admin.windyword.ai/v1/events"
|
||||
|
||||
# Internal callers (the Cloud portal calling /internal/*). A first-class
|
||||
# caller class, not a bypass: unset means service calls are REFUSED.
|
||||
service_token: str = ""
|
||||
|
||||
# ⚠️ FAIL-CLOSED GATE. Full RS256/ES256 JWKS verification lands in G3.2.
|
||||
# Until it does, the human token path must not be reachable in production —
|
||||
# accepting an unverified JWT is not a shortcut, it is an authentication
|
||||
# bypass. Agents are unaffected: their authority comes from a live Eternitas
|
||||
# trust lookup, not from anything the token asserts about itself.
|
||||
# ⚠️ FAIL-CLOSED GATE. Human tokens are verified against the hub's JWKS
|
||||
# (G3.2, hub_jwt.py). False only enables the unverified local-dev path, and
|
||||
# production verifies regardless — an unverified JWT is a bypass, not a
|
||||
# shortcut.
|
||||
require_verified_jwt: bool = True
|
||||
|
||||
# ---- storage law (I-3, G4.4) ------------------------------------------
|
||||
|
||||
157
api/app/ept.py
Normal file
157
api/app/ept.py
Normal file
@@ -0,0 +1,157 @@
|
||||
"""EPT signature verification (G3.2 / G9.1) — the gate that makes an agent an agent.
|
||||
|
||||
Trust is not authentication. A trust lookup answers *"is this passport
|
||||
reputable?"*; only a signature answers *"does this caller actually hold it?"*.
|
||||
Skipping the second question was a live impersonation bypass on 2026-08-13 — a
|
||||
forged `alg:none` token naming a passport read out of the logs returned HTTP 200.
|
||||
|
||||
What this module refuses, deliberately and by construction:
|
||||
|
||||
* **`alg: none`** — the original exploit. `algorithms=["ES256"]` makes it
|
||||
unrepresentable rather than merely unlikely.
|
||||
* **Algorithm confusion.** Only ES256 is accepted. If an attacker presents an
|
||||
HS256 token, PyJWT will not try to use an EC public key as an HMAC secret,
|
||||
which is the classic way "verified" JWTs get forged.
|
||||
* **An unknown `kid`.** The key must be one Eternitas currently publishes.
|
||||
* **A wrong issuer or an expired token** — checked by the library, not by us.
|
||||
|
||||
What it deliberately does NOT decide: whether the agent is *allowed* to act.
|
||||
The EPT carries `rev` and `tru` claims baked in at issuance, and these tokens
|
||||
live for a year (observed `exp` ≈ 365 days). A year-old `rev: false` is not
|
||||
evidence of anything. **Revocation and trust must come from a live lookup**, so
|
||||
this module returns only identity and the caller re-checks standing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
import httpx
|
||||
import jwt
|
||||
from jwt import PyJWKClient
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
ISSUER = "eternitas.ai"
|
||||
ALGORITHMS = ["ES256"] # exactly one. Never widen this list.
|
||||
|
||||
_jwks_client: PyJWKClient | None = None
|
||||
_jwks_url: str | None = None
|
||||
|
||||
|
||||
class EptInvalid(Exception):
|
||||
"""The token is not a valid, currently-signed Eternitas EPT."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VerifiedEpt:
|
||||
passport: str
|
||||
operator: str | None
|
||||
bot_name: str | None
|
||||
issued_at: int | None
|
||||
expires_at: int | None
|
||||
|
||||
|
||||
def _client(base_url: str) -> PyJWKClient:
|
||||
"""One cached JWKS client. PyJWKClient caches keys and refetches on an
|
||||
unknown kid, so a key rotation heals itself without a redeploy."""
|
||||
global _jwks_client, _jwks_url
|
||||
url = f"{base_url.rstrip('/')}/.well-known/eternitas-keys"
|
||||
if _jwks_client is None or _jwks_url != url:
|
||||
_jwks_client = PyJWKClient(url, cache_keys=True, lifespan=300)
|
||||
_jwks_url = url
|
||||
return _jwks_client
|
||||
|
||||
|
||||
def verify_ept(token: str, eternitas_base_url: str) -> VerifiedEpt:
|
||||
"""Verify an EPT's signature and claims. Raises EptInvalid on ANY doubt.
|
||||
|
||||
There is no partial success and no "probably fine" path: every failure mode
|
||||
below produces the same refusal, because a caller that cannot prove
|
||||
possession is indistinguishable from an attacker.
|
||||
"""
|
||||
try:
|
||||
signing_key = _client(eternitas_base_url).get_signing_key_from_jwt(token)
|
||||
except Exception as exc: # noqa: BLE001 - unknown kid, unreachable JWKS, malformed
|
||||
raise EptInvalid(f"no usable signing key: {type(exc).__name__}: {exc}") from exc
|
||||
|
||||
try:
|
||||
claims = jwt.decode(
|
||||
token,
|
||||
signing_key.key,
|
||||
algorithms=ALGORITHMS, # ES256 only — closes alg:none and alg confusion
|
||||
issuer=ISSUER,
|
||||
options={
|
||||
"require": ["sub", "iss", "exp"],
|
||||
"verify_signature": True,
|
||||
"verify_exp": True,
|
||||
"verify_iss": True,
|
||||
},
|
||||
)
|
||||
except jwt.PyJWTError as exc:
|
||||
raise EptInvalid(f"{type(exc).__name__}: {exc}") from exc
|
||||
|
||||
# The REAL claim name. Eternitas puts the passport in `sub`; the previous
|
||||
# code looked for `passport` / `sub_passport`, which no genuine EPT carries —
|
||||
# so real agents were never recognised and only forged tokens ever "worked".
|
||||
passport = claims.get("sub")
|
||||
if not isinstance(passport, str) or not passport.strip():
|
||||
raise EptInvalid("EPT carried no passport in `sub`")
|
||||
|
||||
return VerifiedEpt(
|
||||
passport=passport,
|
||||
operator=claims.get("ope"),
|
||||
bot_name=claims.get("bot"),
|
||||
issued_at=claims.get("iat"),
|
||||
expires_at=claims.get("exp"),
|
||||
)
|
||||
|
||||
|
||||
def looks_like_ept(token: str) -> bool:
|
||||
"""Cheap, unauthenticated triage: is this token even *claiming* to be an EPT?
|
||||
|
||||
Used ONLY to route a token to the right verifier. It decides nothing about
|
||||
trust — an attacker controls every byte it reads.
|
||||
|
||||
Decodes the header segment directly rather than via
|
||||
`jwt.get_unverified_header`, which validates the whole token structure and
|
||||
therefore rejects anything with a malformed SIGNATURE. That made routing
|
||||
depend on signature well-formedness: an EPT-shaped token with a bad
|
||||
signature fell through to the human path, where it was refused for the wrong
|
||||
reason ("signing in isn't switched on") and — with `require_verified_jwt`
|
||||
off — could have been read as a human identity via its `sub` claim.
|
||||
|
||||
Routing must depend only on what the token claims to be. Whether it is
|
||||
authentic is `verify_ept`'s job, and it says no.
|
||||
"""
|
||||
try:
|
||||
head_b64 = token.split(".", 1)[0]
|
||||
head_b64 += "=" * (-len(head_b64) % 4)
|
||||
header = json.loads(base64.urlsafe_b64decode(head_b64))
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
if not isinstance(header, dict):
|
||||
return False
|
||||
return header.get("typ") == "EPT" or header.get("alg") in ("ES256", "none")
|
||||
|
||||
|
||||
async def eternitas_reachable(base_url: str) -> bool:
|
||||
"""Whether the key set can be fetched at all. Used by /health/full so an
|
||||
unreachable JWKS is reported rather than discovered during an outage."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, connect=3.0)) as c:
|
||||
r = await c.get(
|
||||
f"{base_url.rstrip('/')}/.well-known/eternitas-keys",
|
||||
headers={"User-Agent": "windy-git/1.0"},
|
||||
)
|
||||
return r.status_code == 200 and "keys" in r.json()
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
|
||||
|
||||
def seconds_until_expiry(ept: VerifiedEpt) -> int | None:
|
||||
return None if ept.expires_at is None else int(ept.expires_at - time.time())
|
||||
133
api/app/hub_jwt.py
Normal file
133
api/app/hub_jwt.py
Normal file
@@ -0,0 +1,133 @@
|
||||
"""Human token verification (G3.2) — hub access tokens from account.windyword.ai.
|
||||
|
||||
Until this existed the human path refused every token in production (503
|
||||
`human_signin_not_ready`), because reading an unverified JWT's claims is an
|
||||
authentication bypass, not a shortcut. This module is what lets it say yes.
|
||||
|
||||
The token it accepts is the hub's ACCESS token, as observed live 2026-09-23:
|
||||
|
||||
header {alg: RS256, typ: JWT, kid: <published at /.well-known/jwks.json>}
|
||||
claims iss = "windy-identity" (contract v1 also allows the discovery URL)
|
||||
type = "human", exp - iat = 900 s
|
||||
sub = per-row user id ← NOT the cross-product identity
|
||||
windy_identity_id = the Windy Account UUID (what Gitea's OIDC links on)
|
||||
no `aud` yet
|
||||
|
||||
What it refuses, by construction:
|
||||
|
||||
* **Anything but RS256.** One algorithm, never a list. Closes `alg: none` and
|
||||
HS256-with-the-public-key confusion.
|
||||
* **An unknown `kid`**, a wrong issuer, an expired token — library-checked.
|
||||
* **An id_token used as a bearer.** id_tokens prove a login happened to a
|
||||
relying party (for the forge: aud `windy-git`), not that this caller may act
|
||||
here. They carry no `type` and no `windy_identity_id`, and their aud is a
|
||||
client id, not the product name `windy_git` — any one of the three refuses.
|
||||
* **A non-human `type`.** An agent's authority comes from its EPT and a live
|
||||
Eternitas lookup, never from a hub token dressed as a person.
|
||||
* **A token with no `windy_identity_id`.** `sub` is a different namespace (the
|
||||
per-row user id); falling back to it would silently mint identities that
|
||||
match nothing Gitea knows.
|
||||
|
||||
`aud` (token contract v1, lane 8c): an array; first-party tokens list every
|
||||
product and Windy Git's is `windy_git`. Optional until the hub emits it; when
|
||||
present it MUST include `windy_git`.
|
||||
`hub_require_aud=True` makes it mandatory — flip it once the hub emits it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import jwt
|
||||
from jwt import PyJWKClient
|
||||
|
||||
ALGORITHMS = ["RS256"] # exactly one. Never widen this list.
|
||||
|
||||
_jwks_client: PyJWKClient | None = None
|
||||
_jwks_url: str | None = None
|
||||
|
||||
|
||||
class HubTokenInvalid(Exception):
|
||||
"""Not a valid, currently-signed hub access token for a human."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VerifiedHuman:
|
||||
identity_id: str
|
||||
email: str | None
|
||||
expires_at: int | None
|
||||
|
||||
|
||||
def _client(base_url: str) -> PyJWKClient:
|
||||
"""Cached JWKS client; refetches on an unknown kid so rotation self-heals."""
|
||||
global _jwks_client, _jwks_url
|
||||
url = f"{base_url.rstrip('/')}/.well-known/jwks.json"
|
||||
if _jwks_client is None or _jwks_url != url:
|
||||
_jwks_client = PyJWKClient(url, cache_keys=True, lifespan=300)
|
||||
_jwks_url = url
|
||||
return _jwks_client
|
||||
|
||||
|
||||
def verify_hub_token(
|
||||
token: str,
|
||||
base_url: str,
|
||||
*,
|
||||
issuers: tuple[str, ...],
|
||||
audiences: tuple[str, ...],
|
||||
require_aud: bool,
|
||||
signing_key=None,
|
||||
) -> VerifiedHuman:
|
||||
"""Verify a hub access token. Raises HubTokenInvalid on ANY doubt.
|
||||
|
||||
`signing_key` exists for tests only (a locally generated key, no network).
|
||||
"""
|
||||
try:
|
||||
key = (
|
||||
signing_key
|
||||
if signing_key is not None
|
||||
else _client(base_url).get_signing_key_from_jwt(token).key
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - unknown kid, unreachable JWKS, malformed
|
||||
raise HubTokenInvalid(f"no usable signing key: {type(exc).__name__}: {exc}") from exc
|
||||
|
||||
try:
|
||||
claims = jwt.decode(
|
||||
token,
|
||||
key,
|
||||
algorithms=ALGORITHMS,
|
||||
issuer=list(issuers),
|
||||
options={
|
||||
"require": ["iss", "exp", "iat"],
|
||||
"verify_signature": True,
|
||||
"verify_exp": True,
|
||||
"verify_iss": True,
|
||||
# Checked by hand below: PyJWT rejects any token CARRYING aud
|
||||
# when no audience is passed, which would break the moment the
|
||||
# hub starts emitting it — the exact trap the SSO matrix names.
|
||||
"verify_aud": False,
|
||||
},
|
||||
)
|
||||
except jwt.PyJWTError as exc:
|
||||
raise HubTokenInvalid(f"{type(exc).__name__}: {exc}") from exc
|
||||
|
||||
aud = claims.get("aud")
|
||||
if aud is None:
|
||||
if require_aud:
|
||||
raise HubTokenInvalid("token carries no aud and hub_require_aud is on")
|
||||
else:
|
||||
presented = {aud} if isinstance(aud, str) else set(aud) if isinstance(aud, list) else set()
|
||||
if not presented & set(audiences):
|
||||
raise HubTokenInvalid(f"aud {sorted(presented)} does not name Windy Git")
|
||||
|
||||
# REQUIRED, not defaulted: id_tokens carry no `type`, and this is one of the
|
||||
# two claims (with windy_identity_id) that keep them from acting as bearers.
|
||||
if claims.get("type") != "human":
|
||||
raise HubTokenInvalid(f"token type {claims.get('type')!r} is not a human access token")
|
||||
|
||||
identity = claims.get("windy_identity_id") or claims.get("windyIdentityId")
|
||||
if not isinstance(identity, str) or not identity.strip():
|
||||
raise HubTokenInvalid("token carries no windy_identity_id")
|
||||
|
||||
return VerifiedHuman(
|
||||
identity_id=identity, email=claims.get("email"), expires_at=claims.get("exp")
|
||||
)
|
||||
@@ -6,11 +6,13 @@ component and is reached only over its REST API (D-2 / I-1).
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import socket
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
@@ -25,6 +27,7 @@ from api.app.providers.registry import (
|
||||
R2Provider,
|
||||
)
|
||||
from api.app.routes import health, repos, webhooks
|
||||
from api.app.telemetry import SYNTHETIC, Telemetry, caller_class, is_synthetic
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
@@ -44,9 +47,7 @@ def _refuse_kit_zero(settings) -> None:
|
||||
if not settings.is_production:
|
||||
return
|
||||
try:
|
||||
local_ips = {
|
||||
info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)
|
||||
}
|
||||
local_ips = {info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)}
|
||||
except socket.gaierror:
|
||||
return
|
||||
if settings.kit0_host in local_ips:
|
||||
@@ -98,8 +99,23 @@ async def lifespan(app: FastAPI):
|
||||
# systemd Restart=always, plus the runbook's `systemctl status`.
|
||||
]
|
||||
|
||||
telemetry = Telemetry(
|
||||
settings.telemetry_ingest_url,
|
||||
settings.windygit_telemetry_token,
|
||||
environment=settings.environment,
|
||||
commit_sha=info.commit_sha,
|
||||
version=info.version,
|
||||
)
|
||||
app.state.telemetry = telemetry
|
||||
telemetry.boot()
|
||||
await telemetry.flush()
|
||||
task = asyncio.create_task(telemetry.run()) if telemetry.enabled else None
|
||||
|
||||
yield
|
||||
|
||||
if task is not None:
|
||||
task.cancel()
|
||||
await telemetry.flush()
|
||||
if engine is not None:
|
||||
await engine.dispose()
|
||||
|
||||
@@ -119,8 +135,44 @@ app.include_router(repos.router)
|
||||
app.include_router(webhooks.router)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def _count_requests(request: Request, call_next):
|
||||
"""Heartbeat counts (requests, 4xx/5xx, refusals, p95). Never raises."""
|
||||
start = time.perf_counter()
|
||||
marker = SYNTHETIC.set(is_synthetic(request.headers))
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
SYNTHETIC.reset(marker)
|
||||
tel = getattr(request.app.state, "telemetry", None)
|
||||
if tel is not None:
|
||||
tel.record_request(
|
||||
response.status_code,
|
||||
(time.perf_counter() - start) * 1000,
|
||||
refused=getattr(request.state, "refused", False),
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(RepairPointer)
|
||||
async def _repair_pointer_handler(_, exc: RepairPointer) -> JSONResponse:
|
||||
async def _repair_pointer_handler(request: Request, exc: RepairPointer) -> JSONResponse:
|
||||
tel = getattr(request.app.state, "telemetry", None)
|
||||
detail = exc.detail if isinstance(exc.detail, dict) else {}
|
||||
code = detail.get("code")
|
||||
if tel is not None and code in tel.auth_codes:
|
||||
# A refusal is a failure row (field-visibility rule 1). The caller is
|
||||
# unauthenticated by definition, so: system actor, no actor_id, and
|
||||
# the route TEMPLATE, never the concrete path.
|
||||
request.state.refused = True
|
||||
route = request.scope.get("route")
|
||||
tel.auth_failed(
|
||||
code=code,
|
||||
http_status=exc.status_code,
|
||||
caller=caller_class(request.headers),
|
||||
route=getattr(route, "path", None),
|
||||
upstream_status=getattr(exc, "upstream_status", None),
|
||||
synthetic=is_synthetic(request.headers),
|
||||
)
|
||||
return JSONResponse(status_code=exc.status_code, content=exc.detail)
|
||||
|
||||
|
||||
|
||||
@@ -105,7 +105,7 @@ class DatabaseProvider(Provider):
|
||||
|
||||
|
||||
# TunnelProvider was removed deliberately. See the note in main.py: cloudflared
|
||||
# binds 127.0.0.1:2000 on the HOST, and this process runs in a container whose
|
||||
# binds 127.0.0.1:2001 on the HOST, and this process runs in a container whose
|
||||
# only route to the host is the bridge gateway (172.17.0.1), where nothing is
|
||||
# listening. Binding the metrics endpoint wider would fix the probe and make a
|
||||
# metrics bind failure able to take down ingress -- a worse trade than losing
|
||||
|
||||
@@ -26,6 +26,7 @@ from pydantic import BaseModel, Field, field_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from api.app import throttle
|
||||
from api.app.auth import ActorType, Caller, get_caller
|
||||
from api.app.errors import RepairPointer
|
||||
from api.app.models.core import (
|
||||
@@ -208,6 +209,11 @@ async def create_repo(
|
||||
remediation_tool=None,
|
||||
)
|
||||
|
||||
# Throttle before any side effect. Checking after would let a rate-limited
|
||||
# agent still create the Gitea repo and only then be told no.
|
||||
async with _sessionmaker(request)() as session:
|
||||
await throttle.enforce(session, settings, caller, "repo.create")
|
||||
|
||||
gitea = GiteaClient(settings)
|
||||
owner = _owner_login(caller)
|
||||
await gitea.ensure_user(owner, f"{owner}@windygit.com")
|
||||
@@ -234,6 +240,8 @@ async def create_repo(
|
||||
),
|
||||
)
|
||||
session.add(repo)
|
||||
await session.flush()
|
||||
await throttle.record(session, caller, "repo.create", repo_id=repo.id)
|
||||
await session.commit()
|
||||
await session.refresh(repo)
|
||||
|
||||
@@ -337,6 +345,7 @@ async def create_grant(
|
||||
"""
|
||||
settings = request.app.state.settings
|
||||
async with _sessionmaker(request)() as session:
|
||||
await throttle.enforce(session, settings, caller, "grant.create")
|
||||
repo = await _load_repo(session, repo_id, caller)
|
||||
is_owner = (caller.identity_id and repo.identity_id == caller.identity_id) or (
|
||||
caller.passport and repo.passport == caller.passport
|
||||
@@ -364,6 +373,8 @@ async def create_grant(
|
||||
expires_at=expires,
|
||||
)
|
||||
session.add(grant)
|
||||
await session.flush()
|
||||
await throttle.record(session, caller, "grant.create", repo_id=repo.id)
|
||||
await session.commit()
|
||||
await session.refresh(grant)
|
||||
grant_id, role = grant.id, grant.role
|
||||
|
||||
@@ -20,6 +20,7 @@ import httpx
|
||||
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer, provider_unconfigured
|
||||
from api.app.telemetry import synthetic_headers
|
||||
|
||||
_TIMEOUT = httpx.Timeout(20.0, connect=5.0)
|
||||
|
||||
@@ -36,6 +37,7 @@ class GiteaClient:
|
||||
return {
|
||||
"Authorization": f"token {self._s.gitea_admin_token}",
|
||||
"Content-Type": "application/json",
|
||||
**synthetic_headers(), # end-to-end synthetic convention (Telemetry UPDATE 4)
|
||||
}
|
||||
|
||||
async def _request(self, method: str, path: str, **kw: Any) -> httpx.Response:
|
||||
|
||||
255
api/app/telemetry.py
Normal file
255
api/app/telemetry.py
Normal file
@@ -0,0 +1,255 @@
|
||||
"""Field telemetry to the admin ledger (admin.windyword.ai) — step 2, 2026-09-23.
|
||||
|
||||
Shapes are DECLARED with Telemetry Boss (the ledger owner); the server
|
||||
quarantines any row that doesn't match, so never add a key or a code here
|
||||
without re-declaring it first:
|
||||
|
||||
service.boot once per process start {commit_sha, version, environment}
|
||||
service.health hourly, in-process interval_s, uptime_s, requests,
|
||||
errors_5xx, errors_4xx,
|
||||
refusals_4xx, p95_ms
|
||||
forge.auth.failed every refused request {code, http_status, caller,
|
||||
route?, upstream_status?}
|
||||
|
||||
Refusals come first: a refused caller is the most expensive silent failure
|
||||
("the button did nothing"). An UNAUTHENTICATED caller has no trustworthy id, so
|
||||
per the all-lanes actor rule the row is actor_type "system", no actor_id, and
|
||||
the caller class goes in metadata.caller.
|
||||
|
||||
Privacy: codes, statuses, route TEMPLATES, counts, durations. Never a passport
|
||||
number, an email, a token fragment or a concrete path with names in it.
|
||||
|
||||
No token → nothing is sent and nothing is buffered. A failed flush keeps the
|
||||
rows (bounded) and retries on the next tick; it never raises into a request.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextvars
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import urllib.request
|
||||
from datetime import UTC, datetime
|
||||
|
||||
log = logging.getLogger("windy-git.telemetry")
|
||||
|
||||
PLATFORM, SERVICE = "windy-git", "api"
|
||||
FLUSH_EVERY_S = 60
|
||||
HEALTH_EVERY_S = 3600
|
||||
MAX_BUFFER = 5000
|
||||
MAX_LATENCY_SAMPLES = 20000
|
||||
|
||||
# The declared forge.auth.failed code enum (Telemetry Boss, 2026-09-23). A code
|
||||
# outside this set is NOT a refusal row — it counts in errors_* instead.
|
||||
AUTH_CODES = frozenset(
|
||||
{
|
||||
"not_signed_in",
|
||||
"token_invalid",
|
||||
"token_unrecognised",
|
||||
"ept_invalid",
|
||||
"passport_revoked",
|
||||
"passport_unresolvable",
|
||||
"agent_read_only",
|
||||
"agent_rate_limited",
|
||||
"quota_exceeded",
|
||||
"trust_unavailable",
|
||||
"throttle_unavailable",
|
||||
"service_token_invalid",
|
||||
"service_auth_unconfigured",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _iso(epoch: float) -> str:
|
||||
return datetime.fromtimestamp(epoch, UTC).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
# Ecosystem convention (Telemetry UPDATE 4): synthetic traffic travels END TO
|
||||
# END. Originators (canaries, probes, journeys) send `X-Windy-Synthetic: 1`;
|
||||
# every service marks all of that request's rows synthetic:true AND forwards the
|
||||
# header on every downstream call. Absent = real. Never strip it, never set it
|
||||
# on real traffic. The label separates rows — it never suppresses them.
|
||||
SYNTHETIC: contextvars.ContextVar[bool] = contextvars.ContextVar("windy_synthetic", default=False)
|
||||
|
||||
|
||||
def is_synthetic(headers) -> bool:
|
||||
return bool((headers.get("x-windy-synthetic") or "").strip())
|
||||
|
||||
|
||||
def synthetic_headers() -> dict:
|
||||
"""Merge into every downstream request made while serving this one."""
|
||||
return {"X-Windy-Synthetic": "1"} if SYNTHETIC.get() else {}
|
||||
|
||||
|
||||
def caller_class(headers) -> str:
|
||||
"""Declared values: anonymous_human | anonymous_agent | unknown."""
|
||||
from api.app.ept import looks_like_ept
|
||||
|
||||
if headers.get("x-service-token"):
|
||||
return "unknown"
|
||||
auth = headers.get("authorization") or ""
|
||||
if not auth.lower().startswith("bearer "):
|
||||
return "unknown"
|
||||
return "anonymous_agent" if looks_like_ept(auth.split(" ", 1)[1].strip()) else "anonymous_human"
|
||||
|
||||
|
||||
class Telemetry:
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
token: str,
|
||||
*,
|
||||
environment: str = "",
|
||||
commit_sha: str | None = None,
|
||||
version: str = "",
|
||||
) -> None:
|
||||
self.url, self.token = url, token
|
||||
self.environment, self.commit_sha, self.version = environment, commit_sha, version
|
||||
self.started = time.time()
|
||||
self.buffer: list[dict] = []
|
||||
self.auth_codes = AUTH_CODES
|
||||
self._reset_window()
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return bool(self.token)
|
||||
|
||||
def _reset_window(self) -> None:
|
||||
self.window_start = time.time()
|
||||
self.requests = self.errors_5xx = self.errors_4xx = self.refusals_4xx = 0
|
||||
self.latencies_ms: list[float] = []
|
||||
# UPDATE 7: rows the ledger quarantined (it still answers 202) and rows
|
||||
# this process lost (buffer overflow). Non-zero = a bug in this emitter.
|
||||
self.quarantined = self.dropped = 0
|
||||
|
||||
# ---- recording (never raises into a request) --------------------------
|
||||
def record_request(self, status: int, duration_ms: float, *, refused: bool = False) -> None:
|
||||
self.requests += 1
|
||||
if status >= 500:
|
||||
self.errors_5xx += 1
|
||||
elif refused:
|
||||
self.refusals_4xx += 1
|
||||
elif status >= 400:
|
||||
self.errors_4xx += 1
|
||||
if len(self.latencies_ms) < MAX_LATENCY_SAMPLES:
|
||||
self.latencies_ms.append(duration_ms)
|
||||
|
||||
def _event(self, event_type: str, metadata: dict, *, ts: float | None = None) -> None:
|
||||
if not self.enabled:
|
||||
return
|
||||
self.buffer.append(
|
||||
{
|
||||
"ts": _iso(ts or time.time()),
|
||||
"platform": PLATFORM,
|
||||
"service": SERVICE,
|
||||
"event_type": event_type,
|
||||
"actor_type": "system",
|
||||
"metadata": metadata,
|
||||
}
|
||||
)
|
||||
if len(self.buffer) > MAX_BUFFER:
|
||||
self.dropped += len(self.buffer) - MAX_BUFFER
|
||||
del self.buffer[: len(self.buffer) - MAX_BUFFER]
|
||||
|
||||
def boot(self) -> None:
|
||||
meta = {"version": self.version, "environment": self.environment}
|
||||
if self.commit_sha: # unknown is absent, never invented (I-12)
|
||||
meta["commit_sha"] = self.commit_sha
|
||||
self._event("service.boot", meta, ts=self.started)
|
||||
|
||||
def auth_failed(
|
||||
self,
|
||||
*,
|
||||
code: str,
|
||||
http_status: int,
|
||||
caller: str,
|
||||
route: str | None = None,
|
||||
upstream_status: int | None = None,
|
||||
synthetic: bool = False,
|
||||
) -> None:
|
||||
if code not in AUTH_CODES:
|
||||
return
|
||||
meta: dict = {
|
||||
"code": code,
|
||||
"http_status": int(http_status),
|
||||
"caller": caller,
|
||||
"synthetic": bool(synthetic),
|
||||
}
|
||||
if route:
|
||||
meta["route"] = route
|
||||
if upstream_status is not None:
|
||||
meta["upstream_status"] = int(upstream_status)
|
||||
self._event("forge.auth.failed", meta)
|
||||
|
||||
def health_row(self) -> dict:
|
||||
now = time.time()
|
||||
meta = {
|
||||
"interval_s": int(now - self.window_start),
|
||||
"uptime_s": int(now - self.started),
|
||||
"requests": self.requests,
|
||||
"errors_5xx": self.errors_5xx,
|
||||
"errors_4xx": self.errors_4xx,
|
||||
"refusals_4xx": self.refusals_4xx,
|
||||
"telemetry_quarantined": self.quarantined,
|
||||
"telemetry_dropped": self.dropped,
|
||||
}
|
||||
if self.latencies_ms: # no traffic = no p95, not a fake 0
|
||||
s = sorted(self.latencies_ms)
|
||||
meta["p95_ms"] = int(s[min(len(s) - 1, int(0.95 * (len(s) - 1) + 0.5))])
|
||||
return meta
|
||||
|
||||
def health(self) -> None:
|
||||
self._event("service.health", self.health_row())
|
||||
self._reset_window()
|
||||
|
||||
# ---- sending ------------------------------------------------------------
|
||||
def _post(self, batch: list[dict]) -> tuple[int, dict]:
|
||||
req = urllib.request.Request(
|
||||
self.url,
|
||||
data=json.dumps({"events": batch}).encode(),
|
||||
method="POST",
|
||||
headers={
|
||||
"Authorization": f"Bearer {self.token}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "windy-git-api-telemetry/1",
|
||||
},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
try:
|
||||
body = json.loads(r.read() or b"{}")
|
||||
except ValueError:
|
||||
body = {}
|
||||
return r.status, body if isinstance(body, dict) else {}
|
||||
|
||||
async def flush(self) -> None:
|
||||
if not self.enabled or not self.buffer:
|
||||
return
|
||||
batch = self.buffer[:500]
|
||||
try:
|
||||
status, body = await asyncio.to_thread(self._post, batch)
|
||||
except Exception as exc: # noqa: BLE001 - telemetry must never take the API down
|
||||
log.warning("telemetry flush failed (%d rows kept): %s", len(self.buffer), exc)
|
||||
return
|
||||
if 200 <= status < 300:
|
||||
del self.buffer[: len(batch)]
|
||||
self.note_quarantine(body)
|
||||
|
||||
def note_quarantine(self, body: dict) -> None:
|
||||
# 202 does NOT mean every row landed: refused rows are dead-lettered.
|
||||
q = body.get("quarantined")
|
||||
if isinstance(q, int) and q > 0:
|
||||
self.quarantined += q
|
||||
log.warning("telemetry: %d row(s) QUARANTINED by the ledger: %s", q,
|
||||
"; ".join(map(str, body.get("rejections") or [])) or "no reason given")
|
||||
|
||||
async def run(self) -> None:
|
||||
"""The one in-process timer: flush every minute, heartbeat every hour."""
|
||||
last_health = time.monotonic()
|
||||
while True:
|
||||
await asyncio.sleep(FLUSH_EVERY_S)
|
||||
if time.monotonic() - last_health >= HEALTH_EVERY_S:
|
||||
self.health()
|
||||
last_health = time.monotonic()
|
||||
await self.flush()
|
||||
187
api/app/throttle.py
Normal file
187
api/app/throttle.py
Normal file
@@ -0,0 +1,187 @@
|
||||
"""EI-band velocity limits (G3.4) — throttle by trust, never by omission.
|
||||
|
||||
`BAND_MULTIPLIER` and the `rate_*_per_day` settings existed since G0 and were
|
||||
**read by nothing**: a documented invariant with no implementation, which is the
|
||||
exact failure pattern the ecosystem audits kept finding. This module is the
|
||||
missing consumer.
|
||||
|
||||
The doctrine (§0.6) is *capability-completeness*: an agent is never denied a
|
||||
capability it should have, it is **rate-limited by how much it has proven**.
|
||||
Platinum gets 10x, gold 4x, standard 1x, watch 0.5x, and untrusted is read-only.
|
||||
|
||||
Counting is done against `agent_actions`, which is already the append-only record
|
||||
of every agent write. That is deliberate: a limiter with its own private counter
|
||||
disagrees with the audit log the moment either is restarted, and then nobody can
|
||||
say what actually happened. One source of truth, queried.
|
||||
|
||||
**Fail-closed.** If the count cannot be taken, the action is refused. A limiter
|
||||
that fails open is decoration — it protects you right up until the moment
|
||||
something is wrong, which is the only moment it matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from api.app.auth import BAND_MULTIPLIER, ActorType, Caller
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
from api.app.models.core import AgentAction
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
WINDOW = timedelta(days=1)
|
||||
|
||||
# Actions this module ACTUALLY enforces: they pass through our API, so we can
|
||||
# both count and refuse them.
|
||||
ACTION_BASE: dict[str, str] = {
|
||||
"repo.create": "rate_repo_creates_per_day",
|
||||
"grant.create": "rate_grants_per_day",
|
||||
}
|
||||
|
||||
# ⚠️ DECLARED BUT NOT ENFORCEABLE HERE — and named, rather than quietly listed
|
||||
# alongside the real ones.
|
||||
#
|
||||
# `git push` goes straight to Gitea over HTTPS and never touches this API, so
|
||||
# nothing records a `push` action and a count of them would be zero forever.
|
||||
# Listing these in ACTION_BASE (as this module first did) would make `enforce`
|
||||
# look up a limit, count nothing, and allow everything — a silent no-op wearing
|
||||
# the costume of a control. That is the same dead-code pattern this module was
|
||||
# written to remove, and it is worse here because the config name implies the
|
||||
# protection exists.
|
||||
#
|
||||
# Enforcing push velocity requires a Gitea-side hook (pre-receive or the push
|
||||
# webhook) that reports into `agent_actions`. Until that exists these settings
|
||||
# are inert, and saying so is the honest option.
|
||||
NOT_ENFORCED_HERE: dict[str, str] = {
|
||||
"push": "rate_pushes_per_day",
|
||||
"push.force": "rate_force_pushes_per_day",
|
||||
}
|
||||
|
||||
|
||||
def limit_for(settings: Settings, action: str, band: str | None) -> int:
|
||||
"""Effective per-day allowance. Unknown bands get the standard rate.
|
||||
|
||||
Unknown-band handling is a real decision, not a default. Eternitas began
|
||||
emitting `unproven` on 2026-07-30 without it appearing in the documented
|
||||
enum. Treating an unrecognised band as untrusted would lock out every freshly
|
||||
hatched agent the day Eternitas adds a name; treating it as platinum would be
|
||||
a hole. Standard-with-no-bonus is the honest middle.
|
||||
"""
|
||||
base = getattr(settings, ACTION_BASE[action])
|
||||
mult = BAND_MULTIPLIER.get((band or "").lower(), 1.0)
|
||||
return int(base * mult)
|
||||
|
||||
|
||||
async def enforce(
|
||||
session: AsyncSession,
|
||||
settings: Settings,
|
||||
caller: Caller,
|
||||
action: str,
|
||||
) -> None:
|
||||
"""Raise 429 if this agent has spent its allowance. No-op for humans.
|
||||
|
||||
Humans are governed by account tier and their own session; this is the
|
||||
agent-velocity control specifically (I-6: parity in capability, asymmetry in
|
||||
throttle).
|
||||
"""
|
||||
if caller.actor_type != ActorType.agent or not caller.passport:
|
||||
return
|
||||
if action not in ACTION_BASE: # unknown action = unlimited would be a hole
|
||||
raise RepairPointer(
|
||||
status_code=500,
|
||||
code="throttle_unknown_action",
|
||||
speak="Something went wrong on our side. Nothing was changed.",
|
||||
machine_cause=f"no rate base configured for action {action!r}",
|
||||
remediation_tool=None,
|
||||
)
|
||||
|
||||
band = (caller.band or "").lower()
|
||||
|
||||
# Untrusted is read-only. This is a capability decision, not a rate: it gets
|
||||
# a 403 with a different explanation, because "slow down" would be a lie.
|
||||
if BAND_MULTIPLIER.get(band, 1.0) <= 0:
|
||||
raise RepairPointer(
|
||||
status_code=403,
|
||||
code="agent_read_only",
|
||||
speak="That helper can look, but it isn't allowed to make changes yet.",
|
||||
machine_cause=f"passport {caller.passport} band={band!r} is read-only",
|
||||
remediation_tool=None,
|
||||
)
|
||||
|
||||
allowed = limit_for(settings, action, band)
|
||||
since = datetime.now(UTC) - WINDOW
|
||||
|
||||
try:
|
||||
used = (
|
||||
await session.execute(
|
||||
select(func.count())
|
||||
.select_from(AgentAction)
|
||||
.where(
|
||||
AgentAction.passport == caller.passport,
|
||||
AgentAction.action == action,
|
||||
AgentAction.result == "ok",
|
||||
AgentAction.ts >= since,
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# FAIL CLOSED. A limiter that fails open protects you until the moment
|
||||
# something is wrong, which is the only moment it matters.
|
||||
log.warning("throttle count failed for %s/%s: %s", caller.passport, action, exc)
|
||||
raise RepairPointer(
|
||||
status_code=503,
|
||||
code="throttle_unavailable",
|
||||
speak="We couldn't check that helper's limits, so we didn't make the change.",
|
||||
machine_cause=f"agent_actions count failed: {type(exc).__name__}",
|
||||
remediation_tool=None,
|
||||
) from exc
|
||||
|
||||
if used >= allowed:
|
||||
raise RepairPointer(
|
||||
status_code=429,
|
||||
code="agent_rate_limited",
|
||||
speak=(
|
||||
"That helper has done a lot in the last day, so we've paused it. "
|
||||
"It'll be able to continue shortly."
|
||||
),
|
||||
machine_cause=(
|
||||
f"passport {caller.passport} used {used}/{allowed} of {action} "
|
||||
f"in 24h (band={band or 'unknown'})"
|
||||
),
|
||||
remediation_tool=None,
|
||||
used=used,
|
||||
allowed=allowed,
|
||||
band=band or "unknown",
|
||||
)
|
||||
|
||||
|
||||
async def record(
|
||||
session: AsyncSession,
|
||||
caller: Caller,
|
||||
action: str,
|
||||
result: str = "ok",
|
||||
repo_id=None,
|
||||
) -> None:
|
||||
"""Append the action that was just allowed.
|
||||
|
||||
Written AFTER the work succeeds, on purpose: counting attempts would let a
|
||||
failing agent exhaust its own allowance by retrying, turning a transient
|
||||
error into a lockout.
|
||||
"""
|
||||
if caller.actor_type != ActorType.agent or not caller.passport:
|
||||
return
|
||||
session.add(
|
||||
AgentAction(
|
||||
passport=caller.passport,
|
||||
repo_id=repo_id,
|
||||
action=action,
|
||||
ei_at_action=caller.band,
|
||||
result=result,
|
||||
)
|
||||
)
|
||||
await session.flush()
|
||||
92
api/tests/test_canary_cleanup.py
Normal file
92
api/tests/test_canary_cleanup.py
Normal file
@@ -0,0 +1,92 @@
|
||||
"""The canary's login probe must end the session it opens (journey cleanup rule)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import io
|
||||
import sys
|
||||
import urllib.error
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
_spec = importlib.util.spec_from_file_location("canary", ROOT / "scripts" / "canary.py")
|
||||
canary = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["canary"] = canary # dataclasses resolve their module by name
|
||||
_spec.loader.exec_module(canary)
|
||||
|
||||
|
||||
class _Resp:
|
||||
def __init__(self, status=200, body=b"{}"):
|
||||
self.status, self._body = status, body
|
||||
|
||||
def read(self):
|
||||
return self._body
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
|
||||
def _err(code):
|
||||
return urllib.error.HTTPError(canary.LOGOUT_URL, code, "x", {}, io.BytesIO(b""))
|
||||
|
||||
|
||||
def _script(monkeypatch, outcomes):
|
||||
calls = []
|
||||
|
||||
def fake(req, timeout=None):
|
||||
calls.append((req.get_method(), req.full_url, req.get_header("Authorization")))
|
||||
o = outcomes.pop(0)
|
||||
if isinstance(o, Exception):
|
||||
raise o
|
||||
return o
|
||||
|
||||
monkeypatch.setattr(canary.urllib.request, "urlopen", fake)
|
||||
return calls
|
||||
|
||||
|
||||
def test_logout_ends_the_session(monkeypatch):
|
||||
calls = _script(monkeypatch, [_Resp(200)])
|
||||
r = canary.logout("tok", sleep=lambda s: None)
|
||||
assert r.status == "ok"
|
||||
assert calls == [("POST", canary.LOGOUT_URL, "Bearer tok")]
|
||||
|
||||
|
||||
def test_5xx_and_no_response_are_retried_then_succeed(monkeypatch):
|
||||
calls = _script(monkeypatch, [_err(502), OSError("reset"), _Resp(200)])
|
||||
assert canary.logout("tok", sleep=lambda s: None).status == "ok"
|
||||
assert len(calls) == 3
|
||||
|
||||
|
||||
def test_already_over_counts_as_done(monkeypatch):
|
||||
_script(monkeypatch, [_err(401)])
|
||||
assert canary.logout("tok", sleep=lambda s: None).status == "ok"
|
||||
|
||||
|
||||
def test_other_4xx_fails_fast_and_honestly(monkeypatch):
|
||||
calls = _script(monkeypatch, [_err(400)])
|
||||
r = canary.logout("tok", sleep=lambda s: None)
|
||||
assert r.status == "down" and r.detail.startswith("CLEANUP FAILED") and len(calls) == 1
|
||||
|
||||
|
||||
def test_retries_are_bounded_and_reported(monkeypatch):
|
||||
calls = _script(monkeypatch, [_err(503)] * 8)
|
||||
r = canary.logout("tok", attempts=8, sleep=lambda s: None)
|
||||
assert r.status == "down" and "CLEANUP FAILED after 8 tries" in r.detail and len(calls) == 8
|
||||
|
||||
|
||||
def test_login_probe_logs_out_with_the_token_it_got(monkeypatch):
|
||||
calls = _script(monkeypatch, [_Resp(200, b'{"token": "abc"}'), _Resp(200)])
|
||||
c = canary.Check("identity.login", "https://account.windyword.ai/api/v1/auth/login", "x",
|
||||
method="POST", body={"email": "e", "password": "p"},
|
||||
after=canary._logout_after_login)
|
||||
r = canary._probe(c)
|
||||
assert r.status == "ok" and [f.status for f in r.followups] == ["ok"]
|
||||
assert calls[1] == ("POST", canary.LOGOUT_URL, "Bearer abc")
|
||||
|
||||
|
||||
def test_login_without_token_is_a_cleanup_failure_not_a_pass():
|
||||
[f] = canary._logout_after_login(b"{}")
|
||||
assert f.status == "down" and "CLEANUP FAILED" in f.detail
|
||||
164
api/tests/test_ci_hygiene.py
Normal file
164
api/tests/test_ci_hygiene.py
Normal file
@@ -0,0 +1,164 @@
|
||||
"""CI hygiene guard (house rule 6): lockfile-only installs, no host-port services."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
_spec = importlib.util.spec_from_file_location("ci_hygiene", ROOT / "scripts" / "ci_hygiene.py")
|
||||
hy = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["ci_hygiene"] = hy
|
||||
_spec.loader.exec_module(hy)
|
||||
|
||||
WF = ".github/workflows/ci.yml"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, text", [
|
||||
(WF, " .venv/bin/pip install -e \".[dev]\""), # windy-git's own, before e64a1b5
|
||||
("Dockerfile", "RUN pip install --no-cache-dir -e ."), # windy-git image, before e64a1b5
|
||||
(WF, " - run: uv pip install -e \".[dev]\""), # WindyCloud #109's CI
|
||||
(WF, " run: pip install fastapi uvicorn"),
|
||||
(WF, " - run: uv sync --all-extras"), # windy-mind style, not locked
|
||||
(WF, " - run: npm install"), # windy-drops / windytalk
|
||||
(WF, " - run: npm install --no-save --no-audit --no-fund jsdom"), # windy-pro reality-check
|
||||
(WF, " - run: yarn install"),
|
||||
(WF, " - run: cd web && pnpm install"),
|
||||
("docker/api.Dockerfile", "RUN apt-get update && pip install requests"),
|
||||
])
|
||||
def test_floating_installs_are_flagged(path, text):
|
||||
assert [k for k, _ in hy.scan_line(path, text)] == ["floating install"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, text", [
|
||||
(WF, " python3 -m pip install -q uv==0.12.5"), # exact tool pin
|
||||
(WF, " uv sync --locked --extra dev"),
|
||||
(WF, " - run: uv sync --frozen"),
|
||||
(WF, " - run: npm ci"),
|
||||
(WF, " - run: npm install --no-save jsdom@24.1.0"),
|
||||
(WF, " - run: pip install -r requirements.lock --require-hashes"),
|
||||
(WF, " - run: pip install -r requirements.txt"),
|
||||
("Dockerfile", " && pip install --no-cache-dir --require-hashes -r /tmp/requirements.txt \\\\"),
|
||||
("Dockerfile", "RUN pip install --no-cache-dir --no-deps -e ."), # project only, deps from the lock
|
||||
(WF, " .venv/bin/pip install -q --upgrade pip"),
|
||||
(WF, " - run: yarn install --frozen-lockfile"),
|
||||
(WF, " # - run: npm install (commented out)"),
|
||||
(WF, " - run: echo 'pip is great'"),
|
||||
])
|
||||
def test_locked_or_pinned_installs_pass(path, text):
|
||||
assert hy.scan_line(path, text) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text, port", [
|
||||
(" - 5432:5432", "5432"), # windy-mind / eternitas (collided 09-23)
|
||||
(" - '15432:5432'", "15432"), # WindyCloud
|
||||
(' - "6379:6379"', "6379"),
|
||||
])
|
||||
def test_services_publishing_a_host_port_are_flagged(text, port):
|
||||
[(kind, match)] = hy.scan_line(WF, text)
|
||||
assert kind == "host port" and port in match
|
||||
|
||||
|
||||
def test_host_port_rule_is_for_workflows_only():
|
||||
assert hy.scan_line("docker-compose.yml", " - 5432:5432") == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, ok", [
|
||||
(".github/workflows/ci.yml", True), (".gitea/workflows/check.yaml", True),
|
||||
("Dockerfile", True), ("api/Dockerfile.prod", True), ("docker/web.Dockerfile", True),
|
||||
("scripts/setup.sh", False), ("README.md", False), ("node_modules/x/Dockerfile", False),
|
||||
(".github/lint/x.yml", False),
|
||||
])
|
||||
def test_scope_is_ci_workflows_and_dockerfiles(path, ok):
|
||||
assert hy.path_ok(path) is ok
|
||||
|
||||
|
||||
def test_warn_mode_never_turns_red(monkeypatch):
|
||||
monkeypatch.setattr(hy, "MODE", "warn")
|
||||
state, desc, f = hy.status_for([hy.cg.Finding(WF, 12, "floating install", "npm install (use npm ci)")], True)
|
||||
assert state == "success" and desc.startswith("⚠ WARN (not blocking): 1 CI hygiene issue in CI/Dockerfiles")
|
||||
|
||||
|
||||
def test_allow_file_is_line_scoped_exceptions_only():
|
||||
"""Every exception is line-scoped (`matches`), so an allowed file can't hide a
|
||||
NEW floating install or docker step. Today: windy-pro's if:false deploy job."""
|
||||
allow = hy.cg.load_allow(hy.ALLOW_FILE)
|
||||
assert [(e["repo"], e["paths"]) for e in allow] == [("windy-pro", [".github/workflows/ci.yml"])]
|
||||
assert all(e.get("matches") for e in allow)
|
||||
ok = " run: docker build -f account-server/Dockerfile -t windy-pro:${{ github.sha }} ."
|
||||
new = " run: docker build -t windy-pro-api ."
|
||||
assert hy.cg.allowed("windy-pro", WF, allow, ok)
|
||||
assert not hy.cg.allowed("windy-pro", WF, allow, new)
|
||||
assert not hy.cg.allowed("windy-chat", WF, allow, ok)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, text, want", [
|
||||
("Dockerfile", "COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv", "ghcr.io/astral-sh/uv:latest"), # Mail #147
|
||||
("Dockerfile", "FROM python:latest", "python:latest"),
|
||||
("Dockerfile", "FROM --platform=linux/amd64 node:latest AS web", "node:latest"),
|
||||
(WF, " image: postgres:latest", "postgres:latest"),
|
||||
(WF, " - uses: docker://ghcr.io/foo/bar:latest", "ghcr.io/foo/bar:latest"),
|
||||
])
|
||||
def test_latest_images_are_flagged(path, text, want):
|
||||
hits = hy.scan_line(path, text)
|
||||
assert ("floating image", want) in hits
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", [
|
||||
"COPY pyproject.toml uv.lock* ./", # Windy Mail #147
|
||||
"COPY package.json package-lock.json* ./",
|
||||
])
|
||||
def test_optional_lock_globs_are_flagged(text):
|
||||
assert [k for k, _ in hy.scan_line("Dockerfile", text)] == ["optional lock"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, text", [
|
||||
("Dockerfile", "COPY --from=ghcr.io/astral-sh/uv:0.12.5 /uv /usr/local/bin/uv"),
|
||||
("Dockerfile", "FROM python:3.12-slim"),
|
||||
("Dockerfile", "COPY pyproject.toml uv.lock ./"),
|
||||
("Dockerfile", "COPY src/*.py ./src/"),
|
||||
("Dockerfile", "RUN echo latest release notes"),
|
||||
])
|
||||
def test_pinned_images_and_real_locks_pass(path, text):
|
||||
assert hy.scan_line(path, text) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", [
|
||||
" - run: docker compose -f docker-compose.yml -f docker-compose.ci.yml build", # eternitas ci/build
|
||||
" run: docker build -t windy-mail .",
|
||||
" - run: docker-compose up -d",
|
||||
" run: docker buildx build --load .",
|
||||
" - uses: docker/build-push-action@v6",
|
||||
])
|
||||
def test_docker_in_ci_is_flagged_with_the_fix(text):
|
||||
hits = hy.scan_line(WF, text)
|
||||
assert [k for k, _ in hits] == ["needs docker"]
|
||||
assert "no-Docker smoke test" in hits[0][1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, text", [
|
||||
("Dockerfile", "RUN docker build ."), # not a workflow
|
||||
(WF, " run: ssh host 'docker compose up -d'"), # remote host has a daemon
|
||||
(WF, " # docker compose build"), # comment
|
||||
(WF, " run: echo docker build"),
|
||||
])
|
||||
def test_docker_not_flagged_outside_ci_steps(path, text):
|
||||
assert [k for k, _ in hy.scan_line(path, text) if k == "needs docker"] == []
|
||||
|
||||
|
||||
def test_needs_docker_skips_workflows_disabled_on_windy_git(monkeypatch):
|
||||
"""deploy.yml runs on the target host (a real daemon); Gitea has it disabled here."""
|
||||
F = hy.cg.Finding
|
||||
monkeypatch.setattr(hy, "_DISABLED", {"eternitas": {"deploy.yml"}})
|
||||
got = hy._runs_here("Eternitas", [
|
||||
F(".github/workflows/deploy.yml", 70, "needs docker", "docker compose in CI"),
|
||||
F(".github/workflows/ci.yml", 176, "needs docker", "docker compose in CI"),
|
||||
F(".github/workflows/deploy.yml", 12, "floating install", "npm install"),
|
||||
])
|
||||
assert [(f.path.rsplit("/", 1)[1], f.kind) for f in got] == [
|
||||
("ci.yml", "needs docker"), ("deploy.yml", "floating install")]
|
||||
assert hy._runs_here("eternitas", None) is None
|
||||
252
api/tests/test_compute_guard.py
Normal file
252
api/tests/test_compute_guard.py
Normal file
@@ -0,0 +1,252 @@
|
||||
"""Compute guard: Windy Mind is the only door to AI compute (warn-only today)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
_spec = importlib.util.spec_from_file_location("compute_guard", ROOT / "scripts" / "compute_guard.py")
|
||||
cg = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["compute_guard"] = cg
|
||||
_spec.loader.exec_module(cg)
|
||||
|
||||
ALLOW = cg.load_allow(ROOT / "ci" / "compute-guard-allow.yml")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path, text, kind",
|
||||
[
|
||||
# audit #1 (windy-search, closed) and #2 (windy-chat, live): the shapes they had
|
||||
("service/app/anthropic_client.py", 'URL = "https://api.anthropic.com/v1/messages"', "provider host"),
|
||||
("service/app/config.py", 'token = os.environ["ANTHROPIC_OAUTH_TOKEN"]', "provider key"),
|
||||
("services/agent-roster/lib/llm.js", "const url = 'https://api.groq.com/openai/v1/chat/completions'", "provider host"),
|
||||
("docker-compose.yml", " GROQ_API_KEY: ${GROQ_API_KEY}", "provider key"),
|
||||
# audit #3/#4 (windy-pro account-server)
|
||||
("account-server/src/routes/transcription.ts", "const r = await fetch('https://api.openai.com/v1/audio/transcriptions'", "provider host"),
|
||||
("account-server/src/config.ts", "openaiKey: process.env.OPENAI_API_KEY,", "provider key"),
|
||||
# SDKs and deps
|
||||
("app/llm.py", "from anthropic import Anthropic", "provider SDK"),
|
||||
("app/llm.py", "import openai", "provider SDK"),
|
||||
("app/llm.py", "import google.generativeai as genai", "provider SDK"),
|
||||
("src/ai.ts", 'import Anthropic from "@anthropic-ai/sdk";', "provider SDK"),
|
||||
("src/ai.js", "const Groq = require('groq-sdk')", "provider SDK"),
|
||||
("package.json", ' "openai": "^4.52.0",', "provider SDK dep"),
|
||||
("requirements.txt", "anthropic>=0.40", "provider SDK dep"),
|
||||
("pyproject.toml", ' "google-generativeai>=0.8",', "provider SDK dep"),
|
||||
],
|
||||
)
|
||||
def test_audit_shapes_are_flagged(path, text, kind):
|
||||
assert kind in [k for k, _ in cg.scan_line(path, text)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path, text",
|
||||
[
|
||||
("app/mind.py", 'MIND = "https://mind.windyword.ai/v1/chat/completions"'), # the door itself
|
||||
("app/models.py", "openai_compatible = True # Mind speaks the OpenAI wire format"),
|
||||
("app/x.py", "from app.openai_shim import x"), # a local module, not the SDK
|
||||
("package.json", ' "openai-types-lite": "1.0.0",'), # a different package
|
||||
("README.txt", "set OPENAI_API_KEY"), # scanned-by-rule, excluded by SKIP separately
|
||||
],
|
||||
)
|
||||
def test_near_misses_are_not_flagged(path, text):
|
||||
if cg.SKIP.search(path):
|
||||
return
|
||||
assert cg.scan_line(path, text) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
["tests/test_llm.py", "api/tests/x.py", "src/ai.test.ts", "web/foo.spec.js", "docs/setup.md",
|
||||
"README.md", "package-lock.json", "uv.lock", "node_modules/openai/index.js", ".github/workflows/ci.yml",
|
||||
"conftest.py", "app/llm_test.py"],
|
||||
)
|
||||
def test_tests_docs_lockfiles_vendored_ci_are_never_scanned(path):
|
||||
assert cg.SKIP.search(path)
|
||||
|
||||
|
||||
def test_allow_list_needs_a_reason_per_entry(tmp_path):
|
||||
bad = tmp_path / "a.yml"
|
||||
bad.write_text("allow:\n - repo: x\n paths: ['*']\n")
|
||||
with pytest.raises(ValueError):
|
||||
cg.load_allow(bad)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"repo, path, ok",
|
||||
[
|
||||
("windy-mind", "app/providers/anthropic.py", True),
|
||||
("windy-agent", "agent/providers.py", True),
|
||||
("windy-code", "extensions/windy-ai/src/aiProvider.ts", True),
|
||||
("windy-code", "web/server/llm.ts", False), # BYOK is the extension only
|
||||
("windy-connect", "backend/src/writers/claude_code.py", True),
|
||||
("windy-chat", "services/agent-roster/lib/llm.js", False), # audit #2: must be flagged
|
||||
("windy-pro", "account-server/src/routes/translations.ts", False),
|
||||
],
|
||||
)
|
||||
def test_allow_list_entries(repo, path, ok):
|
||||
assert cg.allowed(repo, path, ALLOW) is ok
|
||||
|
||||
|
||||
DIFF = """diff --git a/app/llm.py b/app/llm.py
|
||||
--- a/app/llm.py
|
||||
+++ b/app/llm.py
|
||||
@@ -10,0 +11,2 @@
|
||||
+import anthropic
|
||||
+client = anthropic.Anthropic()
|
||||
diff --git a/tests/test_llm.py b/tests/test_llm.py
|
||||
--- /dev/null
|
||||
+++ b/tests/test_llm.py
|
||||
@@ -0,0 +1 @@
|
||||
+import anthropic
|
||||
@@ -40 +42 @@
|
||||
-x = 1
|
||||
+x = 2
|
||||
"""
|
||||
|
||||
|
||||
def test_only_added_non_test_lines_are_findings():
|
||||
fs = cg.parse_added("windy-chat", DIFF, ALLOW)
|
||||
assert [(f.path, f.line, f.kind) for f in fs] == [("app/llm.py", 11, "provider SDK")]
|
||||
|
||||
|
||||
def _repo(tmp_path, files: dict[str, str]) -> tuple[Path, str]:
|
||||
work = tmp_path / "w"
|
||||
work.mkdir()
|
||||
run = lambda *a: subprocess.run(["git", *a], cwd=work, check=True, capture_output=True) # noqa: E731
|
||||
run("init", "-q", "-b", "main")
|
||||
for p, text in files.items():
|
||||
(work / p).parent.mkdir(parents=True, exist_ok=True)
|
||||
(work / p).write_text(text)
|
||||
run("add", "-A")
|
||||
run("-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "x")
|
||||
bare = tmp_path / "r.git"
|
||||
subprocess.run(["git", "clone", "-q", "--bare", str(work), str(bare)], check=True)
|
||||
sha = subprocess.run(["git", "--git-dir", str(bare), "rev-parse", "main"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
return bare, sha
|
||||
|
||||
|
||||
def test_tree_scan_on_a_real_git_repo(tmp_path):
|
||||
bare, sha = _repo(tmp_path, {
|
||||
"app/llm.py": "import os\nKEY = os.environ['OPENAI_API_KEY']\n",
|
||||
"app/ok.py": "MIND = 'https://mind.windyword.ai'\n",
|
||||
"tests/test_llm.py": "import anthropic\n",
|
||||
"docs/x.md": "api.anthropic.com\n",
|
||||
})
|
||||
fs = cg.scan_tree("windy-chat", bare, sha, ALLOW)
|
||||
assert [(f.path, f.line, f.kind) for f in fs] == [("app/llm.py", 2, "provider key")]
|
||||
|
||||
|
||||
def test_warn_mode_never_turns_red(monkeypatch):
|
||||
monkeypatch.setattr(cg, "MODE", "warn")
|
||||
state, desc, f = cg.status_for([cg.Finding("a.py", 3, "provider host", "api.openai.com")], whole_tree=False)
|
||||
assert state == "success" and desc.startswith("⚠ WARN (not blocking): 1 direct AI-provider use added")
|
||||
assert "a.py:3" in desc and f.path == "a.py"
|
||||
|
||||
|
||||
def test_block_mode_fails(monkeypatch):
|
||||
monkeypatch.setattr(cg, "MODE", "block")
|
||||
state, desc, _ = cg.status_for([cg.Finding("a.py", 3, "provider host", "x")], whole_tree=True)
|
||||
assert state == "failure" and desc.startswith("BLOCKED")
|
||||
|
||||
|
||||
def test_clean_is_ok():
|
||||
assert cg.status_for([], whole_tree=True)[:2] == (
|
||||
"success", "OK: no direct AI-provider use in tree (Windy Mind is the only door)")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
" # The ANTHROPIC_OAUTH_TOKEN setting was removed on 2026-09-23 ON PURPOSE", # windy-search
|
||||
"# ANTHROPIC_API_KEY=",
|
||||
" // fallback used to call https://api.groq.com directly",
|
||||
" * @see https://api.openai.com/v1/audio",
|
||||
"<!-- api.anthropic.com -->",
|
||||
],
|
||||
)
|
||||
def test_comments_are_not_calls(text):
|
||||
assert cg.scan_line("service/app/config.py", text) == []
|
||||
|
||||
|
||||
def test_code_with_a_trailing_comment_still_counts():
|
||||
assert cg.scan_line("a.js", "fetch('https://api.openai.com/v1') // TODO move to Mind")
|
||||
|
||||
|
||||
def test_windy_pro_desktop_is_byok_but_the_account_server_is_not():
|
||||
assert cg.allowed("windy-pro", "src/client/desktop/main.js", ALLOW)
|
||||
assert not cg.allowed("windy-pro", "account-server/src/routes/translations.ts", ALLOW)
|
||||
|
||||
|
||||
def test_a_commit_not_fetched_yet_is_skipped_not_an_error(tmp_path, monkeypatch):
|
||||
bare, sha = _repo(tmp_path, {"app/llm.py": "import anthropic\n"})
|
||||
monkeypatch.setattr(cg, "WORK", tmp_path)
|
||||
monkeypatch.setattr(cg, "CACHE", tmp_path / "cache.json")
|
||||
(tmp_path / "windy-chat.git").symlink_to(bare)
|
||||
assert cg.check("windy-chat", "f" * 40, "main", True) is None # pushed after the fetch
|
||||
assert [f.kind for f in cg.check("windy-chat", sha, "main", True)] == ["provider SDK"]
|
||||
|
||||
|
||||
KEYCHAIN = "src/client/web/src/pages/panels/MindKeychain.jsx"
|
||||
|
||||
|
||||
def test_scoped_allow_admits_only_the_oauth_endpoints():
|
||||
ok = [
|
||||
" window.location.href = `https://openrouter.ai/auth?callback_url=${encodeURIComponent(callback)}`",
|
||||
" const res = await fetch('https://openrouter.ai/api/v1/auth/keys', {",
|
||||
]
|
||||
for line in ok:
|
||||
assert cg.allowed("windy-pro", KEYCHAIN, ALLOW, line)
|
||||
# an inference call smuggled into the same file still flags
|
||||
assert not cg.allowed("windy-pro", KEYCHAIN, ALLOW,
|
||||
" await fetch('https://openrouter.ai/api/v1/chat/completions', {")
|
||||
# a scoped entry never allows a line it can't see
|
||||
assert not cg.allowed("windy-pro", KEYCHAIN, ALLOW)
|
||||
|
||||
|
||||
def test_scoped_allow_in_a_real_diff():
|
||||
diff = f"""--- /dev/null
|
||||
+++ b/{KEYCHAIN}
|
||||
@@ -0,0 +1,3 @@
|
||||
+ window.location.href = `https://openrouter.ai/auth?callback_url=x`
|
||||
+ const res = await fetch('https://openrouter.ai/api/v1/auth/keys', {{
|
||||
+ await fetch('https://openrouter.ai/api/v1/chat/completions', {{
|
||||
"""
|
||||
fs = cg.parse_added("windy-pro", diff, ALLOW)
|
||||
assert [(f.line, f.match) for f in fs] == [(3, "openrouter.ai")]
|
||||
|
||||
|
||||
def test_block_mode_never_blocks_grant_owned(monkeypatch):
|
||||
monkeypatch.setattr(cg, "MODE", "block")
|
||||
g = cg.Finding("src/client/desktop/x.js", 9, "provider host", "api.openai.com")
|
||||
state, desc, f = cg.status_for([], whole_tree=True, grant=[g])
|
||||
assert state == "success" and desc.startswith("⚠ WARN (Grant-owned, not blocking): 1") and f is g
|
||||
lane = cg.Finding("a.py", 3, "provider host", "x")
|
||||
assert cg.status_for([lane], whole_tree=True, grant=[g])[0] == "failure"
|
||||
|
||||
|
||||
def test_veron_ollama_warns_on_added_lines_and_never_blocks(monkeypatch):
|
||||
hits = cg.scan_line("app/llm.py", 'OLLAMA = "http://192.168.1.73:11434/api/generate"')
|
||||
assert [k for k, _ in hits] == ["veron ollama"]
|
||||
assert cg.scan_line("app/llm.py", 'port = 114345') == [] # not the port
|
||||
assert cg.scan_line("app/llm.py", "# was http://x:11434 (removed)") == [] # a comment is not a call
|
||||
f = cg.Finding("app/llm.py", 7, "veron ollama", ":11434")
|
||||
monkeypatch.setattr(cg, "MODE", "block")
|
||||
state, desc, _ = cg.status_for([f], whole_tree=False)
|
||||
assert state == "success" and desc.startswith("⚠ WARN: new Veron Ollama ref app/llm.py:7")
|
||||
assert "Windy Mind" in desc and len(desc) <= 140
|
||||
hard = cg.Finding("app/llm.py", 1, "provider host", "api.openai.com")
|
||||
assert cg.status_for([f, hard], whole_tree=False)[0] == "failure" # a real violation still blocks
|
||||
|
||||
|
||||
def test_ollama_in_added_pr_lines_only():
|
||||
diff = ("+++ b/svc/client.py\n@@ -0,0 +1,2 @@\n+import httpx\n"
|
||||
"+URL = 'http://veron:11434/api/chat'\n")
|
||||
got = cg.parse_added("some-repo", diff, [])
|
||||
assert [(f.kind, f.line) for f in got] == [("veron ollama", 2)]
|
||||
312
api/tests/test_ept_and_throttle.py
Normal file
312
api/tests/test_ept_and_throttle.py
Normal file
@@ -0,0 +1,312 @@
|
||||
"""Behavioral tests for EPT verification and EI throttling.
|
||||
|
||||
These sign real ES256 tokens with a locally-generated key and verify against a
|
||||
locally-served key set, so they exercise the ACTUAL crypto path with no network
|
||||
dependency and no reliance on Eternitas being reachable.
|
||||
|
||||
This file exists because the suite it joins was ~86 "does the source contain
|
||||
this string" assertions and zero that ran the auth decision — which is how a
|
||||
live impersonation bypass passed every test on 2026-08-13.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
import jwt
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
|
||||
from api.app import ept as ept_mod
|
||||
from api.app.auth import BAND_MULTIPLIER
|
||||
from api.app.config import Settings
|
||||
from api.app.ept import EptInvalid, verify_ept
|
||||
from api.app.throttle import ACTION_BASE, limit_for
|
||||
|
||||
ISSUER = "eternitas.ai"
|
||||
KID = "test-key-1"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def signing(monkeypatch):
|
||||
"""A real EC keypair; point the verifier's JWKS lookup at its public half."""
|
||||
key = ec.generate_private_key(ec.SECP256R1())
|
||||
|
||||
class _FakeJWK:
|
||||
def __init__(self, k):
|
||||
self.key = k
|
||||
|
||||
class _FakeClient:
|
||||
def __init__(self, *a, **kw):
|
||||
pass
|
||||
|
||||
def get_signing_key_from_jwt(self, token):
|
||||
header = jwt.get_unverified_header(token)
|
||||
if header.get("kid") != KID:
|
||||
raise Exception(f"unknown kid {header.get('kid')!r}")
|
||||
return _FakeJWK(key.public_key())
|
||||
|
||||
monkeypatch.setattr(ept_mod, "_jwks_client", None)
|
||||
monkeypatch.setattr(ept_mod, "PyJWKClient", _FakeClient)
|
||||
return key
|
||||
|
||||
|
||||
def _sign(key, claims, alg="ES256", kid=KID):
|
||||
return jwt.encode(claims, key, algorithm=alg, headers={"kid": kid, "typ": "EPT"})
|
||||
|
||||
|
||||
def _claims(**over):
|
||||
c = {
|
||||
"sub": "ET26-TEST-0001",
|
||||
"iss": ISSUER,
|
||||
"iat": int(time.time()) - 10,
|
||||
"exp": int(time.time()) + 3600,
|
||||
}
|
||||
c.update(over)
|
||||
return c
|
||||
|
||||
|
||||
# ---- the property that was broken ----------------------------------------
|
||||
def test_genuine_ept_is_accepted(signing):
|
||||
v = verify_ept(_sign(signing, _claims()), "https://api.eternitas.ai")
|
||||
assert v.passport == "ET26-TEST-0001"
|
||||
|
||||
|
||||
def test_passport_comes_from_sub_not_a_passport_claim(signing):
|
||||
"""Real EPTs put the passport in `sub`. The pre-fix code read `passport` /
|
||||
`sub_passport`, which no genuine EPT carries — so real agents were never
|
||||
recognised and only forged tokens ever worked."""
|
||||
tok = _sign(signing, _claims(sub="ET26-REAL-9999", passport="ET26-LIES-0000"))
|
||||
assert verify_ept(tok, "https://api.eternitas.ai").passport == "ET26-REAL-9999"
|
||||
|
||||
|
||||
def test_alg_none_is_refused(signing):
|
||||
"""The exact 2026-08-13 exploit."""
|
||||
import base64
|
||||
import json as _j
|
||||
|
||||
def seg(d):
|
||||
return base64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
|
||||
|
||||
forged = f"{seg({'alg':'none','typ':'EPT','kid':KID})}.{seg(_claims())}."
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(forged, "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_signature_from_a_different_key_is_refused(signing):
|
||||
attacker = ec.generate_private_key(ec.SECP256R1())
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(_sign(attacker, _claims()), "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_tampered_payload_is_refused(signing):
|
||||
tok = _sign(signing, _claims())
|
||||
h, _p, s = tok.split(".")
|
||||
import base64
|
||||
import json as _j
|
||||
|
||||
evil = base64.urlsafe_b64encode(
|
||||
_j.dumps(_claims(sub="ET26-EVIL-0000")).encode()
|
||||
).rstrip(b"=").decode()
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(f"{h}.{evil}.{s}", "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_expired_token_is_refused(signing):
|
||||
"""Genuinely signed, genuinely expired — proves exp is enforced rather than
|
||||
the token merely failing to parse."""
|
||||
tok = _sign(signing, _claims(exp=int(time.time()) - 5, iat=int(time.time()) - 100))
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(tok, "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_wrong_issuer_is_refused(signing):
|
||||
"""Correctly signed by a trusted key but claiming another issuer."""
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(_sign(signing, _claims(iss="evil.example.com")), "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_unknown_kid_is_refused(signing):
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(_sign(signing, _claims(), kid="attacker-key"), "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_missing_required_claims_are_refused(signing):
|
||||
for missing in ("sub", "exp"):
|
||||
c = _claims()
|
||||
c.pop(missing)
|
||||
with pytest.raises(EptInvalid):
|
||||
verify_ept(_sign(signing, c), "https://api.eternitas.ai")
|
||||
|
||||
|
||||
def test_only_es256_is_ever_accepted():
|
||||
"""Widening this list reopens algorithm confusion."""
|
||||
assert ept_mod.ALGORITHMS == ["ES256"]
|
||||
|
||||
|
||||
# ---- the throttle that used to be dead code ------------------------------
|
||||
def test_band_multiplier_is_actually_consumed():
|
||||
"""BAND_MULTIPLIER was defined and read by nothing before this."""
|
||||
s = Settings()
|
||||
assert limit_for(s, "repo.create", "platinum") == s.rate_repo_creates_per_day * 10
|
||||
assert limit_for(s, "repo.create", "gold") == s.rate_repo_creates_per_day * 4
|
||||
assert limit_for(s, "repo.create", "standard") == s.rate_repo_creates_per_day
|
||||
assert limit_for(s, "repo.create", "watch") == s.rate_repo_creates_per_day // 2
|
||||
|
||||
|
||||
def test_unknown_band_gets_standard_not_unlimited_and_not_zero():
|
||||
s = Settings()
|
||||
assert limit_for(s, "repo.create", "a-band-invented-tomorrow") == s.rate_repo_creates_per_day
|
||||
assert limit_for(s, "repo.create", None) == s.rate_repo_creates_per_day
|
||||
|
||||
|
||||
def test_untrusted_band_is_read_only():
|
||||
assert BAND_MULTIPLIER["untrusted"] == 0
|
||||
|
||||
|
||||
def test_every_throttled_action_has_a_configured_base():
|
||||
s = Settings()
|
||||
for action, field in ACTION_BASE.items():
|
||||
assert getattr(s, field) > 0, f"{action} has no positive base rate"
|
||||
|
||||
|
||||
# ---- revocation enforced on the live trust path (not just the webhook) ----
|
||||
def test_revoked_passport_is_refused_by_trust_decision():
|
||||
"""A revoked passport returns HTTP 200, status=revoked, band=unproven,
|
||||
allowed=[] (verified live 2026-08-13). The decision must refuse it — the
|
||||
old code returned band 'unproven' and seated the agent."""
|
||||
from api.app.auth import PassportNotInGoodStanding, decide_trust
|
||||
|
||||
revoked = {"status": "revoked", "band": "unproven", "allowed_actions": []}
|
||||
with pytest.raises(PassportNotInGoodStanding):
|
||||
decide_trust(revoked, "ET26-NJQT-QMR0")
|
||||
|
||||
|
||||
def test_active_passport_is_accepted_by_trust_decision():
|
||||
from api.app.auth import decide_trust
|
||||
|
||||
active = {"status": "active", "band": "gold", "allowed_actions": ["read", "send"]}
|
||||
band, actions = decide_trust(active, "ET26-1EF9-VJAN")
|
||||
assert band == "gold" and actions == ("read", "send")
|
||||
|
||||
|
||||
def test_unknown_or_missing_status_fails_closed():
|
||||
from api.app.auth import PassportNotInGoodStanding, decide_trust
|
||||
|
||||
for body in ({"band": "gold"}, {"status": "suspended"}, {"status": "frozen"},
|
||||
{"status": ""}, {}):
|
||||
with pytest.raises(PassportNotInGoodStanding):
|
||||
decide_trust(body, "ET26-X")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_passport_raises_on_revoked_status_wiring(monkeypatch):
|
||||
"""Proves the WIRING, not just the decision: resolve_passport must feed the
|
||||
real trust body through decide_trust and propagate the refusal. A validly
|
||||
minted EPT can outlive the passport by ~a year, so this is the path that
|
||||
stops a revoked-but-still-signed token."""
|
||||
|
||||
from api.app import auth
|
||||
from api.app.auth import PassportNotInGoodStanding, resolve_passport
|
||||
from api.app.config import Settings
|
||||
|
||||
class _Resp:
|
||||
status_code = 200
|
||||
|
||||
def json(self):
|
||||
return {"status": "revoked", "band": "unproven", "allowed_actions": []}
|
||||
|
||||
class _Client:
|
||||
def __init__(self, *a, **k):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def get(self, *a, **k):
|
||||
return _Resp()
|
||||
|
||||
monkeypatch.setattr(auth.httpx, "AsyncClient", _Client)
|
||||
settings = Settings(eternitas_platform_api_key="x", eternitas_base_url="https://api.eternitas.ai")
|
||||
|
||||
with pytest.raises(PassportNotInGoodStanding):
|
||||
await resolve_passport(settings, "ET26-NJQT-QMR0")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_passport_returns_band_on_active_wiring(monkeypatch):
|
||||
import httpx # noqa: F401
|
||||
|
||||
from api.app import auth
|
||||
from api.app.auth import resolve_passport
|
||||
from api.app.config import Settings
|
||||
|
||||
class _Resp:
|
||||
status_code = 200
|
||||
|
||||
def json(self):
|
||||
return {"status": "active", "band": "gold", "allowed_actions": ["read"]}
|
||||
|
||||
class _Client:
|
||||
def __init__(self, *a, **k): pass
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *a): return False
|
||||
async def get(self, *a, **k): return _Resp()
|
||||
|
||||
monkeypatch.setattr(auth.httpx, "AsyncClient", _Client)
|
||||
settings = Settings(eternitas_platform_api_key="x")
|
||||
band, actions = await resolve_passport(settings, "ET26-1EF9-VJAN")
|
||||
assert band == "gold" and actions == ("read",)
|
||||
|
||||
|
||||
def test_routing_does_not_depend_on_signature_wellformedness():
|
||||
"""An EPT-shaped token must route to the EPT verifier even when its
|
||||
signature is malformed. jwt.get_unverified_header() validates the whole
|
||||
token and rejects bad signature padding, which used to push such tokens to
|
||||
the human path — refused for the wrong reason, and readable as a human
|
||||
identity via `sub` whenever require_verified_jwt was off."""
|
||||
import base64
|
||||
import json as _j
|
||||
|
||||
from api.app.ept import looks_like_ept
|
||||
|
||||
def seg(d):
|
||||
return base64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
|
||||
|
||||
for hdr in ({"alg": "none", "typ": "EPT"}, {"alg": "ES256", "typ": "EPT"},
|
||||
{"alg": "ES256"}):
|
||||
tok = f"{seg(hdr)}.{seg({'sub': 'ET26-X'})}.x" # deliberately bad signature
|
||||
assert looks_like_ept(tok), f"{hdr} did not route to the EPT verifier"
|
||||
|
||||
assert not looks_like_ept("not-a-token")
|
||||
assert not looks_like_ept("")
|
||||
|
||||
|
||||
def test_only_actions_that_route_through_this_api_are_claimed_enforced():
|
||||
"""git push never touches this API, so a push limit here would count zero
|
||||
forever and allow everything — a silent no-op wearing the costume of a
|
||||
control. Push limits must stay in NOT_ENFORCED_HERE until a Gitea-side hook
|
||||
reports pushes into agent_actions."""
|
||||
from api.app.throttle import ACTION_BASE, NOT_ENFORCED_HERE
|
||||
|
||||
assert "push" not in ACTION_BASE
|
||||
assert "push.force" not in ACTION_BASE
|
||||
assert "push" in NOT_ENFORCED_HERE
|
||||
assert not set(ACTION_BASE) & set(NOT_ENFORCED_HERE)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_enforce_refuses_an_action_it_cannot_actually_limit():
|
||||
"""Asking to throttle 'push' must raise, not silently allow."""
|
||||
from api.app.auth import ActorType, Caller
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
from api.app.throttle import enforce
|
||||
|
||||
caller = Caller(actor_type=ActorType.agent, passport="ET26-X", band="gold")
|
||||
with pytest.raises(RepairPointer) as exc:
|
||||
await enforce(None, Settings(), caller, "push")
|
||||
assert exc.value.code == "throttle_unknown_action"
|
||||
77
api/tests/test_guards_report.py
Normal file
77
api/tests/test_guards_report.py
Normal file
@@ -0,0 +1,77 @@
|
||||
"""guards_report: job attribution and the Grant-owned split."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
_spec = importlib.util.spec_from_file_location("guards_report", ROOT / "scripts" / "guards_report.py")
|
||||
gr = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["guards_report"] = gr
|
||||
_spec.loader.exec_module(gr)
|
||||
|
||||
OWNED = yaml.safe_load((ROOT / "ci" / "grant-owned.yml").read_text())["grant_owned"]
|
||||
WF = """name: CI
|
||||
on:
|
||||
push:
|
||||
jobs:
|
||||
reality-check:
|
||||
runs-on: x
|
||||
steps:
|
||||
- run: npm install jsdom
|
||||
test-backend:
|
||||
runs-on: x
|
||||
steps:
|
||||
- run: pip install pytest
|
||||
"""
|
||||
|
||||
|
||||
def test_job_of_attributes_lines_to_their_job():
|
||||
assert gr.job_of(WF, 3) is None # `on:` block, not a job
|
||||
assert gr.job_of(WF, 8) == "reality-check"
|
||||
assert gr.job_of(WF, 12) == "test-backend"
|
||||
|
||||
|
||||
def test_windy_pro_desktop_jobs_and_paths_are_grant_owned():
|
||||
ci = ".github/workflows/ci.yml"
|
||||
assert gr.grant_owned("windy-pro", ci, "reality-check", OWNED)
|
||||
assert gr.grant_owned("windy-pro", ci, "build-electron", OWNED)
|
||||
assert not gr.grant_owned("windy-pro", ci, "test-backend", OWNED) # server side: 8c
|
||||
assert gr.grant_owned("windy-pro", ".github/workflows/release-mac.yml", None, OWNED)
|
||||
assert gr.grant_owned("windy-pro", "src/client/desktop/main.js", None, OWNED)
|
||||
assert not gr.grant_owned("windy-pro", "services/account-server/Dockerfile", None, OWNED)
|
||||
assert not gr.grant_owned("windy-chat", "src/client/desktop/main.js", None, OWNED)
|
||||
|
||||
|
||||
def test_render_splits_lane_and_grant_counts():
|
||||
F = gr.cg.Finding
|
||||
res = {"windy-pro": {"sha": "a" * 40, "compute": [],
|
||||
"hygiene": [(F("ci.yml", 8, "floating install", "npm install"), "reality-check", True),
|
||||
(F("ci.yml", 12, "floating install", "pip x"), "test-backend", False)]},
|
||||
"windy-git": {"sha": "b" * 40, "compute": [], "hygiene": []}}
|
||||
md = gr.render(res)
|
||||
assert "| ci-hygiene (house rule 6) | 1 | 1 | ❌ not yet |" in md
|
||||
assert "| compute-guard (Mind is the only door) | 0 | 0 | ✅ YES |" in md
|
||||
assert "| windy-git | Windy Git | bbbbbbb | 0 | 0 | 0 | clean ✅ |" in md
|
||||
assert "| windy-pro | Windy Hub | aaaaaaa |" in md # owner = session to message
|
||||
assert "(job reality-check)" in md
|
||||
|
||||
|
||||
def test_windy_pro_root_env_example_is_grant_owned_but_not_the_account_servers():
|
||||
assert gr.grant_owned("windy-pro", ".env.example", None, OWNED)
|
||||
assert not gr.grant_owned("windy-pro", "account-server/.env.example", None, OWNED)
|
||||
|
||||
|
||||
def test_split_grant_sends_desktop_code_to_grant(monkeypatch):
|
||||
F = gr.cg.Finding
|
||||
fs = [F("src/client/desktop/main.js", 3, "provider host", "x"),
|
||||
F("account-server/src/llm.ts", 5, "provider host", "y")]
|
||||
lane, grant = gr.split_grant("windy-pro", "a" * 40, fs)
|
||||
assert [f.path for f in grant] == ["src/client/desktop/main.js"]
|
||||
assert [f.path for f in lane] == ["account-server/src/llm.ts"]
|
||||
assert gr.split_grant("windy-chat", "a" * 40, fs) == (fs, [])
|
||||
169
api/tests/test_hub_jwt.py
Normal file
169
api/tests/test_hub_jwt.py
Normal file
@@ -0,0 +1,169 @@
|
||||
"""G3.2 / I-8 — human tokens are verified, never read (SSO #14, 2026-09-23).
|
||||
|
||||
Behavioral: every case signs a real RS256 token with a locally generated key
|
||||
and drives `get_caller`, so a green run means the gate refuses what it must —
|
||||
not that some string appears in auth.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import time
|
||||
|
||||
import jwt
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
|
||||
from api.app import hub_jwt
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
KEY = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
OTHER = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
IDENTITY = "5e1b9569-7f01-489d-bf14-6fe5a367fa3f"
|
||||
|
||||
|
||||
def _claims(**over):
|
||||
now = int(time.time())
|
||||
c = {
|
||||
"iss": "windy-identity",
|
||||
"type": "human",
|
||||
"sub": "row-id-not-identity",
|
||||
"windy_identity_id": IDENTITY,
|
||||
"email": "grant@example.com",
|
||||
"iat": now,
|
||||
"exp": now + 900,
|
||||
}
|
||||
c.update(over)
|
||||
return {k: v for k, v in c.items() if v is not None}
|
||||
|
||||
|
||||
def _sign(claims, key=KEY, alg="RS256"):
|
||||
return jwt.encode(claims, key, algorithm=alg, headers={"kid": "test"})
|
||||
|
||||
|
||||
class _Req:
|
||||
def __init__(self, settings):
|
||||
self.app = type("A", (), {"state": type("S", (), {"settings": settings})()})()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _local_jwks(monkeypatch):
|
||||
"""The hub's JWKS, served from KEY's public half — no network."""
|
||||
|
||||
class _Key:
|
||||
key = KEY.public_key()
|
||||
|
||||
class _Client:
|
||||
def get_signing_key_from_jwt(self, token):
|
||||
return _Key()
|
||||
|
||||
monkeypatch.setattr(hub_jwt, "_client", lambda base_url: _Client())
|
||||
|
||||
|
||||
async def _caller(token, **settings):
|
||||
from api.app.auth import get_caller
|
||||
|
||||
s = Settings(environment="production", **settings)
|
||||
return await get_caller(_Req(s), authorization=f"Bearer {token}", x_service_token=None)
|
||||
|
||||
|
||||
async def _refused(token, **settings):
|
||||
with pytest.raises(RepairPointer) as exc:
|
||||
await _caller(token, **settings)
|
||||
assert exc.value.status_code == 401 and exc.value.code == "token_invalid"
|
||||
return exc.value
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_genuine_hub_token_is_a_human_named_by_windy_identity_id():
|
||||
c = await _caller(_sign(_claims()))
|
||||
assert c.actor_type == "human"
|
||||
assert c.identity_id == IDENTITY # NOT `sub`, which is the per-row user id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_forged_signature_is_refused():
|
||||
await _refused(_sign(_claims(), key=OTHER))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expired_token_is_refused():
|
||||
await _refused(_sign(_claims(iat=int(time.time()) - 2000, exp=int(time.time()) - 60)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wrong_issuer_and_id_tokens_are_refused():
|
||||
await _refused(_sign(_claims(iss="https://evil.example")))
|
||||
# Contract v1: the discovery-URL issuer is legal for ACCESS tokens...
|
||||
c = await _caller(_sign(_claims(iss="https://account.windyword.ai")))
|
||||
assert c.identity_id == IDENTITY
|
||||
# ...but an id_token minted for the forge (aud = Gitea's client id
|
||||
# "windy-git", no type, sub = identity) must never act as a bearer here.
|
||||
id_token = _claims(
|
||||
iss="https://account.windyword.ai",
|
||||
aud="windy-git",
|
||||
type=None,
|
||||
windy_identity_id=None,
|
||||
sub=IDENTITY,
|
||||
)
|
||||
await _refused(_sign(id_token))
|
||||
await _refused(_sign(dict(id_token, windy_identity_id=IDENTITY, type="human")))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hs256_confusion_is_refused():
|
||||
# The classic forgery: HMAC the token with the PUBLIC key as the secret.
|
||||
pub = KEY.public_key().public_bytes(
|
||||
serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo
|
||||
)
|
||||
header = base64.urlsafe_b64encode(json.dumps({"alg": "HS256", "typ": "JWT"}).encode()).rstrip(
|
||||
b"="
|
||||
)
|
||||
body = base64.urlsafe_b64encode(json.dumps(_claims()).encode()).rstrip(b"=")
|
||||
sig = base64.urlsafe_b64encode(
|
||||
hmac.new(pub, header + b"." + body, hashlib.sha256).digest()
|
||||
).rstrip(b"=")
|
||||
await _refused((header + b"." + body + b"." + sig).decode())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_human_or_missing_type_is_refused():
|
||||
await _refused(_sign(_claims(type="agent")))
|
||||
await _refused(_sign(_claims(type=None)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_windy_identity_is_refused_not_read_from_sub():
|
||||
await _refused(_sign(_claims(windy_identity_id=None)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aud_is_tolerated_when_it_names_windy_git_and_refused_otherwise():
|
||||
"""PyJWT rejects ANY aud-bearing token when no audience is configured — the
|
||||
trap that would break the day the hub starts emitting aud."""
|
||||
c = await _caller(_sign(_claims(aud=["windy_chat", "windy_git", "windy_mail"])))
|
||||
assert c.identity_id == IDENTITY
|
||||
await _refused(_sign(_claims(aud=["windy_chat"])))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_require_aud_refuses_tokens_without_it():
|
||||
await _refused(_sign(_claims()), hub_require_aud=True)
|
||||
c = await _caller(_sign(_claims(aud=["windy_git"])), hub_require_aud=True)
|
||||
assert c.identity_id == IDENTITY
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_production_verifies_even_if_the_flag_is_off():
|
||||
"""require_verified_jwt=False is a local-dev convenience; production must
|
||||
never take the unverified path."""
|
||||
await _refused(_sign(_claims(), key=OTHER), require_verified_jwt=False)
|
||||
|
||||
|
||||
def test_algorithm_list_is_exactly_rs256():
|
||||
assert hub_jwt.ALGORITHMS == ["RS256"]
|
||||
@@ -10,12 +10,16 @@ happened somewhere in this ecosystem and cost real time.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64 as _b64
|
||||
import json as _json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import types as _types
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import pytest as _pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
@@ -304,12 +308,13 @@ def test_g36_trust_client_never_soft_allows():
|
||||
|
||||
def test_g36_unverified_human_jwt_is_refused_in_production():
|
||||
"""I-8 applied to ourselves: an unverified JWT is an authentication bypass,
|
||||
not a shortcut. Until G3.2's JWKS verifier exists, production refuses."""
|
||||
not a shortcut. G3.2's verifier now exists; behavioral proof that forged,
|
||||
expired, mis-issued and mis-audienced tokens are refused lives in
|
||||
test_hub_jwt.py. Here: the gate defaults closed."""
|
||||
from api.app.config import Settings
|
||||
|
||||
assert Settings().require_verified_jwt is True
|
||||
src = (ROOT / "api" / "app" / "auth.py").read_text()
|
||||
assert "human_signin_not_ready" in src
|
||||
assert "windy-git" not in Settings().hub_audiences # Gitea's client_id: id_token confusion
|
||||
|
||||
|
||||
def test_no_auth_bypass_env_var_anywhere():
|
||||
@@ -420,9 +425,28 @@ def test_i05_jobs_get_a_network_per_job_not_a_shared_bridge():
|
||||
|
||||
|
||||
def test_i05_jobs_cannot_bind_mount_from_the_daemon_host():
|
||||
cfg = (ROOT / "deploy" / "runner" / "config.yaml").read_text()
|
||||
assert "valid_volumes: []" in cfg
|
||||
assert 'docker_host: "-"' in cfg
|
||||
"""Narrowed 2026-09-23 (orchestrator-approved): a job may bind-mount EXACTLY
|
||||
one daemon path, windy-pro's non-secret build inputs, and only because dind
|
||||
itself has that path READ-ONLY. Anything more (a second path, a writable
|
||||
one, a glob) reopens the host to CI code. Still no docker socket for jobs."""
|
||||
import re
|
||||
|
||||
import yaml
|
||||
|
||||
rd = ROOT / "deploy" / "runner"
|
||||
cfg = yaml.safe_load((rd / "config.yaml").read_text())
|
||||
allowed = cfg["container"]["valid_volumes"]
|
||||
assert allowed in ([], ["/ci-inputs/windy-pro"]), f"I-5: jobs may mount nothing else: {allowed}"
|
||||
assert cfg["container"]["docker_host"] == "-"
|
||||
if allowed:
|
||||
compose = yaml.safe_load((rd / "docker-compose.yml").read_text())
|
||||
binds = [v for v in compose["services"]["dind"]["volumes"] if v.startswith("/")]
|
||||
assert binds == ["/home/user1-gpu/ci-inputs/windy-pro:/ci-inputs/windy-pro:ro"], (
|
||||
f"I-5: dind's only host bind must be the ci-inputs path, READ-ONLY: {binds}")
|
||||
for name, svc in compose["services"].items():
|
||||
if name != "dind":
|
||||
for v in svc.get("volumes") or []:
|
||||
assert not re.match(r"^/home/user1-gpu/ci-inputs", v), f"I-5: {name} mounts ci-inputs"
|
||||
|
||||
|
||||
def test_i05_no_ci_container_can_reach_the_forge_network():
|
||||
@@ -488,6 +512,11 @@ def test_g73_workflow_pins_a_python_that_satisfies_requires_python():
|
||||
|
||||
for wf in ROOT.rglob(".gitea/workflows/*.y*ml"):
|
||||
text = wf.read_text()
|
||||
# Only workflows that actually RUN Python need to pin it. A workflow
|
||||
# that shells out to psql or curl does not, and demanding a pin from
|
||||
# it is the test being wrong rather than the workflow.
|
||||
if not _re.search(r"\bpython3?\b|pytest|pip ", text):
|
||||
continue
|
||||
# Either a pinned container image or an explicit setup-python version.
|
||||
pin = _re.search(r"image:\s*python:(\d+)\.(\d+)", text) or _re.search(
|
||||
r'python-version:\s*"?(\d+)\.(\d+)"?', text
|
||||
@@ -633,3 +662,112 @@ def test_g09_backup_fails_loudly():
|
||||
src = (ROOT / "scripts" / "backup.sh").read_text()
|
||||
assert "COMPLETED WITH FAILURES" in src
|
||||
assert "refusing to report a backup that did not happen" in src
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# SECURITY (behavioral, not string-grep): the agent path must not authenticate
|
||||
# an unverified token. Regression guard for the 2026-08-13 forged-token bypass.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _forged_bearer(passport: str) -> str:
|
||||
def seg(d):
|
||||
return _b64.urlsafe_b64encode(_json.dumps(d).encode()).rstrip(b"=").decode()
|
||||
return f"{seg({'alg':'none','typ':'JWT'})}.{seg({'passport':passport})}.not-a-signature"
|
||||
|
||||
|
||||
def _fake_request(settings):
|
||||
app = _types.SimpleNamespace(state=_types.SimpleNamespace(settings=settings))
|
||||
return _types.SimpleNamespace(app=app)
|
||||
|
||||
|
||||
@_pytest.mark.asyncio
|
||||
async def test_security_forged_agent_token_is_refused_in_production():
|
||||
"""The 2026-08-13 exploit: an alg:none token naming a real passport returned
|
||||
HTTP 200 as that agent. It must now be refused whichever gate catches it —
|
||||
an EPT-shaped forgery by signature verification, a JWT-shaped one by the
|
||||
human gate. What is asserted is REFUSAL, not a particular error code."""
|
||||
from api.app.auth import get_caller
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
settings = Settings(environment="production", require_verified_jwt=True,
|
||||
eternitas_platform_api_key="x")
|
||||
req = _fake_request(settings)
|
||||
|
||||
# Both shapes now route to the EPT verifier, because alg:none is never
|
||||
# valid for ANY caller — so 401 "your token is bad" is the honest answer,
|
||||
# not 503 "that feature isn't ready".
|
||||
for typ, expected in (("JWT", "ept_invalid"), ("EPT", "ept_invalid")):
|
||||
def seg(d):
|
||||
return _b64.urlsafe_b64encode(_json.dumps(d).encode()).rstrip(b"=").decode()
|
||||
forged = (f"{seg({'alg':'none','typ':typ})}"
|
||||
f".{seg({'passport':'ET26-1EF9-VJAN','sub':'ET26-1EF9-VJAN'})}.sig")
|
||||
with _pytest.raises(RepairPointer) as exc:
|
||||
await get_caller(req, authorization=f"Bearer {forged}", x_service_token=None)
|
||||
assert exc.value.status_code in (401, 403, 503), f"{typ} was not refused"
|
||||
assert exc.value.code == expected, f"{typ} -> {exc.value.code}"
|
||||
|
||||
|
||||
@_pytest.mark.asyncio
|
||||
async def test_security_no_bearer_is_still_401():
|
||||
from api.app.auth import get_caller
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
req = _fake_request(Settings(environment="production"))
|
||||
with _pytest.raises(RepairPointer) as exc:
|
||||
await get_caller(req, authorization=None, x_service_token=None)
|
||||
assert exc.value.status_code == 401
|
||||
|
||||
|
||||
def test_i12_build_fails_when_commit_sha_is_empty():
|
||||
"""The sed+grep pair silently accepted an empty COMMIT_SHA: it replaced ""
|
||||
with "" and then matched that same empty string, shipping a container that
|
||||
reported commit_sha: null. That is the exact defect I-12 exists to prevent,
|
||||
and it happened on 2026-08-14."""
|
||||
df = (ROOT / "Dockerfile").read_text()
|
||||
assert 'test -n "${COMMIT_SHA}"' in df
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# G2.3 — branding lives in the repo, not only on one host's disk
|
||||
# --------------------------------------------------------------------------
|
||||
def test_g23_branding_is_version_controlled():
|
||||
"""It was applied directly to Veron's disk first, which is the config-drift
|
||||
trap this project documents: the running system and the repo disagree, and
|
||||
a rebuild silently reverts to stock Gitea."""
|
||||
b = ROOT / "deploy" / "branding"
|
||||
for f in ("apply.sh", "README.md", "templates/home.tmpl",
|
||||
"templates/custom/header.tmpl"):
|
||||
assert (b / f).exists(), f"missing {f}"
|
||||
|
||||
|
||||
def test_g23_brand_css_filename_is_versioned():
|
||||
"""Cloudflare caches /assets/* for 6h and no token here can purge, so a
|
||||
fixed filename leaves stale bytes live for hours."""
|
||||
import re as _re
|
||||
|
||||
hdr = (ROOT / "deploy" / "branding" / "templates" / "custom" / "header.tmpl").read_text()
|
||||
m = _re.search(r"theme-windy\.v(\d+)\.css", hdr)
|
||||
assert m, "brand CSS must carry a version in its FILENAME"
|
||||
assert (ROOT / "deploy" / "branding" / "public" / "assets" / "css"
|
||||
/ f"theme-windy.v{m.group(1)}.css").exists()
|
||||
|
||||
|
||||
def test_backup_never_bundles_credential_repos_to_r2():
|
||||
"""kit-army-config (the lockbox) and the *-soul / anima repos carry
|
||||
credentials; the R2 bundles are plaintext. Behavioural: run the script's
|
||||
own exclusion function against the names."""
|
||||
import subprocess
|
||||
|
||||
script = (ROOT / "scripts" / "backup.sh").read_text()
|
||||
fn = script[script.index('EXCLUDE="'):script.index("cleanup()")]
|
||||
# A file named like a pattern in cwd must not break the match (glob expansion).
|
||||
probe = "cd \"$(mktemp -d)\" && touch x-soul && " + fn + (
|
||||
'for n in kit-army-config anima windy-0-soul kit-0c5-soul herm-0-soul '
|
||||
'soulsafe windy-chat eternitas; do excluded "$n" && echo "X $n" || echo "- $n"; done'
|
||||
)
|
||||
out = subprocess.run(["bash", "-c", probe], capture_output=True, text=True, check=True).stdout
|
||||
skipped = {ln[2:] for ln in out.splitlines() if ln.startswith("X ")}
|
||||
assert skipped == {"kit-army-config", "anima", "windy-0-soul", "kit-0c5-soul", "herm-0-soul"}
|
||||
|
||||
64
api/tests/test_lockbox_names.py
Normal file
64
api/tests/test_lockbox_names.py
Normal file
@@ -0,0 +1,64 @@
|
||||
"""lockbox-names: headings + labels + resolvable, NEVER a value or prose after a label."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
V1 = "Qm7xT2vLp9RkZw4HnB8dYc3S" # synthetic values
|
||||
V2 = "other-fake-value-1234567890"
|
||||
V3 = "dup-one-aaaaaaaaaaaaaaaa"
|
||||
V4 = "dup-two-bbbbbbbbbbbbbbbb"
|
||||
PROSE = "ZZPROSEZZ-not-for-printing"
|
||||
|
||||
|
||||
def make(tmp_path):
|
||||
r = tmp_path / "kit"
|
||||
(r / "secrets" / "x").mkdir(parents=True)
|
||||
(r / "ACCESS_LOCKBOX.md").write_text(
|
||||
"# LOCKBOX\n\n## 🔷 AZURE signing (added 10-01)\n"
|
||||
f"- **Tenant:** {PROSE} lives in the portal\n"
|
||||
f"- **`AZURE_CLIENT_ID`**: `{V1}`\n"
|
||||
f"- **Secret (AZURE_CLIENT_SECRET):** `{V2}`\n"
|
||||
"## GOOGLE oauth\n"
|
||||
f"GOOGLE_OAUTH_CLIENT_ID={V2}\n"
|
||||
f"- **`DUP_KEY`**: `{V3}`\n- **`DUP_KEY`**: `{V4}`\n"
|
||||
f"## stray\n**{V1}** is a heading-like bold that is secret shaped? no, just label\n")
|
||||
(r / "secrets" / "x" / "a.env").write_text(f"FILE_KEY={V1}\n")
|
||||
subprocess.run(["git", "init", "-q", "-b", "main"], cwd=r, check=True)
|
||||
return r
|
||||
|
||||
|
||||
def run(r, *args):
|
||||
e = {**os.environ, "LOCKBOX_REPO": str(r), "LOCKBOX_REF": "WORKTREE"}
|
||||
p = subprocess.run([sys.executable, str(ROOT / "scripts" / "lockbox_names.py"), *args],
|
||||
capture_output=True, text=True, env=e)
|
||||
return p.returncode, p.stdout + p.stderr
|
||||
|
||||
|
||||
def test_lists_labels_and_resolvable_without_values_or_prose(tmp_path):
|
||||
r = make(tmp_path)
|
||||
rc, out = run(r, "AZURE|GOOGLE|FILE|DUP")
|
||||
assert rc == 0
|
||||
assert "AZURE signing (added 10-01) | AZURE_CLIENT_ID | md | yes" in out
|
||||
assert "| GOOGLE_OAUTH_CLIENT_ID | env | yes" in out
|
||||
assert "| FILE_KEY | file | yes" in out
|
||||
assert "| DUP_KEY | md | dup" in out
|
||||
# a prose label is listed but never resolvable, and nothing after the label leaks
|
||||
assert "| Tenant: " not in out or "| Tenant" in out
|
||||
assert "| label | no" in out
|
||||
for secret in (V1, V2, V3, V4, PROSE, "lives in the portal"):
|
||||
assert secret not in out
|
||||
for i in range(0, len(secret) - 7):
|
||||
assert secret[i:i + 8] not in out
|
||||
|
||||
|
||||
def test_filter_and_bad_regex(tmp_path):
|
||||
r = make(tmp_path)
|
||||
rc, out = run(r, "NOSUCHTHING")
|
||||
assert rc == 0 and "0 entries" in out
|
||||
rc, out = run(r, "(")
|
||||
assert rc == 2 and "bad regex" in out
|
||||
77
api/tests/test_lockbox_put.py
Normal file
77
api/tests/test_lockbox_put.py
Normal file
@@ -0,0 +1,77 @@
|
||||
"""lockbox-put against a LOCAL fake lockbox repo only (never the real one)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
FAKE = "Zk3vQ8mT1pLw7Xn2Rb5Hd9Yc" # synthetic
|
||||
|
||||
|
||||
def sh(*a, cwd=None):
|
||||
return subprocess.run(a, cwd=cwd, check=True, capture_output=True, text=True)
|
||||
|
||||
|
||||
def make_remote(tmp_path):
|
||||
work = tmp_path / "seed"
|
||||
work.mkdir()
|
||||
sh("git", "init", "-q", "-b", "main", cwd=work)
|
||||
(work / "ACCESS_LOCKBOX.md").write_text("# LOCKBOX\n\n- **`EXISTING_KEY`**: `abcdefgh12345678`\nOLD_ENV_KEY=whatever123456\n")
|
||||
sh("git", "add", "-A", cwd=work)
|
||||
sh("git", "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", "seed", cwd=work)
|
||||
bare = tmp_path / "remote.git"
|
||||
sh("git", "clone", "-q", "--bare", str(work), str(bare))
|
||||
return bare
|
||||
|
||||
|
||||
def put(tmp_path, bare, key, content, mode=0o600):
|
||||
f = tmp_path / "val"
|
||||
f.write_text(content)
|
||||
os.chmod(f, mode)
|
||||
e = {**os.environ, "LOCKBOX_PUT_REPO": str(bare), "LOCKBOX_PUT_NO_PR": "1", "HOME": str(tmp_path / "h")}
|
||||
(tmp_path / "h" / ".cache").mkdir(parents=True, exist_ok=True)
|
||||
r = subprocess.run([sys.executable, str(ROOT / "scripts" / "lockbox_put.py"), key, str(f), "--lane", "test", "--note", "n"],
|
||||
capture_output=True, text=True, env=e)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
|
||||
|
||||
def test_appends_one_key_by_branch_and_never_echoes_value(tmp_path):
|
||||
bare = make_remote(tmp_path)
|
||||
rc, out = put(tmp_path, bare, "NEW_TEST_KEY", FAKE)
|
||||
assert rc == 0 and "pushed branch lockbox-put/new_test_key-" in out
|
||||
assert FAKE not in out and FAKE[:6] not in out
|
||||
br = [b.strip() for b in sh("git", "branch", "--list", "lockbox-put/*", cwd=bare).stdout.splitlines()]
|
||||
assert len(br) == 1
|
||||
diff = sh("git", "diff", "--numstat", f"main..{br[0]}", cwd=bare).stdout.split()
|
||||
assert diff[1] == "0" and diff[2] == "ACCESS_LOCKBOX.md" # additions only
|
||||
content = sh("git", "show", f"{br[0]}:ACCESS_LOCKBOX.md", cwd=bare).stdout
|
||||
assert f"- **`NEW_TEST_KEY`**: `{FAKE}`" in content and "EXISTING_KEY" in content
|
||||
# main is untouched
|
||||
assert FAKE not in sh("git", "show", "main:ACCESS_LOCKBOX.md", cwd=bare).stdout
|
||||
|
||||
|
||||
def test_refuses_existing_key_both_formats_and_bad_input(tmp_path):
|
||||
bare = make_remote(tmp_path)
|
||||
for k in ("EXISTING_KEY", "OLD_ENV_KEY"):
|
||||
rc, out = put(tmp_path, bare, k, FAKE)
|
||||
assert rc == 3 and "already exists" in out and FAKE not in out
|
||||
assert put(tmp_path, bare, "lower_case", FAKE)[0] == 2
|
||||
assert put(tmp_path, bare, "OK_KEY_1", FAKE, mode=0o644)[0] == 2 # not 0600
|
||||
assert put(tmp_path, bare, "OK_KEY_2", "has space `tick`")[0] == 2 # unsafe value
|
||||
assert not sh("git", "branch", "--list", "lockbox-put/*", cwd=bare).stdout.strip() # nothing pushed
|
||||
|
||||
|
||||
def test_symlink_refused(tmp_path):
|
||||
bare = make_remote(tmp_path)
|
||||
real = tmp_path / "real"
|
||||
real.write_text(FAKE)
|
||||
os.chmod(real, 0o600)
|
||||
link = tmp_path / "link"
|
||||
link.symlink_to(real)
|
||||
e = {**os.environ, "LOCKBOX_PUT_REPO": str(bare), "LOCKBOX_PUT_NO_PR": "1"}
|
||||
r = subprocess.run([sys.executable, str(ROOT / "scripts" / "lockbox_put.py"), "SYM_KEY", str(link)],
|
||||
capture_output=True, text=True, env=e)
|
||||
assert r.returncode == 2 and FAKE not in r.stdout + r.stderr
|
||||
463
api/tests/test_pr_status_bridge.py
Normal file
463
api/tests/test_pr_status_bridge.py
Normal file
@@ -0,0 +1,463 @@
|
||||
"""Behavioral tests for scripts/pr_status_bridge.py.
|
||||
|
||||
The bridge is the ONLY CI signal the private platform repos get on GitHub, so
|
||||
these drive its real functions against fake Gitea/GitHub APIs rather than
|
||||
grepping its source: a status painted green that nobody tested is worse than
|
||||
no status at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"pr_status_bridge", ROOT / "scripts" / "pr_status_bridge.py"
|
||||
)
|
||||
bridge = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(bridge)
|
||||
|
||||
SHA = "a" * 40
|
||||
|
||||
|
||||
def _run(i, wf, job, status, sha=SHA, n=1):
|
||||
return {
|
||||
"id": i,
|
||||
"workflow_id": wf,
|
||||
"name": job,
|
||||
"status": status,
|
||||
"head_sha": sha,
|
||||
"run_number": n,
|
||||
}
|
||||
|
||||
|
||||
class Fake:
|
||||
def __init__(self, runs=(), statuses=(), gh_prs=(), wg_prs=(), workflows=None):
|
||||
self.runs, self.statuses = list(runs), list(statuses)
|
||||
self.workflows = workflows or {} # {path: yaml text} at every commit
|
||||
self.gh_prs, self.wg_prs = list(gh_prs), list(wg_prs)
|
||||
self.posted, self.opened, self.closed = [], [], []
|
||||
|
||||
def gitea(self, method, path, body=None):
|
||||
if "/contents/" in path:
|
||||
want = path.split("/contents/", 1)[1].split("?", 1)[0]
|
||||
if want in self.workflows:
|
||||
return 200, {"content": base64.b64encode(self.workflows[want].encode()).decode()}
|
||||
files = [
|
||||
{"type": "file", "name": k.rsplit("/", 1)[1], "path": k}
|
||||
for k in self.workflows
|
||||
if k.rsplit("/", 1)[0] == want
|
||||
]
|
||||
return (200, files) if files else (404, None)
|
||||
if "/actions/tasks" in path:
|
||||
page = int(path.rsplit("page=", 1)[1])
|
||||
return 200, {"workflow_runs": self.runs[(page - 1) * 50 : page * 50]}
|
||||
if method == "GET" and path.endswith("/pulls?state=open&limit=50"):
|
||||
return 200, self.wg_prs
|
||||
if method == "POST" and path.endswith("/pulls"):
|
||||
self.opened.append(body)
|
||||
return 201, {}
|
||||
if method == "PATCH":
|
||||
self.closed.append(path)
|
||||
return 201, {}
|
||||
raise AssertionError(path)
|
||||
|
||||
def github(self, method, path, body=None):
|
||||
if "/statuses" in path and method == "GET":
|
||||
return 200, self.statuses
|
||||
if "/statuses/" in path and method == "POST":
|
||||
self.posted.append(body)
|
||||
return 201, {}
|
||||
if "/pulls?" in path:
|
||||
return 200, self.gh_prs
|
||||
raise AssertionError(path)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake(monkeypatch):
|
||||
def make(queued=(), **kw):
|
||||
f = Fake(**kw)
|
||||
monkeypatch.setattr(bridge, "gitea", f.gitea)
|
||||
monkeypatch.setattr(bridge, "github", f.github)
|
||||
monkeypatch.setattr(bridge, "queued_jobs", lambda repo, sha: list(queued))
|
||||
return f
|
||||
|
||||
return make
|
||||
|
||||
|
||||
def test_posts_latest_verdict_per_job(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "failure"), _run(2, "ci.yml", "test", "success", n=2)])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "success")]
|
||||
assert f.posted[0]["target_url"].endswith("/actions/runs/2")
|
||||
|
||||
|
||||
def test_unchanged_state_is_not_reposted(fake):
|
||||
f = fake(
|
||||
runs=[_run(1, "ci.yml", "test", "success")],
|
||||
statuses=[{"context": "windy-git/ci/test", "state": "success"}],
|
||||
)
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_skipped_job_is_never_painted_green(fake):
|
||||
f = fake(runs=[_run(1, "substrate-drift.yml", "check", "skipped")])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_other_commits_runs_are_ignored(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "failure", sha="b" * 40)])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_runs_past_the_first_page_are_seen(fake):
|
||||
noise = [_run(100 + i, "drift.yml", "x", "skipped", sha="c" * 40) for i in range(50)]
|
||||
f = fake(runs=noise + [_run(1, "ci.yml", "test", "success")])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert [p["state"] for p in f.posted] == ["success"]
|
||||
|
||||
|
||||
def _gh_pr(n, repo="sneakyfree/r"):
|
||||
return {
|
||||
"number": n,
|
||||
"title": "t",
|
||||
"html_url": "u",
|
||||
"head": {"ref": f"b{n}", "sha": SHA, "repo": {"full_name": repo} if repo else None},
|
||||
"base": {"ref": "main"},
|
||||
}
|
||||
|
||||
|
||||
def test_fork_prs_are_never_mirrored(fake, monkeypatch):
|
||||
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
|
||||
f = fake(gh_prs=[_gh_pr(1, repo="stranger/r"), _gh_pr(2, repo=None)])
|
||||
assert bridge.sync_prs("r") == []
|
||||
assert f.opened == []
|
||||
|
||||
|
||||
def test_pr_mirror_opened_once_and_closed_when_github_closes(fake, monkeypatch):
|
||||
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
|
||||
f = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 3, "title": "[GH#5] gone"}])
|
||||
assert bridge.sync_prs("r") == [SHA]
|
||||
assert [o["head"] for o in f.opened] == ["b7"]
|
||||
assert f.closed == ["/repos/windyadmin/r/pulls/3"]
|
||||
|
||||
f2 = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 4, "title": "[GH#7] t"}])
|
||||
bridge.sync_prs("r")
|
||||
assert f2.opened == [] and f2.closed == []
|
||||
|
||||
|
||||
def test_image_build_jobs_are_not_posted(fake):
|
||||
"""No Docker daemon in job containers (I-5): a build job's red is structural."""
|
||||
f = fake(
|
||||
runs=[_run(1, "ci.yml", "Docker Build", "failure"), _run(2, "ci.yml", "docker", "failure")]
|
||||
)
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_non_blocking_jobs_are_not_posted_for_that_repo_only(fake, monkeypatch):
|
||||
"""Grant ruled windy-pro's desktop/installer jobs non-blocking: they must not
|
||||
reach GitHub for windy-pro, and the rule must not leak to other repos."""
|
||||
monkeypatch.setattr(bridge, "NON_BLOCKING", {"windy-pro": {"ci/build-desktop"}})
|
||||
runs = [_run(1, "ci.yml", "build-desktop", "failure"), _run(2, "ci.yml", "test", "success")]
|
||||
f = fake(runs=runs)
|
||||
bridge.post_statuses("windy-pro", SHA)
|
||||
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
|
||||
f2 = fake(runs=runs)
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert sorted(p["context"] for p in f2.posted) == [
|
||||
"windy-git/ci/build-desktop",
|
||||
"windy-git/ci/test",
|
||||
]
|
||||
|
||||
|
||||
def test_default_non_blocking_is_grants_ruling():
|
||||
assert bridge.NON_BLOCKING.get("windy-pro") == {
|
||||
"ci/build-desktop",
|
||||
"ci/test-installer",
|
||||
"ci/reality-check",
|
||||
}
|
||||
|
||||
|
||||
def test_transport_blips_are_retried_but_http_errors_are_not(monkeypatch):
|
||||
import urllib.error
|
||||
|
||||
calls = {"n": 0}
|
||||
|
||||
class _R:
|
||||
status = 200
|
||||
|
||||
def read(self):
|
||||
return b"{}"
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def flaky(req, timeout):
|
||||
calls["n"] += 1
|
||||
if calls["n"] < 3:
|
||||
raise urllib.error.URLError("_ssl.c:983: The handshake operation timed out")
|
||||
return _R()
|
||||
|
||||
monkeypatch.setattr(bridge.urllib.request, "urlopen", flaky)
|
||||
monkeypatch.setattr(bridge.time, "sleep", lambda s: None)
|
||||
assert bridge._call("http://x", "t", "GET", "/p") == (200, {})
|
||||
assert calls["n"] == 3
|
||||
|
||||
def forbidden(req, timeout):
|
||||
calls["n"] += 1
|
||||
raise urllib.error.HTTPError("http://x/p", 403, "no", {}, None)
|
||||
|
||||
calls["n"] = 0
|
||||
monkeypatch.setattr(bridge.urllib.request, "urlopen", forbidden)
|
||||
assert bridge._call("http://x", "t", "GET", "/p") == (403, None)
|
||||
assert calls["n"] == 1
|
||||
|
||||
|
||||
GOOD = "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: []\n"
|
||||
BROKEN = "on: push\njobs:\n test:\n runs-on: x\n steps: [\n"
|
||||
|
||||
|
||||
def test_invalid_workflow_gets_an_error_status_even_with_no_runs(fake):
|
||||
# Gitea fires NO run for an invalid file: without this the PR shows nothing.
|
||||
f = fake(workflows={".github/workflows/ci.yml": BROKEN})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/workflow", "error")]
|
||||
assert "invalid YAML at line 5" in f.posted[0]["description"]
|
||||
assert f.posted[0]["target_url"].endswith(f"/src/commit/{SHA}/.github/workflows/ci.yml")
|
||||
|
||||
|
||||
def test_valid_workflows_post_nothing_extra(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "success")], workflows={".github/workflows/ci.yml": GOOD})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
|
||||
|
||||
|
||||
def test_workflow_error_is_not_reposted(fake):
|
||||
f = fake(
|
||||
workflows={".github/workflows/ci.yml": BROKEN},
|
||||
statuses=[{"context": "windy-git/ci/workflow", "state": "error"}],
|
||||
)
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_gitea_dir_wins_over_github_dir(fake):
|
||||
# Gitea runs .gitea/workflows when it has files and ignores .github/workflows.
|
||||
f = fake(workflows={".gitea/workflows/ci.yml": GOOD, ".github/workflows/old.yml": BROKEN})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text, problem",
|
||||
[
|
||||
(GOOD, None),
|
||||
("on: push\njobs:\n a:\n uses: ./x.yml\n", None),
|
||||
(BROKEN, "invalid YAML at line 5"),
|
||||
("jobs:\n a:\n runs-on: x\n", "no `on:` trigger"),
|
||||
("on: push\n", "no `jobs:`"),
|
||||
("on: push\njobs:\n a:\n steps: []\n", "job `a` has no `runs-on:`"),
|
||||
("- a\n", "not a YAML mapping"),
|
||||
],
|
||||
)
|
||||
def test_workflow_problem(text, problem):
|
||||
assert bridge.workflow_problem(text) == problem
|
||||
|
||||
|
||||
def _q(n, wf, job):
|
||||
return {"run_number": n, "workflow_id": wf, "name": job}
|
||||
|
||||
|
||||
def test_queued_job_shows_pending_instead_of_nothing(fake):
|
||||
f = fake(queued=[_q(5, "ci.yml", "test")])
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "pending")]
|
||||
assert f.posted[0]["target_url"].endswith("/actions/runs/5")
|
||||
|
||||
|
||||
def test_queued_rerun_supersedes_the_stale_failure(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "failure", n=4)], queued=[_q(7, "ci.yml", "test")])
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "pending")]
|
||||
|
||||
|
||||
def test_older_queued_job_never_overrides_a_newer_verdict(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "success", n=9)], queued=[_q(3, "ci.yml", "test")])
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "success")]
|
||||
|
||||
|
||||
def test_queued_docker_and_non_blocking_jobs_stay_unposted(fake):
|
||||
f = fake(queued=[_q(2, "ci.yml", "docker-build"), _q(2, "ci.yml", "build-desktop")])
|
||||
bridge.post_statuses("windy-pro", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_queued_lookup_refuses_unsafe_input():
|
||||
assert bridge.queued_jobs("x'; drop table t;--", SHA) == []
|
||||
assert bridge.queued_jobs("windy-chat", "not-a-sha") == []
|
||||
|
||||
|
||||
def _db(monkeypatch, jobs, labels):
|
||||
import json as _json
|
||||
import subprocess as _sp
|
||||
|
||||
payload = _json.dumps({"jobs": jobs, "labels": [_json.dumps(x) for x in labels]})
|
||||
monkeypatch.setattr(
|
||||
bridge.subprocess, "run",
|
||||
lambda *a, **k: _sp.CompletedProcess(a, 0, stdout=payload, stderr=""),
|
||||
)
|
||||
|
||||
|
||||
RUNNER = ["veron-1", "linux-x64", "self-hosted", "linux", "x64"]
|
||||
|
||||
|
||||
def test_only_jobs_a_runner_can_take_are_pending(monkeypatch):
|
||||
# macos-latest is cancelled unpicked by the janitor: pending would never resolve.
|
||||
_db(monkeypatch, [
|
||||
{"run_number": 3, "workflow_id": "ci.yml", "name": "test", "runs_on": '["self-hosted","linux","x64"]'},
|
||||
{"run_number": 3, "workflow_id": "ci.yml", "name": "mac", "runs_on": '["macos-latest"]'},
|
||||
], [RUNNER])
|
||||
assert [j["name"] for j in bridge.queued_jobs("windy-chat", SHA)] == ["test"]
|
||||
|
||||
|
||||
def test_lookup_failure_is_non_fatal(monkeypatch):
|
||||
import subprocess as _sp
|
||||
|
||||
def boom(*a, **k):
|
||||
raise _sp.TimeoutExpired("docker", 30)
|
||||
|
||||
monkeypatch.setattr(bridge.subprocess, "run", boom)
|
||||
assert bridge.queued_jobs("windy-chat", SHA) == []
|
||||
|
||||
|
||||
class _Guard:
|
||||
def __init__(self, findings):
|
||||
self.findings = findings
|
||||
|
||||
def check(self, repo, sha, default_branch, is_default_head):
|
||||
return self.findings
|
||||
|
||||
@staticmethod
|
||||
def status_for(findings, whole_tree, grant=()):
|
||||
if not findings and grant:
|
||||
return "success", f"GRANT-WARN {len(grant)}", grant[0]
|
||||
if not findings:
|
||||
return "success", "OK: clean", None
|
||||
return "failure", f"BLOCK {len(findings)}", findings[0]
|
||||
|
||||
|
||||
class _F:
|
||||
path, line = "app/llm.py", 7
|
||||
|
||||
|
||||
def test_guard_posts_warn_with_a_link_to_the_first_finding(fake, monkeypatch):
|
||||
f = fake()
|
||||
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
|
||||
bridge.post_compute_guard("windy-chat", SHA, "main", False)
|
||||
assert [(p["context"], p["state"], p["description"]) for p in f.posted] == [
|
||||
("windy-git/compute-guard", "failure", "BLOCK 1")]
|
||||
assert f.posted[0]["target_url"].endswith(f"/src/commit/{SHA}/app/llm.py#L7")
|
||||
|
||||
|
||||
def test_guard_same_status_is_not_reposted(fake, monkeypatch):
|
||||
f = fake(statuses=[{"context": "windy-git/compute-guard", "state": "failure", "description": "BLOCK 1"}])
|
||||
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
|
||||
bridge.post_compute_guard("windy-chat", SHA, "main", False)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_guard_that_cannot_run_posts_nothing(fake, monkeypatch):
|
||||
f = fake()
|
||||
monkeypatch.setitem(sys.modules, "compute_guard", _Guard(None))
|
||||
bridge.post_compute_guard("windy-chat", SHA, "main", True)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_ci_hygiene_posts_under_its_own_context(fake, monkeypatch):
|
||||
f = fake(statuses=[{"context": "windy-git/compute-guard", "state": "failure", "description": "BLOCK 1"}])
|
||||
monkeypatch.setitem(sys.modules, "ci_hygiene", _Guard([_F()]))
|
||||
bridge.post_ci_hygiene("windy-chat", SHA, "main", True)
|
||||
# the compute-guard status with the same description must not suppress it
|
||||
assert [(p["context"], p["description"]) for p in f.posted] == [("windy-git/ci-hygiene", "BLOCK 1")]
|
||||
|
||||
|
||||
def test_retargeted_pr_gets_a_fresh_mirror_on_the_new_base(fake):
|
||||
# eternitas #167: stacked on fix/one-hallway, retargeted to main on GitHub.
|
||||
gh = [{"number": 167, "title": "feat", "html_url": "u",
|
||||
"head": {"ref": "feat/x", "sha": SHA, "repo": {"full_name": f"{bridge.GH_OWNER}/eternitas"}},
|
||||
"base": {"ref": "main"}}]
|
||||
wg = [{"number": 9, "title": "[GH#167] feat", "base": {"ref": "fix/one-hallway"}}]
|
||||
f = fake(gh_prs=gh, wg_prs=wg)
|
||||
assert bridge.sync_prs("eternitas") == [SHA]
|
||||
assert f.closed == [f"/repos/{bridge.WG_OWNER}/eternitas/pulls/9"]
|
||||
assert [(o["base"], o["head"]) for o in f.opened] == [("main", "feat/x")]
|
||||
|
||||
|
||||
def test_unchanged_base_leaves_the_mirror_alone(fake):
|
||||
gh = [{"number": 5, "title": "t", "html_url": "u",
|
||||
"head": {"ref": "b", "sha": SHA, "repo": {"full_name": f"{bridge.GH_OWNER}/windy-chat"}},
|
||||
"base": {"ref": "main"}}]
|
||||
f = fake(gh_prs=gh, wg_prs=[{"number": 3, "title": "[GH#5] t", "base": {"ref": "main"}}])
|
||||
bridge.sync_prs("windy-chat")
|
||||
assert f.closed == [] and f.opened == []
|
||||
|
||||
|
||||
def test_named_no_daemon_job_is_not_posted_for_that_repo_only(fake, monkeypatch):
|
||||
"""eternitas ci/build needs Docker but its name doesn't say so (option A, 09-23)."""
|
||||
monkeypatch.setattr(bridge, "NO_DAEMON_NAMED", {"eternitas": {"ci/build"}})
|
||||
runs = [_run(1, "ci.yml", "build", "failure"), _run(2, "ci.yml", "test", "success")]
|
||||
f = fake(runs=runs)
|
||||
bridge.post_statuses("eternitas", SHA)
|
||||
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
|
||||
f2 = fake(runs=runs)
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert sorted(p["context"] for p in f2.posted) == ["windy-git/ci/build", "windy-git/ci/test"]
|
||||
|
||||
|
||||
def test_default_no_daemon_named_is_empty():
|
||||
"""eternitas converted its ci/build to a no-Docker ci/smoke (#179); nothing left."""
|
||||
assert bridge.NO_DAEMON_NAMED == {}
|
||||
|
||||
|
||||
def test_grant_owned_findings_never_block(fake, monkeypatch):
|
||||
"""Orchestrator 09-23: compute-guard blocks lane-owned code only."""
|
||||
f = fake()
|
||||
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
|
||||
monkeypatch.setitem(sys.modules, "guards_report",
|
||||
type("GR", (), {"split_grant": staticmethod(lambda r, s, fs: ([], list(fs)))}))
|
||||
bridge.post_compute_guard("windy-pro", SHA, "main", True)
|
||||
assert [(p["state"], p["description"]) for p in f.posted] == [("success", "GRANT-WARN 1")]
|
||||
|
||||
|
||||
def test_failed_grant_split_warns_instead_of_blocking(fake, monkeypatch):
|
||||
def boom(*a):
|
||||
raise RuntimeError("no bare clone")
|
||||
|
||||
f = fake()
|
||||
monkeypatch.setitem(sys.modules, "compute_guard", _Guard([_F()]))
|
||||
monkeypatch.setitem(sys.modules, "guards_report", type("GR", (), {"split_grant": staticmethod(boom)}))
|
||||
bridge.post_compute_guard("windy-pro", SHA, "main", True)
|
||||
assert [p["state"] for p in f.posted] == ["success"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name, hidden", [
|
||||
("Docker Build", True), ("docker-build", True), ("Docker build", True), ("docker", True),
|
||||
("Boot smoke (no Docker)", False), ("smoke (no-docker)", False), ("boot without Docker", False),
|
||||
("smoke", False),
|
||||
])
|
||||
def test_no_docker_smoke_jobs_are_posted(name, hidden):
|
||||
"""windy-search #96: its replacement job says "no Docker" and was hidden."""
|
||||
assert bridge.needs_daemon("windy-search", "ci", name) is hidden
|
||||
82
api/tests/test_push_velocity.py
Normal file
82
api/tests/test_push_velocity.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""Push-velocity detection (scripts/telemetry_emit.py): detect + alert only.
|
||||
|
||||
Driven through the real function with rows shaped like the Gitea query's.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
_spec = importlib.util.spec_from_file_location("telemetry_emit", ROOT / "scripts" / "telemetry_emit.py")
|
||||
te = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(te)
|
||||
|
||||
NOW = 1_800_000_000.0
|
||||
|
||||
|
||||
def row(login="agent-et26abcd1234", uid=7, p1h=0, p24h=0, d24h=0, repos=1, wid=None):
|
||||
return {"uid": uid, "login": login, "wid": wid, "p1h": p1h, "p24h": p24h, "d24h": d24h, "repos": repos}
|
||||
|
||||
|
||||
def test_under_every_threshold_emits_nothing():
|
||||
ev, keep = te.push_velocity_events([row(p1h=60, p24h=500, d24h=10)], NOW, {})
|
||||
assert ev == [] and keep == {}
|
||||
|
||||
|
||||
def test_burst_emits_one_declared_row_with_the_passport():
|
||||
ev, keep = te.push_velocity_events([row(p1h=61, p24h=61, repos=3)], NOW, {})
|
||||
assert len(ev) == 1
|
||||
e = ev[0]
|
||||
assert e["event_type"] == "forge.push_velocity" and e["service"] == "forge"
|
||||
assert e["actor_type"] == "agent" and e["actor_id"] == "ET26-ABCD-1234"
|
||||
assert e["metadata"] == {
|
||||
"rule": "pushes_1h", "window_s": 3600, "count": 61, "threshold": 60,
|
||||
"repos": 3, "gitea_user_id": 7,
|
||||
}
|
||||
assert keep == {"7:pushes_1h": NOW}
|
||||
|
||||
|
||||
def test_still_over_is_reported_once_per_window_not_every_run():
|
||||
_, keep = te.push_velocity_events([row(p1h=90)], NOW, {})
|
||||
ev, keep = te.push_velocity_events([row(p1h=95)], NOW + 300, keep)
|
||||
assert ev == [] and keep == {"7:pushes_1h": NOW}
|
||||
ev, _ = te.push_velocity_events([row(p1h=95)], NOW + 3601, keep)
|
||||
assert len(ev) == 1
|
||||
|
||||
|
||||
def test_dropping_back_under_rearms():
|
||||
_, keep = te.push_velocity_events([row(p1h=90)], NOW, {})
|
||||
_, keep = te.push_velocity_events([row(p1h=5)], NOW + 300, keep)
|
||||
assert keep == {}
|
||||
ev, _ = te.push_velocity_events([row(p1h=90)], NOW + 600, keep)
|
||||
assert len(ev) == 1
|
||||
|
||||
|
||||
def test_the_sync_account_is_exempt():
|
||||
ev, _ = te.push_velocity_events([row(login="windyadmin", uid=1, p1h=9999, p24h=9999)], NOW, {})
|
||||
assert ev == []
|
||||
|
||||
|
||||
def test_sso_human_is_keyed_on_windy_identity_id():
|
||||
ev, _ = te.push_velocity_events([row(login="u-5e1b9569abc", wid="5e1b9569-full-id", d24h=11)], NOW, {})
|
||||
assert [(e["actor_type"], e["actor_id"], e["metadata"]["rule"]) for e in ev] == [
|
||||
("human", "5e1b9569-full-id", "ref_deletes_24h")
|
||||
]
|
||||
assert "caller" not in ev[0]["metadata"]
|
||||
|
||||
|
||||
def test_no_provable_id_is_system_plus_caller_never_an_invented_id():
|
||||
# UPDATE 2 actor rule: agent/human rows without an actor_id are quarantined.
|
||||
for login in ("u-nolink", "agent-weird"):
|
||||
ev, _ = te.push_velocity_events([row(login=login, p24h=501)], NOW, {})
|
||||
assert ev[0]["actor_type"] == "system" and "actor_id" not in ev[0]
|
||||
assert ev[0]["metadata"]["caller"] == "unknown"
|
||||
|
||||
|
||||
def test_passport_round_trip():
|
||||
assert te.passport_from_login("agent-et26p1zgttp8") == "ET26-P1ZG-TTP8"
|
||||
assert te.passport_from_login("u-abc") is None
|
||||
70
api/tests/test_runner_guard.py
Normal file
70
api/tests/test_runner_guard.py
Normal file
@@ -0,0 +1,70 @@
|
||||
"""runner-guard: workflow shapes that hand a self-hosted runner to strangers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
_spec = importlib.util.spec_from_file_location("runner_guard", ROOT / "scripts" / "runner_guard.py")
|
||||
rg = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["runner_guard"] = rg
|
||||
_spec.loader.exec_module(rg)
|
||||
|
||||
|
||||
def rules(text):
|
||||
return [(r, ln) for _p, ln, r, _m in rg.lint_text("w.yml", text)]
|
||||
|
||||
|
||||
def test_pull_request_target_always_fails():
|
||||
assert rules("on: pull_request_target\njobs:\n a:\n runs-on: ubuntu-latest\n steps: []\n")[0][0] == "R1"
|
||||
|
||||
|
||||
def test_fork_pr_on_self_hosted_fails_and_points_at_runs_on():
|
||||
wf = "on:\n pull_request:\njobs:\n t:\n runs-on: [self-hosted, linux, x64]\n steps: []\n"
|
||||
assert rules(wf) == [("R2", 5)]
|
||||
|
||||
|
||||
def test_same_repo_gate_or_environment_passes():
|
||||
gated = ("on: [pull_request]\njobs:\n t:\n if: github.event.pull_request.head.repo.full_name == github.repository\n"
|
||||
" runs-on: [self-hosted]\n steps: []\n")
|
||||
env = "on: [pull_request]\njobs:\n t:\n environment: ci\n runs-on: self-hosted\n steps: []\n"
|
||||
assert rules(gated) == [] and rules(env) == []
|
||||
|
||||
|
||||
def test_outsider_events_on_self_hosted_fail_but_writer_events_pass():
|
||||
wf = "on:\n issue_comment:\n workflow_run:\n workflows: [x]\njobs:\n t:\n runs-on: self-hosted\n steps: []\n"
|
||||
assert sorted(r for r, _ in rules(wf)) == ["R3", "R3"]
|
||||
ok = "on:\n push:\n tags: ['v*']\n workflow_dispatch:\n schedule:\n - cron: '0 3 * * *'\njobs:\n t:\n runs-on: self-hosted\n steps: []\n"
|
||||
assert rules(ok) == []
|
||||
|
||||
|
||||
def test_expression_runs_on_is_treated_as_self_hosted_and_hosted_runner_is_fine():
|
||||
expr = "on: pull_request\njobs:\n t:\n runs-on: ${{ matrix.os }}\n steps: []\n"
|
||||
hosted = "on: pull_request\njobs:\n t:\n runs-on: ubuntu-latest\n steps: []\n"
|
||||
assert rules(expr) == [("R2", 4)] and rules(hosted) == []
|
||||
|
||||
|
||||
def test_broken_yaml_is_a_finding_and_non_workflows_are_ignored():
|
||||
assert rules("on: [push\njobs: {")[0][0] == "R0"
|
||||
assert rules("name: just a file\n") == []
|
||||
|
||||
|
||||
def test_pr_mode_posts_each_status_once(monkeypatch, tmp_path):
|
||||
calls = []
|
||||
monkeypatch.setattr(rg, "STATE", str(tmp_path / "s.json"))
|
||||
monkeypatch.setattr(rg, "public_repos", lambda owners: [("o/r", "main")])
|
||||
monkeypatch.setattr(rg, "file_at", lambda *a: "on: push\\njobs:\\n t:\\n runs-on: self-hosted\\n steps: []\\n")
|
||||
|
||||
def fake_gh(*args, check=True):
|
||||
if args[0].startswith("repos/o/r/pulls?"):
|
||||
return "7 abc123 o/r\\n"
|
||||
if args[0].endswith("/files?per_page=100"):
|
||||
return ".github/workflows/ci.yml\\n"
|
||||
calls.append(args[0])
|
||||
return ""
|
||||
monkeypatch.setattr(rg, "gh", fake_gh)
|
||||
rg.cmd_pr(["o"], post=True)
|
||||
rg.cmd_pr(["o"], post=True)
|
||||
assert calls == ["repos/o/r/statuses/abc123"]
|
||||
156
api/tests/test_secret_guard.py
Normal file
156
api/tests/test_secret_guard.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""Secret guard: shapes, hash-only findings, allow by hash."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
_spec = importlib.util.spec_from_file_location("secret_guard", ROOT / "scripts" / "secret_guard.py")
|
||||
sg = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["secret_guard"] = sg
|
||||
_spec.loader.exec_module(sg)
|
||||
ss = sg.ss
|
||||
|
||||
# Synthetic shapes only: none of these is a real credential.
|
||||
TG = "1234567890:" + "A" * 35
|
||||
CASES = [
|
||||
("telegram bot token", f"TELEGRAM_BOT_TOKEN={TG}"),
|
||||
("github token", "token = 'ghp_" + "a1" * 18 + "'"),
|
||||
("aws access key", "aws_access_key_id = AKIA" + "ABCDEFGHIJKLMNOP"),
|
||||
("slack token", "xoxb-" + "1234567890-abcdefghij"),
|
||||
("anthropic key", "ANTHROPIC_API_KEY=sk-ant-" + "x" * 30),
|
||||
("openai key", "OPENAI_API_KEY=sk-proj-" + "y" * 40),
|
||||
("stripe live key", "STRIPE=sk_live_" + "z" * 24),
|
||||
("google api key", "key=AIza" + "B" * 35),
|
||||
("private key block", "-----BEGIN OPENSSH PRIVATE KEY-----"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind, text", CASES)
|
||||
def test_each_shape_is_found_and_only_its_hash_is_kept(kind, text):
|
||||
hits = sg.scan_line("app.py", text)
|
||||
assert [k for k, _ in hits] == [kind]
|
||||
match = hits[0][1]
|
||||
assert match.startswith(f"{kind} #") and len(match.rsplit("#", 1)[1]) == 8
|
||||
# house rule 10: the value itself must never appear in a finding
|
||||
secret = text.split("=", 1)[-1].strip(" '")
|
||||
assert secret not in match
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", [
|
||||
"sha512-" + "Q" * 86 + "==", # lockfile integrity
|
||||
"version: 12345678:abc", # short, not a token
|
||||
"sk-ant-short", # too short
|
||||
"re_test_register_sends_verification", # windy-pro's fake Resend key
|
||||
"ANTHROPIC_API_KEY=", # a name, not a value
|
||||
])
|
||||
def test_non_secrets_are_not_flagged(text):
|
||||
assert sg.scan_line("x", text) == []
|
||||
|
||||
|
||||
def test_anthropic_key_is_not_double_counted_as_openai():
|
||||
assert [k for k, _ in sg.scan_line("x", "sk-ant-" + "q" * 40)] == ["anthropic key"]
|
||||
|
||||
|
||||
def test_allow_is_by_hash_only():
|
||||
F = sg.cg.Finding
|
||||
fake = F("tests/t.py", 3, "telegram bot token", f"telegram bot token #{ss.h8(TG)}")
|
||||
real = F("tests/t.py", 9, "telegram bot token", "telegram bot token #deadbeef")
|
||||
allow = {"windy-chat": {"hashes": {ss.h8(TG)}, "paths": []}}
|
||||
assert sg._drop_allowed("windy-chat", [fake, real], allow) == [real]
|
||||
assert sg._drop_allowed("windy-mail", [fake], allow) == [fake]
|
||||
|
||||
|
||||
def test_private_key_blocks_are_allowed_by_path_never_by_hash():
|
||||
F = sg.cg.Finding
|
||||
hdr = "private key block #" + ss.h8("-----BEGIN PRIVATE KEY-----")
|
||||
test_key = F("tests/keys/test.pem", 1, "private key block", hdr)
|
||||
prod_key = F("deploy/prod.pem", 1, "private key block", hdr)
|
||||
allow = {"r": {"hashes": {hdr.rsplit("#", 1)[1]}, "paths": [("tests/keys/*", {"private key block"})]}}
|
||||
assert sg._drop_allowed("r", [test_key, prod_key], allow) == [prod_key]
|
||||
|
||||
|
||||
def test_path_allow_cannot_cover_real_token_kinds(tmp_path):
|
||||
bad = tmp_path / "a.yml"
|
||||
bad.write_text("allow:\n - repo: r\n paths: [tests/*]\n kinds: [telegram bot token]\n reason: no\n")
|
||||
with pytest.raises(ValueError):
|
||||
sg.load_allow(bad)
|
||||
|
||||
|
||||
def test_shipped_allow_file_never_excuses_the_real_leaked_tokens():
|
||||
a = sg.load_allow()
|
||||
every = set().union(*(v["hashes"] for v in a.values())) if a else set()
|
||||
assert not {"1354fc9b", "d49dc2ba"} & every # real (now revoked) credentials: remove, never allow
|
||||
|
||||
|
||||
def test_allow_file_loads_and_needs_reasons(tmp_path):
|
||||
assert isinstance(sg.load_allow(), dict)
|
||||
bad = tmp_path / "a.yml"
|
||||
bad.write_text("allow:\n - repo: r\n hashes: [abcd1234]\n")
|
||||
with pytest.raises(ValueError):
|
||||
sg.load_allow(bad)
|
||||
|
||||
|
||||
def test_block_and_warn(monkeypatch):
|
||||
f = sg.cg.Finding("a.py", 1, "github token", "github token #abcd1234")
|
||||
monkeypatch.setattr(sg, "MODE", "block")
|
||||
assert sg.status_for([f], False)[0] == "failure"
|
||||
assert sg.status_for([], False, grant=[f])[0] == "success"
|
||||
monkeypatch.setattr(sg, "MODE", "warn")
|
||||
state, desc, _ = sg.status_for([f], True)
|
||||
assert state == "success" and desc.startswith("⚠ WARN (not blocking): 1 secret-shaped string in tree")
|
||||
|
||||
|
||||
def test_public_scan_excuses_by_hash_and_by_path():
|
||||
spec = importlib.util.spec_from_file_location("public_secret_scan", ROOT / "scripts" / "public_secret_scan.py")
|
||||
ps = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(ps)
|
||||
allow = {"windy-agent": {"hashes": {"aaaa1111"}, "paths": [("tests/keys/*", {"private key block"})]}}
|
||||
assert ps.excused("windy-agent", "openai key", "aaaa1111", ["tests/x.py"], allow)
|
||||
assert not ps.excused("windy-agent", "telegram bot token", "1354fc9b", ["tests/test_log_redaction.py"], allow)
|
||||
assert ps.excused("windy-agent", "private key block", "ffff0000", ["tests/keys/a.pem"], allow)
|
||||
# a PEM header anywhere outside the allowed paths still counts
|
||||
assert not ps.excused("windy-agent", "private key block", "ffff0000", ["tests/keys/a.pem", "deploy/k.pem"], allow)
|
||||
assert not ps.excused("other", "openai key", "aaaa1111", ["x"], allow)
|
||||
|
||||
|
||||
HEX32 = "0123456789abcdef" * 2 # synthetic
|
||||
|
||||
|
||||
def test_twilio_shapes_hash_only_and_no_md5_noise():
|
||||
kinds = lambda t: [k for k, _ in ss.find(t)] # noqa: E731
|
||||
assert kinds(f'TWILIO_AUTH_TOKEN = "{HEX32}"') == ["32-hex secret assignment"]
|
||||
assert kinds(f"auth_token: {HEX32}") == ["32-hex secret assignment"]
|
||||
assert kinds("AC" + HEX32) == ["twilio sid/api key"]
|
||||
assert kinds("SK" + HEX32) == ["twilio sid/api key"]
|
||||
# plain md5 / uuid-without-dashes / a 64-hex sha256 are NOT secrets by shape
|
||||
assert kinds(f"md5 = {HEX32}") == []
|
||||
assert kinds(f"checksum_key = {HEX32}{HEX32}") == []
|
||||
assert kinds(f"name = 'x{HEX32}'") == []
|
||||
# the hash is of the value alone, so renaming the variable keeps the same allow hash
|
||||
a = ss.find(f"A_TOKEN={HEX32}")[0][1]
|
||||
b = ss.find(f"OTHER_SECRET: '{HEX32}'")[0][1]
|
||||
assert a == b == ss.h8(HEX32)
|
||||
assert HEX32 not in repr(ss.find(f"A_TOKEN={HEX32}"))
|
||||
|
||||
|
||||
def test_warn_kinds_do_not_block(monkeypatch):
|
||||
f = sg.cg.Finding("a.py", 1, "32-hex secret assignment", "32-hex secret assignment #abcd1234")
|
||||
monkeypatch.setattr(sg, "MODE", "block")
|
||||
monkeypatch.setattr(sg, "WARN_KINDS", {"32-hex secret assignment"})
|
||||
assert sg.status_for([f], True)[0] == "success"
|
||||
monkeypatch.setattr(sg, "WARN_KINDS", set())
|
||||
assert sg.status_for([f], True)[0] == "failure"
|
||||
|
||||
|
||||
def test_pypi_token_shape_hash_only():
|
||||
tok = "pypi-AgE" + "Ab1_-" * 20 # synthetic
|
||||
got = ss.find(f"password = {tok}")
|
||||
assert [k for k, _ in got] == ["pypi token"] and got[0][1] == ss.h8(tok)
|
||||
assert tok not in repr(got)
|
||||
assert ss.find("pypi-AgE-too-short") == []
|
||||
106
api/tests/test_secret_scan.py
Normal file
106
api/tests/test_secret_scan.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""secret-scan + env-names: findings carry label/location/hash, NEVER a value or fragment."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SCRIPTS = ROOT / "scripts"
|
||||
# Synthetic, random-looking, never real. FAKE_LB is in the fake lockbox; TWI matches a shape.
|
||||
FAKE_LB = "k7Xv2QpLm9RtZw4HnB8dYc3S"
|
||||
TWI = "9f3a7c1e5b2d48806a1f4e7d2c9b0835"
|
||||
|
||||
|
||||
def run(script, *args, env=None):
|
||||
e = {**os.environ, **(env or {})}
|
||||
r = subprocess.run([sys.executable, str(SCRIPTS / script), *args], capture_output=True, text=True, env=e)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
|
||||
|
||||
def no_fragment(out: str, value: str, n: int = 6):
|
||||
assert value not in out
|
||||
for i in range(len(value) - n + 1):
|
||||
assert value[i:i + n] not in out, f"fragment of the value leaked at {i}"
|
||||
|
||||
|
||||
def seeded(tmp_path):
|
||||
lb = tmp_path / "lockbox.md"
|
||||
lb.write_text(f"FAKE_VENDOR_API_KEY={FAKE_LB}\nNOTE: nothing here\n")
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
g = lambda *a: subprocess.run(["git", "-C", str(repo), *a], check=True, capture_output=True) # noqa: E731
|
||||
g("init", "-q", "-b", "main")
|
||||
g("config", "user.email", "t@t")
|
||||
g("config", "user.name", "t")
|
||||
(repo / "app.py").write_text(f'KEY = "{FAKE_LB}"\nTWILIO_AUTH_TOKEN = "{TWI}"\n')
|
||||
g("add", "-A")
|
||||
g("commit", "-qm", "add secrets")
|
||||
(repo / "app.py").write_text("KEY = None\n") # removed from HEAD, still in history
|
||||
g("commit", "-qam", "remove")
|
||||
return lb, repo
|
||||
|
||||
|
||||
def test_tree_scan_labels_locations_no_value(tmp_path):
|
||||
lb, repo = seeded(tmp_path)
|
||||
(repo / "live.py").write_text(f'x = "{FAKE_LB}"\n')
|
||||
rc, out = run("secret_scan.py", str(repo / "live.py"), env={"SECRET_SCAN_LOCKBOX_PATHS": str(lb)})
|
||||
assert rc == 1
|
||||
assert "live.py:1" in out and "lockbox:FAKE_VENDOR_API_KEY" in out
|
||||
no_fragment(out, FAKE_LB)
|
||||
|
||||
|
||||
def test_history_finds_removed_secret_with_commit(tmp_path):
|
||||
lb, repo = seeded(tmp_path)
|
||||
rc, out = run("secret_scan.py", str(repo), "--history", env={"SECRET_SCAN_LOCKBOX_PATHS": str(lb)})
|
||||
assert rc == 1
|
||||
assert "app.py:1" in out and "commit=" in out and "lockbox:FAKE_VENDOR_API_KEY" in out
|
||||
assert "app.py:2" in out and "32-hex secret assignment" in out
|
||||
no_fragment(out, FAKE_LB)
|
||||
no_fragment(out, TWI)
|
||||
|
||||
|
||||
def test_repo_flag_clones_scans_and_cleans_up(tmp_path):
|
||||
lb, repo = seeded(tmp_path)
|
||||
cache = tmp_path / "home"
|
||||
(cache / ".cache").mkdir(parents=True)
|
||||
rc, out = run("secret_scan.py", "--repo", str(repo),
|
||||
env={"SECRET_SCAN_LOCKBOX_PATHS": str(lb), "HOME": str(cache)})
|
||||
assert rc == 1 and "app.py:1" in out
|
||||
no_fragment(out, FAKE_LB)
|
||||
assert not [p for p in (cache / ".cache").iterdir() if p.name.startswith("secret-scan-")]
|
||||
|
||||
|
||||
def test_clean_tree_and_bad_input(tmp_path):
|
||||
(tmp_path / "ok.txt").write_text("hello world\n")
|
||||
rc, out = run("secret_scan.py", str(tmp_path / "ok.txt"), "--no-lockbox")
|
||||
assert rc == 0 and "0 finding(s)" in out
|
||||
rc, out = run("secret_scan.py", str(tmp_path), "--history", "--no-lockbox")
|
||||
assert rc == 2 and "needs a git repo" in out
|
||||
|
||||
|
||||
def test_env_names_never_prints_values(tmp_path):
|
||||
a = tmp_path / "a.env"
|
||||
b = tmp_path / "b.env"
|
||||
a.write_text(f"# c\nexport DB_PASSWORD={FAKE_LB}\nTOKEN=\"{TWI}\"\nEMPTY=\nONLY_A=1\n")
|
||||
b.write_text(f"DB_PASSWORD={FAKE_LB}\nTOKEN=different-value-here\nONLY_B=2\n")
|
||||
rc, out = run("env_names.py", str(a), "--hash")
|
||||
assert rc == 0 and "DB_PASSWORD" in out and "set" in out and "empty" in out and "len=24" in out
|
||||
assert "sha256:" in out
|
||||
no_fragment(out, FAKE_LB)
|
||||
no_fragment(out, TWI, 7)
|
||||
rc, out = run("env_names.py", str(a), "--compare", str(b))
|
||||
assert "DB_PASSWORD" in out and "SAME" in out and "DIFFERENT" in out
|
||||
assert "only-in-A" in out and "only-in-B" in out
|
||||
no_fragment(out, FAKE_LB)
|
||||
rc, out = run("env_names.py", str(tmp_path / "missing.env"))
|
||||
assert rc == 2
|
||||
|
||||
|
||||
def test_shapes_are_the_guards_shapes():
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
import secret_scan as sc
|
||||
import secret_shapes as ss
|
||||
assert sc.ss is ss # one source of shapes: a new guard shape is automatically a scan shape
|
||||
202
api/tests/test_telemetry.py
Normal file
202
api/tests/test_telemetry.py
Normal file
@@ -0,0 +1,202 @@
|
||||
"""Field telemetry from the API (step 2): behavioural, no network.
|
||||
|
||||
A refusal must become exactly one forge.auth.failed row in the DECLARED shape
|
||||
(the ledger quarantines anything else); non-refusal errors must not; the
|
||||
heartbeat must count what happened and never invent a p95 for no traffic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import Depends, FastAPI
|
||||
|
||||
from api.app import telemetry as tmod
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
DECLARED_AUTH_KEYS = {"code", "http_status", "caller", "route", "upstream_status", "synthetic"}
|
||||
|
||||
|
||||
def _app(tel: tmod.Telemetry) -> FastAPI:
|
||||
"""The real middleware + handler, re-registered on a bare app (no DB)."""
|
||||
from api.app import main
|
||||
|
||||
app = FastAPI()
|
||||
app.state.telemetry = tel
|
||||
app.middleware("http")(main._count_requests)
|
||||
app.exception_handler(RepairPointer)(main._repair_pointer_handler)
|
||||
|
||||
def refuse(code: str, status: int):
|
||||
def dep():
|
||||
raise RepairPointer(
|
||||
status_code=status,
|
||||
code=code,
|
||||
speak="no",
|
||||
machine_cause="test",
|
||||
remediation_tool=None,
|
||||
)
|
||||
|
||||
return dep
|
||||
|
||||
@app.get("/api/v1/repos/{repo}/grants", dependencies=[Depends(refuse("passport_revoked", 403))])
|
||||
async def grants(repo: str):
|
||||
return {}
|
||||
|
||||
@app.get("/api/v1/nope", dependencies=[Depends(refuse("repo_not_found", 404))])
|
||||
async def nope():
|
||||
return {}
|
||||
|
||||
@app.get("/ok")
|
||||
async def ok():
|
||||
return {"ok": True}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
async def _get(app, path, headers=None):
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://t") as c:
|
||||
return await c.get(path, headers=headers or {})
|
||||
|
||||
|
||||
def _tel():
|
||||
return tmod.Telemetry(
|
||||
"http://ledger.invalid/v1/events",
|
||||
"tok",
|
||||
environment="test",
|
||||
commit_sha="abc1234",
|
||||
version="0.1.0",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refusal_emits_one_declared_row_with_route_template_not_path():
|
||||
tel = _tel()
|
||||
r = await _get(
|
||||
_app(tel),
|
||||
"/api/v1/repos/grandmas-secret-project/grants",
|
||||
{"Authorization": "Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkVQVCJ9.e30.x"},
|
||||
)
|
||||
assert r.status_code == 403
|
||||
rows = [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row["actor_type"] == "system" and "actor_id" not in row
|
||||
assert set(row["metadata"]) <= DECLARED_AUTH_KEYS
|
||||
assert row["metadata"]["code"] == "passport_revoked"
|
||||
assert row["metadata"]["http_status"] == 403
|
||||
assert row["metadata"]["caller"] == "anonymous_agent"
|
||||
assert row["metadata"]["route"] == "/api/v1/repos/{repo}/grants"
|
||||
assert "grandmas-secret-project" not in str(row)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_auth_errors_are_counted_but_not_refusal_rows():
|
||||
tel = _tel()
|
||||
await _get(_app(tel), "/api/v1/nope")
|
||||
assert not [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
|
||||
assert tel.errors_4xx == 1 and tel.refusals_4xx == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_heartbeat_counts_requests_refusals_and_p95():
|
||||
tel = _tel()
|
||||
app = _app(tel)
|
||||
for _ in range(3):
|
||||
await _get(app, "/ok")
|
||||
await _get(app, "/api/v1/repos/x/grants")
|
||||
meta = tel.health_row()
|
||||
assert meta["requests"] == 4 and meta["refusals_4xx"] == 1 and meta["errors_5xx"] == 0
|
||||
assert isinstance(meta["p95_ms"], int)
|
||||
assert {"interval_s", "uptime_s"} <= set(meta)
|
||||
|
||||
|
||||
def test_no_traffic_means_no_p95_not_a_fake_zero():
|
||||
assert "p95_ms" not in _tel().health_row()
|
||||
|
||||
|
||||
def test_no_token_sends_and_buffers_nothing():
|
||||
tel = tmod.Telemetry("http://ledger.invalid", "")
|
||||
tel.boot()
|
||||
tel.auth_failed(code="token_invalid", http_status=401, caller="unknown")
|
||||
assert tel.buffer == []
|
||||
|
||||
|
||||
def test_unknown_code_is_never_sent_as_a_refusal():
|
||||
tel = _tel()
|
||||
tel.auth_failed(code="made_up_code", http_status=401, caller="unknown")
|
||||
assert tel.buffer == []
|
||||
|
||||
|
||||
def test_boot_omits_an_unknown_commit_rather_than_inventing_one():
|
||||
tel = tmod.Telemetry("http://x", "tok", commit_sha=None, version="0.1.0")
|
||||
tel.boot()
|
||||
assert "commit_sha" not in tel.buffer[0]["metadata"]
|
||||
|
||||
|
||||
def test_caller_classes_are_the_declared_three():
|
||||
assert tmod.caller_class({}) == "unknown"
|
||||
assert tmod.caller_class({"x-service-token": "s"}) == "unknown"
|
||||
assert (
|
||||
tmod.caller_class({"authorization": "Bearer eyJhbGciOiJSUzI1NiJ9.e30.x"})
|
||||
== "anonymous_human"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_synthetic_header_marks_the_row_and_absent_means_real():
|
||||
async def refusal(headers):
|
||||
tel = _tel()
|
||||
await _get(_app(tel), "/api/v1/repos/x/grants", headers)
|
||||
return [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"][0]["metadata"]["synthetic"]
|
||||
|
||||
assert await refusal({"X-Windy-Synthetic": "1"}) is True
|
||||
assert await refusal({}) is False
|
||||
|
||||
|
||||
def test_synthetic_is_forwarded_downstream_only_for_synthetic_requests():
|
||||
token = tmod.SYNTHETIC.set(True)
|
||||
try:
|
||||
assert tmod.synthetic_headers() == {"X-Windy-Synthetic": "1"}
|
||||
finally:
|
||||
tmod.SYNTHETIC.reset(token)
|
||||
assert tmod.synthetic_headers() == {}
|
||||
|
||||
|
||||
# ---- UPDATE 7: the ledger answers 202 even when it quarantines rows ----------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quarantined_rows_are_warned_and_counted_on_the_next_heartbeat(monkeypatch, caplog):
|
||||
tel = _tel()
|
||||
tel.boot()
|
||||
monkeypatch.setattr(
|
||||
tel, "_post", lambda b: (202, {"accepted": 0, "quarantined": 1, "rejections": ["undeclared key"]})
|
||||
)
|
||||
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
|
||||
await tel.flush()
|
||||
assert tel.buffer == [] # sent; the ledger dead-lettered it, retrying won't help
|
||||
assert "QUARANTINED" in caplog.text and "undeclared key" in caplog.text
|
||||
tel.health()
|
||||
assert tel.buffer[-1]["metadata"]["telemetry_quarantined"] == 1
|
||||
assert tel.health_row()["telemetry_quarantined"] == 0 # reset per heartbeat window
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_clean_send_reports_zero_and_logs_nothing(monkeypatch, caplog):
|
||||
tel = _tel()
|
||||
tel.boot()
|
||||
monkeypatch.setattr(tel, "_post", lambda b: (202, {"accepted": 1, "quarantined": 0, "rejections": []}))
|
||||
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
|
||||
await tel.flush()
|
||||
assert caplog.text == ""
|
||||
row = tel.health_row()
|
||||
assert row["telemetry_quarantined"] == 0 and row["telemetry_dropped"] == 0
|
||||
|
||||
|
||||
def test_buffer_overflow_is_counted_as_dropped(monkeypatch):
|
||||
monkeypatch.setattr(tmod, "MAX_BUFFER", 3)
|
||||
tel = _tel()
|
||||
for _ in range(5):
|
||||
tel.boot()
|
||||
assert len(tel.buffer) == 3
|
||||
assert tel.health_row()["telemetry_dropped"] == 2
|
||||
106
api/tests/test_webhooks_behavior.py
Normal file
106
api/tests/test_webhooks_behavior.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""G3.5 — the Eternitas webhook receiver, driven over HTTP (audit 2026-08-13).
|
||||
|
||||
The G3.5 invariants in test_invariants.py grep webhooks.py for strings; a
|
||||
refactor that kept the strings and broke the behaviour would pass them all.
|
||||
These send real requests through the real route (no DB: every case here stops
|
||||
before the revocation handler) and assert what the receiver DOES.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
from api.app.routes import webhooks
|
||||
|
||||
SECRET = "s" * 64
|
||||
URL = "/api/v1/webhooks/eternitas"
|
||||
|
||||
|
||||
def _app(secret: str = SECRET) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.include_router(webhooks.router)
|
||||
app.state.settings = Settings(eternitas_webhook_secret=secret)
|
||||
|
||||
@app.exception_handler(RepairPointer)
|
||||
async def _h(_, exc: RepairPointer) -> JSONResponse:
|
||||
return JSONResponse(status_code=exc.status_code, content=exc.detail)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
async def _post(body: bytes, headers: dict, secret: str = SECRET) -> httpx.Response:
|
||||
transport = httpx.ASGITransport(app=_app(secret))
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
|
||||
return await c.post(
|
||||
URL, content=body, headers={"content-type": "application/json", **headers}
|
||||
)
|
||||
|
||||
|
||||
def _sig(raw: bytes, secret: str = SECRET) -> str:
|
||||
return hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
# Deliberately odd spacing/key order: a receiver that re-serialises before
|
||||
# hashing produces a different digest and must fail.
|
||||
RAW = b'{"event":"windygit.selftest", "data": {"b": 2, "a": 1}}'
|
||||
EVENT = {"x-eternitas-event": "windygit.selftest"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prefixed_and_bare_digests_are_both_accepted():
|
||||
for header in (f"sha256={_sig(RAW)}", _sig(RAW)):
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["acted"] is False # unknown event: received, nothing done
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_digest_of_reserialised_json_is_refused():
|
||||
reserialised = json.dumps(json.loads(RAW)).encode()
|
||||
assert reserialised != RAW
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(reserialised)}"})
|
||||
assert r.status_code == 401 and r.json()["code"] == "webhook_signature_invalid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_forged_or_wrong_key_signature_is_refused():
|
||||
for header in ("sha256=" + "0" * 64, f"sha256={_sig(RAW, 'other-secret')}", "garbage"):
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
|
||||
assert r.status_code == 401, header
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_signed_event_without_signature_is_refused():
|
||||
r = await _post(RAW, EVENT)
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unset_secret_refuses_rather_than_accepts():
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(RAW)}"}, secret="")
|
||||
assert r.status_code == 503 and r.json()["code"] == "webhook_secret_unset"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revocation_with_bad_signature_never_reaches_the_handler():
|
||||
body = b'{"event":"passport.revoked","passport":"ET26-TEST-GOOD"}'
|
||||
r = await _post(
|
||||
body,
|
||||
{"x-eternitas-event": "passport.revoked", "x-eternitas-signature": "sha256=" + "f" * 64},
|
||||
)
|
||||
assert r.status_code == 401 # refused before any DB work
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reachability_ping_acknowledges_but_never_acts():
|
||||
r = await _post(b'{"anything": "at all"}', {"x-eternitas-event": "platform.test_ping"})
|
||||
assert r.status_code == 200 and r.json()["acted"] is False
|
||||
13
ci/ci-hygiene-allow.yml
Normal file
13
ci/ci-hygiene-allow.yml
Normal file
@@ -0,0 +1,13 @@
|
||||
# CI hygiene allow-list: installs that may float, or services that may publish
|
||||
# a host port. House rule 6 (09-23): installs come from a lockfile. Every entry
|
||||
# is an exception and MUST say why. Paths are fnmatch globs from the repo root.
|
||||
# Owner: Windy Git lane (13); changes go through the orchestrator.
|
||||
allow:
|
||||
- repo: windy-pro
|
||||
paths: [".github/workflows/ci.yml"]
|
||||
# ONLY the old deploy job's two docker lines. That job is `if: false`
|
||||
# (CD boundary, 2026-07), and the compose line runs ON windyword.ai inside
|
||||
# the ssh string. Any other docker step in ci.yml still flags.
|
||||
matches: ['docker build -f account-server/Dockerfile -t windy-pro:', 'docker compose down && docker compose up -d --build']
|
||||
reason: "needs-docker false positive: the deploy job is if: false and its compose runs on the remote host over ssh. Added with the needs-docker rule (orchestrator option A, 09-23)."
|
||||
|
||||
52
ci/compute-guard-allow.yml
Normal file
52
ci/compute-guard-allow.yml
Normal file
@@ -0,0 +1,52 @@
|
||||
# Compute guard allow-list: code that MAY talk to an AI provider directly.
|
||||
# Windy Mind is the ONLY door to AI compute (Grant, 2026-09-23). Every entry
|
||||
# here is an exception to that rule and MUST say why. Paths are fnmatch globs
|
||||
# relative to the repo root. Owner of this file: Windy Git lane (13); changes
|
||||
# go through the orchestrator. Source of the first entries: COMPUTE_BYPASS_AUDIT.md.
|
||||
allow:
|
||||
- repo: windy-mind
|
||||
paths: ["*"]
|
||||
reason: "Windy Mind IS the door: provider clients belong here by definition."
|
||||
|
||||
- repo: windy-agent
|
||||
paths: ["*"]
|
||||
reason: >-
|
||||
User BYOK: self-hosted agents call providers on the USER's own keys.
|
||||
Mind stays opt-in there, or every self-hosted user's inference lands on
|
||||
Grant's bill (no-cloud-cost-liability rule; audit #7).
|
||||
|
||||
- repo: windy-code
|
||||
paths: ["extensions/windy-ai/*"]
|
||||
reason: "User BYOK AI extension: the user's own provider keys; Mind is one opt-in provider (audit #8)."
|
||||
|
||||
- repo: windy-connect
|
||||
paths: ["*writers/*"]
|
||||
reason: "Writes client configs that NAME the user's own provider env vars; makes no provider calls (audit #11)."
|
||||
|
||||
- repo: windy-pro
|
||||
paths: ["src/client/desktop/*"]
|
||||
reason: >-
|
||||
User BYOK desktop client: cloud STT/translate keys come from what the USER
|
||||
enters (renderer localStorage -> electron-store; env var only for dev), and
|
||||
the CSP line allows exactly those user-keyed hosts (audit #10). The
|
||||
account-server is NOT covered: server-side calls go through Mind.
|
||||
|
||||
- repo: windy-pro
|
||||
paths: ["src/client/web/src/pages/panels/MindPanel.jsx"]
|
||||
reason: "Validates the USER's own OpenRouter key for BYOK (audit #10); spends no house money."
|
||||
|
||||
- repo: windy-pro
|
||||
paths: ["src/client/web/src/pages/panels/MindKeychain.jsx"]
|
||||
# ONLY these two endpoints: any other openrouter.ai call in this file (e.g.
|
||||
# /api/v1/chat, i.e. inference) still flags. Orchestrator-approved 09-23.
|
||||
matches: ['openrouter\.ai/auth\?', 'openrouter\.ai/api/v1/auth/keys']
|
||||
reason: "BYOK key acquisition via OpenRouter OAuth PKCE; no inference; successor of MindPanel allow (ADR-064)."
|
||||
|
||||
- repo: windy-git
|
||||
paths: ["scripts/compute_guard.py", "ci/compute-guard-allow.yml"]
|
||||
reason: "The guard's own pattern list and this file."
|
||||
|
||||
- repo: windy-mind
|
||||
paths: ["*"]
|
||||
matches: [':11434']
|
||||
reason: "Windy Mind IS the compute door (endpoint + key); it may call Ollama. Only the Ollama port is allowed here, any provider host/SDK in Mind still flags."
|
||||
19
ci/grant-owned.yml
Normal file
19
ci/grant-owned.yml
Normal file
@@ -0,0 +1,19 @@
|
||||
# Code Grant owns directly (orchestrator, 09-23): guard findings here are listed
|
||||
# SEPARATELY in the guards status page and never hold up "block". Changes to
|
||||
# these files are proposals for Grant / Windy Word 44, not a lane's fix.
|
||||
grant_owned:
|
||||
- repo: windy-pro
|
||||
reason: "Windy Word desktop (Electron) + its release/installer builds: Grant's, built from the Mac mini."
|
||||
paths:
|
||||
- "src/client/desktop/*"
|
||||
# ROOT .env.example only (exact path): documents the desktop app's BYOK dev
|
||||
# fallback keys; the hub reads neither (8c, 09-23). account-server/.env.example
|
||||
# is NOT matched and stays the hub lane's.
|
||||
- ".env.example"
|
||||
- "installer-v2/*"
|
||||
- ".github/workflows/build-windows.yml"
|
||||
- ".github/workflows/release-mac.yml"
|
||||
- ".github/workflows/build-installer.yml"
|
||||
- ".github/workflows/build-offline-installers.yml"
|
||||
jobs:
|
||||
".github/workflows/ci.yml": [reality-check, build-desktop, test-installer, build-electron]
|
||||
62
ci/secret-guard-allow.yml
Normal file
62
ci/secret-guard-allow.yml
Normal file
@@ -0,0 +1,62 @@
|
||||
# Secret guard allow-list: KNOWN FAKE values that look like secrets (test
|
||||
# fixtures, docs). Allowed BY HASH (sha256[:8] of the value), so a real secret
|
||||
# in the same file still flags. Private-key blocks (the match is only the BEGIN
|
||||
# line, same hash everywhere) are allowed by PATH + kind instead.
|
||||
# Every entry MUST say why. Owner: Windy Git lane (13); changes via the orchestrator.
|
||||
# Triage 09-24 (values never printed): each hash checked against every version of
|
||||
# the lockbox; fakes judged by impossible length for the kind (real Anthropic keys
|
||||
# ~108 chars, OpenAI 51 or 160+), fake-words, or identity with upstream public
|
||||
# fixtures. NOT allowed, remove instead: 1354fc9b (old @Windy_0_bot token) and
|
||||
# d49dc2ba (old Anthropic key), both real and revoked, in public windy-agent.
|
||||
allow:
|
||||
- repo: windy-code
|
||||
hashes: [ac9265e5, 46eb1235]
|
||||
reason: "Upstream microsoft/vscode terminalEnvironment.test.ts fixtures (identical hash upstream; public)."
|
||||
- repo: windy-code
|
||||
paths: ["build/azure-pipelines/common/publish.ts"]
|
||||
kinds: [private key block]
|
||||
reason: "Upstream VS Code build script (PEM header string in code, not a key)."
|
||||
- repo: windy-agent
|
||||
hashes: [a9235a6d, dd6a2baa, 02c362d8, a6f6ff79, f7503b21, d0c94833, 4ab092e3, e3aa1eb8, 833382ee]
|
||||
reason: "Redaction/sanitizer test fixtures; lengths impossible for real Anthropic/OpenAI keys; never in the lockbox."
|
||||
- repo: windy-agent
|
||||
hashes: [89bd408f, b8c94b72, bf23cdf5, d1d85dfe, e3e07f06]
|
||||
reason: "09-24 #395 replacement fixtures (each contains FAKE; token-SHAPED on purpose so redaction tests prove real tokens are scrubbed); verified by Windy Agent sha-for-sha against every lockbox version: never real. Weekly scan 09-28."
|
||||
- repo: windy-agent
|
||||
paths: ["tests/test_agent_keys.py"]
|
||||
kinds: [private key block]
|
||||
reason: "Test-generated key material for agent-key tests."
|
||||
- repo: windy-mind
|
||||
hashes: [f8a630b2]
|
||||
reason: "Provider test fixture (27 chars; a real Anthropic key is ~108)."
|
||||
- repo: windy-pro
|
||||
hashes: [756de8d8, 7828319d, 1a5d44a2]
|
||||
reason: ".env.production.example placeholder + crash-summary test fixtures (AWS doc EXAMPLE key shape, short fake Slack token)."
|
||||
- repo: windy-pro
|
||||
paths: ["account-server/docs/oauth-providers.md"]
|
||||
kinds: [private key block]
|
||||
reason: "Docs show the PEM header format; no key material."
|
||||
- repo: windytalk
|
||||
hashes: [baf8656a]
|
||||
reason: "Diagnostics redaction test fixture (fake-word in value)."
|
||||
- repo: eternitas
|
||||
paths: ["tests/golden_vectors/**", "tests/test_soul_vault_key_separation.py"]
|
||||
kinds: [private key block]
|
||||
reason: "Test vectors and throwaway keys for signature/vault tests."
|
||||
- repo: windy-drops
|
||||
paths: ["tools/conformance/test-keys/*"]
|
||||
kinds: [private key block]
|
||||
reason: "Conformance-suite test keys (named test-private.pem)."
|
||||
- repo: windy-git
|
||||
paths: ["api/tests/test_secret_guard.py"]
|
||||
kinds: [private key block]
|
||||
reason: "The guard's own test uses a PEM header string as a sample."
|
||||
- repo: windy-code
|
||||
hashes: ["23f32607"]
|
||||
reason: "VS Code OSS extensions' package.json aiKey: Microsoft's public telemetry (App Insights) key, shipped in every VS Code build; not a Windy credential."
|
||||
- repo: windytalk
|
||||
hashes: ["c4189d79"]
|
||||
reason: "apps/desktop/test/diagnostics.test.ts redaction fixture (hexSecret beside a fake sk-ant token); hash checked against the lockbox 10-01: not present."
|
||||
- repo: windy-agent
|
||||
hashes: ["84e0c0ea"]
|
||||
reason: "tests/test_log_redaction.py:56 Z.ai redaction fixture REPLACED by windy-agent #412 with a synthetic value; hash checked against the lockbox 10-01: not present."
|
||||
28
deploy/branding/README.md
Normal file
28
deploy/branding/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Windy Git branding (G2.3)
|
||||
|
||||
Gitea's **supported** customisation surface: custom templates and public assets.
|
||||
No Gitea source is modified, so upstream upgrades keep arriving (D-2, I-1).
|
||||
|
||||
deploy/branding/ → $GITEA_CUSTOM (/data/gitea) in the container
|
||||
→ /srv/windygit/git/gitea/ on Veron 1
|
||||
|
||||
Apply with `./deploy/branding/apply.sh`.
|
||||
|
||||
## Two traps this cost, both worth knowing
|
||||
|
||||
**1. `GITEA__DEFAULT__APP_NAME` does not work.** Gitea reads `APP_NAME` from the
|
||||
*top level* of `app.ini` (before any `[section]`). The env var instead created a
|
||||
literal `[default]` section, which Gitea ignores — and the installer's stock
|
||||
`APP_NAME` kept winning, so the site said "Gitea: Git with a cup of tea" while
|
||||
the config looked correct. Worse, the env-to-ini pass **appended** a second
|
||||
`APP_NAME` rather than replacing the first. `apply.sh` sets it at the top level
|
||||
directly.
|
||||
|
||||
**2. Cloudflare caches `/assets/*` for 6 hours and no token in this stack can
|
||||
purge.** Editing a fixed filename leaves the old bytes live for hours — the new
|
||||
logo and CSS were both invisible while being correct at origin. **Version the
|
||||
filename** (`theme-windy.v2.css`) on every brand change; a `?query` is not
|
||||
enough because some caches ignore it.
|
||||
|
||||
The nav logo is swapped in CSS rather than by overriding Gitea's navbar
|
||||
template — a one-line rule instead of a forked template that would drift.
|
||||
23
deploy/branding/apply.sh
Executable file
23
deploy/branding/apply.sh
Executable file
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Apply Windy Git branding to the Gitea custom tree. Idempotent.
|
||||
set -euo pipefail
|
||||
CUSTOM="${GITEA_CUSTOM_HOST:-/srv/windygit/git/gitea}"
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
sudo mkdir -p "$CUSTOM"/templates/custom "$CUSTOM"/public/assets/img "$CUSTOM"/public/assets/css
|
||||
sudo cp -r "$HERE"/templates/. "$CUSTOM"/templates/
|
||||
sudo cp -r "$HERE"/public/. "$CUSTOM"/public/
|
||||
sudo chown -R 1000:1000 "$CUSTOM"/templates "$CUSTOM"/public
|
||||
|
||||
# APP_NAME must sit at the TOP LEVEL. See README trap #1.
|
||||
INI="$CUSTOM/conf/app.ini"
|
||||
sudo cp "$INI" "$INI.bak-brand-$(date +%s)"
|
||||
sudo python3 - "$INI" <<'PY'
|
||||
import sys
|
||||
p = sys.argv[1]
|
||||
lines = [l for l in open(p).read().splitlines()
|
||||
if not l.strip().startswith(("APP_NAME", "APP_SLOGAN"))]
|
||||
lines.insert(0, "APP_NAME = Windy Git")
|
||||
open(p, "w").write("\n".join(lines) + "\n")
|
||||
PY
|
||||
echo "applied. restart gitea to pick it up."
|
||||
27
deploy/branding/public/assets/css/theme-windy.v2.css
Normal file
27
deploy/branding/public/assets/css/theme-windy.v2.css
Normal file
@@ -0,0 +1,27 @@
|
||||
/* Windy Git brand accent. Layered on top of Gitea's theme rather than
|
||||
replacing it, so upstream theme fixes keep arriving. */
|
||||
:root {
|
||||
--color-primary: #0ea5e9;
|
||||
--color-primary-dark-1: #0284c7;
|
||||
--color-primary-dark-2: #0369a1;
|
||||
--color-primary-light-1: #38bdf8;
|
||||
--color-primary-light-2: #7dd3fc;
|
||||
}
|
||||
.wg-hero { max-width: 780px; margin: 4rem auto 2rem; padding: 0 1.5rem; text-align: center; }
|
||||
.wg-hero h1 { font-size: 2.6rem; margin: 1.2rem 0 .4rem; letter-spacing: -.02em; }
|
||||
.wg-hero .wg-sub { font-size: 1.15rem; opacity: .78; margin-bottom: 2.2rem; }
|
||||
.wg-grid { display: grid; gap: 1.1rem; grid-template-columns: repeat(auto-fit,minmax(230px,1fr));
|
||||
max-width: 900px; margin: 0 auto 2.5rem; padding: 0 1.5rem; text-align: left; }
|
||||
.wg-card { border: 1px solid var(--color-secondary); border-radius: 8px; padding: 1.1rem 1.2rem; }
|
||||
.wg-card h3 { margin: 0 0 .35rem; font-size: 1.02rem; }
|
||||
.wg-card p { margin: 0; opacity: .74; font-size: .9rem; line-height: 1.5; }
|
||||
.wg-cta { margin-bottom: 3rem; }
|
||||
|
||||
/* Logo swap via CSS.
|
||||
Cloudflare cached the stock /assets/img/logo.svg for 6h and no available API
|
||||
token can purge. Pointing at a NEW filename sidesteps the stale object
|
||||
without touching Gitea's own templates — the supported customisation surface,
|
||||
per D-2 (membrane, not merge). */
|
||||
img[src$="/assets/img/logo.svg"] {
|
||||
content: url("/assets/img/wg-mark.svg");
|
||||
}
|
||||
9
deploy/branding/public/assets/img/logo.svg
Normal file
9
deploy/branding/public/assets/img/logo.svg
Normal file
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" width="32" height="32">
|
||||
<defs><linearGradient id="w" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0%" stop-color="#38bdf8"/><stop offset="100%" stop-color="#0ea5e9"/>
|
||||
</linearGradient></defs>
|
||||
<circle cx="16" cy="16" r="15" fill="#0b1220"/>
|
||||
<path d="M5 11h13a3.2 3.2 0 1 0-3.1-4" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
<path d="M5 16h17a3.6 3.6 0 1 1-3.5 4.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
<path d="M5 21h9a2.8 2.8 0 1 1-2.7 3.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 660 B |
9
deploy/branding/public/assets/img/wg-mark.svg
Normal file
9
deploy/branding/public/assets/img/wg-mark.svg
Normal file
@@ -0,0 +1,9 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" width="32" height="32">
|
||||
<defs><linearGradient id="w" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0%" stop-color="#38bdf8"/><stop offset="100%" stop-color="#0ea5e9"/>
|
||||
</linearGradient></defs>
|
||||
<circle cx="16" cy="16" r="15" fill="#0b1220"/>
|
||||
<path d="M5 11h13a3.2 3.2 0 1 0-3.1-4" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
<path d="M5 16h17a3.6 3.6 0 1 1-3.5 4.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
<path d="M5 21h9a2.8 2.8 0 1 1-2.7 3.5" fill="none" stroke="url(#w)" stroke-width="2.4" stroke-linecap="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 660 B |
5
deploy/branding/templates/custom/header.tmpl
Normal file
5
deploy/branding/templates/custom/header.tmpl
Normal file
@@ -0,0 +1,5 @@
|
||||
{{/* Windy Git brand layer.
|
||||
The filename carries a version: Cloudflare caches /assets/* for 6h with no
|
||||
purge token available to this stack, so editing a fixed filename leaves the
|
||||
old bytes live for hours. Bump the suffix on every brand change. */}}
|
||||
<link rel="stylesheet" href="{{AssetUrlPrefix}}/css/theme-windy.v2.css">
|
||||
28
deploy/branding/templates/home.tmpl
Normal file
28
deploy/branding/templates/home.tmpl
Normal file
@@ -0,0 +1,28 @@
|
||||
{{template "base/head" .}}
|
||||
<div role="main" aria-label="{{ctx.Locale.Tr "home"}}" class="page-content home">
|
||||
<div class="wg-hero">
|
||||
<img src="{{AssetUrlPrefix}}/img/logo.svg" alt="Windy Git" width="72" height="72">
|
||||
<h1>Windy Git</h1>
|
||||
<p class="wg-sub">Your work, every version — and agents as citizens.</p>
|
||||
<div class="wg-cta">
|
||||
{{if not .IsSigned}}
|
||||
<a class="ui primary button" href="{{AppSubUrl}}/user/login">Sign in with Windy</a>
|
||||
<p class="wg-note" style="margin-top:.9rem;font-size:.95rem;opacity:.75">Invite only at launch: new accounts are not open yet.</p>
|
||||
{{else}}
|
||||
<a class="ui primary button" href="{{AppSubUrl}}/{{.SignedUser.Name}}?tab=repositories">Your repositories</a>
|
||||
{{end}}
|
||||
<a class="ui button" href="{{AppSubUrl}}/explore/repos">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="wg-grid">
|
||||
<div class="wg-card"><h3>Code and models, one host</h3>
|
||||
<p>Git and LFS over Windy Cloud storage. Source, weights and adapters live side by side.</p></div>
|
||||
<div class="wg-card"><h3>Agents are first-class</h3>
|
||||
<p>An agent signs in with its own Eternitas passport — not a human's borrowed token — and its work is attributable to it.</p></div>
|
||||
<div class="wg-card"><h3>Invite only, for now</h3>
|
||||
<p>Windy Git accounts are by invitation while we launch. If you've been invited, you sign in with your Windy account, with no second password.</p></div>
|
||||
<div class="wg-card"><h3>Kept, and kept elsewhere</h3>
|
||||
<p>Every repository is bundled nightly to off-site storage, and the restore is rehearsed rather than assumed.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
{{template "base/footer" .}}
|
||||
61
deploy/branding/templates/user/auth/link_account.tmpl
Normal file
61
deploy/branding/templates/user/auth/link_account.tmpl
Normal file
@@ -0,0 +1,61 @@
|
||||
{{/* Windy Git override of Gitea 1.24.6 templates/user/auth/link_account.tmpl.
|
||||
Forge registration is CLOSED at launch (Grant, 09-23), so a Windy account
|
||||
with no forge account used to land on Gitea's raw "Registration is disabled"
|
||||
error. That case now gets a plain invite-only page; every other case is
|
||||
Gitea's template unchanged (re-diff it on Gitea upgrades). No change to who
|
||||
can register. */}}
|
||||
{{if and .DisableRegistration (not .user_exists)}}
|
||||
{{template "base/head" .}}
|
||||
<div role="main" aria-label="Windy Git is invite-only" class="page-content user link-account">
|
||||
<div class="ui middle very relaxed page grid">
|
||||
<div class="column tw-my-5">
|
||||
<div class="tw-flex tw-flex-col tw-gap-4 tw-max-w-2xl tw-m-auto">
|
||||
<h4 class="ui top attached header center">Windy Git is invite-only while we launch</h4>
|
||||
<div class="ui attached segment">
|
||||
<p>You're signed in with your Windy account{{if or .user_name .email}} as <strong>{{or .user_name .email}}</strong>{{end}}. Nothing is wrong with it: Windy Git just isn't open to every account yet.</p>
|
||||
<p>We'll let you know when it opens.</p>
|
||||
<a class="ui primary button" href="https://app.windyword.ai/dashboard">Back to your dashboard</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{{template "base/footer" .}}
|
||||
{{else}}
|
||||
{{template "base/head" .}}
|
||||
<div role="main" aria-label="{{.Title}}" class="page-content user link-account">
|
||||
<overflow-menu class="ui secondary pointing tabular top attached borderless menu secondary-nav">
|
||||
<div class="overflow-menu-items tw-justify-center">
|
||||
<!-- TODO handle .ShowRegistrationButton once other login bugs are fixed -->
|
||||
{{if not .AllowOnlyInternalRegistration}}
|
||||
<a class="item {{if not .user_exists}}active{{end}}"
|
||||
data-tab="auth-link-signup-tab">
|
||||
{{ctx.Locale.Tr "auth.oauth_signup_tab"}}
|
||||
</a>
|
||||
{{end}}
|
||||
<a class="item {{if .user_exists}}active{{end}}"
|
||||
data-tab="auth-link-signin-tab">
|
||||
{{ctx.Locale.Tr "auth.oauth_signin_tab"}}
|
||||
</a>
|
||||
</div>
|
||||
</overflow-menu>
|
||||
<div class="ui middle very relaxed page grid">
|
||||
<div class="column tw-my-5">
|
||||
{{/* these styles are quite tricky but it needs to be the same as the signin page */}}
|
||||
<div class="ui tab {{if not .user_exists}}active{{end}}" data-tab="auth-link-signup-tab">
|
||||
<div class="tw-flex tw-flex-col tw-gap-4 tw-max-w-2xl tw-m-auto">
|
||||
{{if .AutoRegistrationFailedPrompt}}<div class="ui message">{{.AutoRegistrationFailedPrompt}}</div>{{end}}
|
||||
{{template "user/auth/signup_inner" .}}
|
||||
</div>
|
||||
</div>
|
||||
<div class="ui tab {{if .user_exists}}active{{end}}" data-tab="auth-link-signin-tab">
|
||||
<div class="tw-flex tw-flex-col tw-gap-4 tw-max-w-2xl tw-m-auto">
|
||||
{{template "user/auth/signin_inner" .}}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{template "base/footer" .}}
|
||||
{{end}}
|
||||
@@ -11,8 +11,9 @@ log:
|
||||
|
||||
runner:
|
||||
file: /data/.runner
|
||||
capacity: 4 # concurrent jobs; Veron has 24 cores, dind is capped at 12
|
||||
timeout: 30m
|
||||
capacity: 1 # per runner; parallelism = number of runner services (6). See docker-compose.yml
|
||||
timeout: 90m # hard ceiling per job. eternitas's serial pytest is ~50 min; keep
|
||||
# timeout-minutes in each workflow — a hang still reads as a hang
|
||||
shutdown_timeout: 3m
|
||||
insecure: false
|
||||
fetch_timeout: 5s
|
||||
@@ -53,6 +54,9 @@ container:
|
||||
privileged: false
|
||||
options:
|
||||
workdir_parent: /workspace
|
||||
valid_volumes: [] # a job cannot bind-mount anything from the daemon host
|
||||
# A job may bind-mount exactly ONE daemon path: the read-only windy-pro build
|
||||
# inputs (mounted :ro into dind itself). Per-runner, not per-repo (act_runner
|
||||
# limit): any windyadmin repo could mount it; it is non-secret and read-only.
|
||||
valid_volumes: ["/ci-inputs/windy-pro"]
|
||||
docker_host: "-" # do NOT expose the runner's own docker socket to jobs
|
||||
force_pull: false
|
||||
|
||||
15
deploy/runner/docker-compose.privileged.yml
Normal file
15
deploy/runner/docker-compose.privileged.yml
Normal file
@@ -0,0 +1,15 @@
|
||||
# ROLLBACK ONLY: the pre-Sysbox dind (privileged: true), kept one command away.
|
||||
# Use it if CI breaks under Sysbox:
|
||||
#
|
||||
# cd /srv/windygit/src/deploy/runner
|
||||
# sudo docker compose -f docker-compose.yml -f docker-compose.privileged.yml up -d dind
|
||||
#
|
||||
# (then restart the runners while idle). Compose merges `volumes` by container
|
||||
# path, so this puts back the old `dind-storage` volume with its image cache.
|
||||
# Going forward again: the same command without the second -f.
|
||||
services:
|
||||
dind:
|
||||
runtime: runc
|
||||
privileged: true
|
||||
volumes:
|
||||
- dind-storage:/var/lib/docker
|
||||
@@ -26,8 +26,12 @@
|
||||
# * `dind` and every job container it spawns are UNTRUSTED. They are on a
|
||||
# private network with no access to the forge, its database, or its .env.
|
||||
#
|
||||
# dind itself is privileged — that is the cost, and it is the reason a job
|
||||
# escape lands in a disposable daemon rather than on Grant's workstation.
|
||||
# dind is NOT privileged (2026-09-23): it runs under the Sysbox runtime
|
||||
# (sysbox-ce on Veron, `runtime: sysbox-runc`), a user-namespaced system
|
||||
# container whose root is an unprivileged host uid. A job that escapes its own
|
||||
# container lands in dind as a nobody on the host, not as root on Grant's
|
||||
# workstation. Before Sysbox, dind was `privileged: true`; that config is kept
|
||||
# as docker-compose.privileged.yml (ROLLBACK ONLY, one command, see that file).
|
||||
#
|
||||
# ⚠️ Do NOT "simplify" this by mounting the host docker socket.
|
||||
|
||||
@@ -36,21 +40,49 @@ name: windy-git-runner
|
||||
services:
|
||||
dind:
|
||||
image: docker.io/library/docker:27-dind
|
||||
privileged: true
|
||||
runtime: sysbox-runc # NOT privileged: see the I-5 note above
|
||||
environment:
|
||||
DOCKER_TLS_CERTDIR: "" # plain TCP on an isolated network, no host route
|
||||
command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--tls=false"]
|
||||
networks: [jobs]
|
||||
volumes:
|
||||
- dind-storage:/var/lib/docker
|
||||
# A fresh volume: Sysbox shifts ownership to its own uid range. The old
|
||||
# `dind-storage` is kept untouched for the privileged rollback.
|
||||
- dind-storage-sysbox:/var/lib/docker
|
||||
# READ-ONLY, non-secret build inputs for windy-pro's desktop jobs (models,
|
||||
# linux-x64 portable bundle, enter-monitor build), copied from the frozen
|
||||
# release clone by deploy/runner/refresh-ci-inputs.sh. Jobs may mount ONLY
|
||||
# this path (config.yaml valid_volumes). Orchestrator-approved 09-23.
|
||||
- /home/user1-gpu/ci-inputs/windy-pro:/ci-inputs/windy-pro:ro
|
||||
# G1.5 — bounded so a fork-bomb workflow cannot starve Grant's interactive
|
||||
# session. Veron 1 is his workstation, not a dedicated build box.
|
||||
cpus: 12.0 # 12 of 24 cores
|
||||
mem_limit: 64g
|
||||
restart: unless-stopped
|
||||
|
||||
runner:
|
||||
image: docker.io/gitea/act_runner:0.2.11
|
||||
# ── FOUR runners × capacity 1, not one runner × capacity 4 (2026-09-23) ──
|
||||
#
|
||||
# act caches every action repo at /root/.cache/act/<hash> INSIDE the runner
|
||||
# process and re-fetches it at the start of each job. With capacity 4, four
|
||||
# concurrent jobs share that one directory: one job's refresh rewrites it while
|
||||
# another is tarring it into its job container, and the job dies with
|
||||
# `lstat /root/.cache/act/<hash>/…: no such file or directory` on
|
||||
# `actions/setup-node` / `setup-uv` — a failure that reads like a broken
|
||||
# workflow. windy-chat (~20 jobs per push) hit it on 3 jobs in its first run.
|
||||
# `rm -rf /root/.cache/act` only reset the clock. Separate processes get
|
||||
# separate caches, so the race cannot occur. Same total parallelism, same
|
||||
# single capped dind — the blast radius is unchanged.
|
||||
runner: &runner
|
||||
# 0.2.11 -> 0.6.1 on 2026-08-14. The bundled act in 0.2.11 only knows
|
||||
# `runs.using: node12|node16|node20`, so ANY repo pinning a current action
|
||||
# major dies before its first step with "The runs.using key in action.yml
|
||||
# must be one of: [...], got node24" — Windy-Clone on actions/checkout@v5
|
||||
# is how this surfaced. Verified: `node24` is absent from the 0.2.11 binary
|
||||
# and present in 0.6.1. Every key in this directory's config.yaml still
|
||||
# exists in 0.6.1's schema (0.6.1 only ADDS keys), so the config carries
|
||||
# over unchanged. Rollback is re-pinning 0.2.11; the registration in the
|
||||
# runner-data volume survives either way.
|
||||
image: docker.io/gitea/act_runner:0.6.1
|
||||
depends_on: [dind]
|
||||
environment:
|
||||
# The runner reaches its OWN daemon. Never the host's.
|
||||
@@ -90,6 +122,47 @@ services:
|
||||
mem_limit: 4g
|
||||
restart: unless-stopped
|
||||
|
||||
# Each extra runner registers itself on first start (own name, own volume —
|
||||
# the registration lives in /data/.runner, so volumes must never be shared).
|
||||
runner-2:
|
||||
<<: *runner
|
||||
environment: &env2
|
||||
DOCKER_HOST: tcp://dind:2375
|
||||
GITEA_INSTANCE_URL: https://app.windygit.com
|
||||
GITEA_RUNNER_REGISTRATION_TOKEN: ${RUNNER_TOKEN:?set RUNNER_TOKEN}
|
||||
GITEA_RUNNER_NAME: veron-1-2
|
||||
CONFIG_FILE: /config.yaml
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-2:/data]
|
||||
runner-3:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-3
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-3:/data]
|
||||
runner-4:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-4
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-4:/data]
|
||||
# 5 and 6 added the same day: with ~11 private repos onboarded (windy-chat
|
||||
# alone queues ~24 jobs per push) four runners left 50+ jobs waiting. The
|
||||
# CPU ceiling is dind's (12 of 24 cores, G1.5), not the runner count, so more
|
||||
# runners add concurrency for I/O-bound jobs (npm ci, uv sync) without
|
||||
# taking more of Grant's workstation.
|
||||
runner-5:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-5
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-5:/data]
|
||||
runner-6:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-6
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-6:/data]
|
||||
|
||||
networks:
|
||||
jobs:
|
||||
# Untrusted job containers live here. No route to the forge.
|
||||
@@ -97,5 +170,11 @@ networks:
|
||||
|
||||
volumes:
|
||||
dind-storage:
|
||||
dind-storage-sysbox:
|
||||
runner-data:
|
||||
runner-data-2:
|
||||
runner-data-3:
|
||||
runner-data-4:
|
||||
runner-data-5:
|
||||
runner-data-6:
|
||||
|
||||
|
||||
51
deploy/runner/egress.sh
Executable file
51
deploy/runner/egress.sh
Executable file
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI egress filter (2026-09-23) — jobs reach the internet, never Grant's network.
|
||||
#
|
||||
# Measured before this existed: an ordinary (unprivileged) job container inside
|
||||
# the CI dind could open SSH, Ollama, and every dev server on Veron
|
||||
# (192.168.1.73:22/3000/3300/8080/11434) and anything else on the LAN, WireGuard
|
||||
# or Tailscale. No container escape needed — a malicious npm/pip dependency in
|
||||
# any first-party repo's CI could walk straight onto the fleet.
|
||||
#
|
||||
# All CI traffic leaves through the `windy-git-runner_jobs` bridge (dind NATs
|
||||
# its job containers onto it). This script, run at boot and after any runner
|
||||
# compose change, allows on that bridge:
|
||||
# * traffic between the runners and dind (same bridge)
|
||||
# * replies (ESTABLISHED/RELATED)
|
||||
# * DNS (53) — Docker's embedded resolver forwards to the LAN router
|
||||
# * everything public
|
||||
# and drops: RFC1918, CGNAT/Tailscale (100.64/10), link-local, and ANY packet
|
||||
# addressed to the host itself (INPUT), whatever interface IP it targets.
|
||||
# Idempotent: owned chains are flushed and rebuilt; hooks are added once.
|
||||
set -euo pipefail
|
||||
|
||||
NET=windy-git-runner_jobs
|
||||
id=$(docker network inspect "$NET" --format '{{.Id}}')
|
||||
BR="br-${id:0:12}"
|
||||
ip link show "$BR" >/dev/null
|
||||
|
||||
iptables -N WG-CI-EGRESS 2>/dev/null || iptables -F WG-CI-EGRESS
|
||||
iptables -A WG-CI-EGRESS -o "$BR" -j RETURN
|
||||
iptables -A WG-CI-EGRESS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A WG-CI-EGRESS -p udp --dport 53 -j RETURN
|
||||
iptables -A WG-CI-EGRESS -p tcp --dport 53 -j RETURN
|
||||
for cidr in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16; do
|
||||
iptables -A WG-CI-EGRESS -d "$cidr" -j DROP
|
||||
done
|
||||
iptables -A WG-CI-EGRESS -j RETURN
|
||||
|
||||
iptables -N WG-CI-INPUT 2>/dev/null || iptables -F WG-CI-INPUT
|
||||
iptables -A WG-CI-INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A WG-CI-INPUT -j DROP
|
||||
|
||||
# Hooks: remove any stale ones (the bridge name changes if the network is
|
||||
# recreated), then add exactly one of each at the top.
|
||||
for chain in DOCKER-USER INPUT; do
|
||||
target=$([ "$chain" = INPUT ] && echo WG-CI-INPUT || echo WG-CI-EGRESS)
|
||||
while read -r rule; do
|
||||
iptables -D $chain ${rule#-A $chain }
|
||||
done < <(iptables -S "$chain" | grep -- "-j $target" || true)
|
||||
iptables -I "$chain" 1 -i "$BR" -j "$target"
|
||||
done
|
||||
|
||||
echo "ci egress filter active on $BR ($NET)"
|
||||
28
deploy/runner/prune.sh
Executable file
28
deploy/runner/prune.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Keep CI storage bounded (2026-09-23).
|
||||
#
|
||||
# Kit 0's 09-01 wipe began with CI `_work` dirs (74 GB) + Docker filling the
|
||||
# disk. Windy Git's runners have no host `_work` dir — every job runs in a
|
||||
# container inside the CI-only dind — so the thing that grows here is dind's
|
||||
# image/volume store (38 GB when this was written, never pruned). This prunes
|
||||
# ONLY that daemon, over its own socket. It never touches the host's Docker.
|
||||
#
|
||||
# In-use images/volumes are never removed, so a running job is safe.
|
||||
set -euo pipefail
|
||||
CAP_GB="${CI_STORAGE_CAP_GB:-60}"
|
||||
D=(docker exec windy-git-runner-dind-1 docker -H tcp://127.0.0.1:2375) # dind listens on TCP only
|
||||
|
||||
"${D[@]}" container prune -f --filter until=6h >/dev/null
|
||||
"${D[@]}" volume prune -af >/dev/null # job workspaces of finished jobs
|
||||
"${D[@]}" image prune -af --filter until=168h >/dev/null
|
||||
"${D[@]}" builder prune -af --filter until=168h >/dev/null 2>&1 || true
|
||||
|
||||
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
|
||||
if (( used_gb > CAP_GB )); then
|
||||
# Over the cap even after the age-based pass: drop every unused image. The
|
||||
# next jobs re-pull (the act image is ~2 GB) — slower, never wrong.
|
||||
"${D[@]}" image prune -af >/dev/null
|
||||
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
|
||||
fi
|
||||
echo "ci storage ${used_gb}G (cap ${CAP_GB}G)"
|
||||
(( used_gb <= CAP_GB )) || { echo "STILL OVER CAP"; exit 1; }
|
||||
21
deploy/runner/refresh-ci-inputs.sh
Executable file
21
deploy/runner/refresh-ci-inputs.sh
Executable file
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env bash
|
||||
# Refresh the READ-ONLY CI input cache for windy-pro desktop jobs from the FROZEN
|
||||
# release clone on Veron. Reads ~/windy-pro-release only; never writes to it.
|
||||
# Run when the release lane says engines / wheels / the portable bundle changed.
|
||||
# Linux inputs only: 3 models + requirements-bundle.txt, bundled-portable/linux-x64,
|
||||
# native/enter-monitor/build. Result is chmod a-w and mounted :ro into dind.
|
||||
set -euo pipefail
|
||||
SRC=/home/user1-gpu/windy-pro-release
|
||||
DST=/home/user1-gpu/ci-inputs/windy-pro
|
||||
models=$(ls "$SRC/extraResources/model" | grep -E '^windy-(nano|lite|core)-ct2$')
|
||||
[ "$(wc -w <<<"$models")" = 3 ] || { echo "expected 3 models, got: $models"; exit 1; }
|
||||
mkdir -p "$DST/extraResources/model" "$DST/bundled-portable" "$DST/native-enter-monitor-build"
|
||||
chmod -R u+w "$DST"
|
||||
R="ionice -c3 nice -n 19 rsync -a --delete"
|
||||
for m in $models; do $R "$SRC/extraResources/model/$m/" "$DST/extraResources/model/$m/"; done
|
||||
$R "$SRC/extraResources/requirements-bundle.txt" "$DST/extraResources/requirements-bundle.txt"
|
||||
$R "$SRC/bundled-portable/linux-x64/" "$DST/bundled-portable/linux-x64/"
|
||||
$R "$SRC/native/enter-monitor/build/" "$DST/native-enter-monitor-build/"
|
||||
date -u +%FT%TZ > "$DST/.refreshed-from-windy-pro-release"
|
||||
chmod -R a-w "$DST"
|
||||
du -sh --apparent-size "$DST"
|
||||
13
deploy/runner/windygit-ci-egress.service
Normal file
13
deploy/runner/windygit-ci-egress.service
Normal file
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Windy Git - CI egress filter (jobs reach the internet, never the LAN/host)
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
# The jobs network exists once the runner compose project is up; retry until it does.
|
||||
ExecStart=/bin/bash -c 'for i in $(seq 1 60); do /srv/windygit/src/deploy/runner/egress.sh && exit 0; sleep 5; done; exit 1'
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
6
deploy/runner/windygit-ci-prune.service
Normal file
6
deploy/runner/windygit-ci-prune.service
Normal file
@@ -0,0 +1,6 @@
|
||||
[Unit]
|
||||
Description=Windy Git - prune CI-only dind storage (bounded, never the host daemon)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/srv/windygit/src/deploy/runner/prune.sh
|
||||
9
deploy/runner/windygit-ci-prune.timer
Normal file
9
deploy/runner/windygit-ci-prune.timer
Normal file
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Windy Git - prune CI storage every 6 hours
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 00/6:37:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
13
deploy/systemd/windygit-backup.service
Normal file
13
deploy/systemd/windygit-backup.service
Normal file
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Windy Git nightly backup (git bundles + windgit schema -> R2)
|
||||
After=network-online.target docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/windygit/src
|
||||
# The .env holds the R2 credentials. The script refuses to run without them
|
||||
# rather than reporting a backup that did not happen.
|
||||
EnvironmentFile=/srv/windygit/src/.env
|
||||
ExecStart=/bin/bash /srv/windygit/src/scripts/backup.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
3
deploy/systemd/windygit-backup.service.d/windy-job.conf
Normal file
3
deploy/systemd/windygit-backup.service.d/windy-job.conf
Normal file
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-backup 26h --expect "ok — [0-9]+ repos" --owner 13 -- /bin/bash /srv/windygit/src/scripts/backup.sh
|
||||
12
deploy/systemd/windygit-backup.timer
Normal file
12
deploy/systemd/windygit-backup.timer
Normal file
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Nightly Windy Git backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 04:17:00
|
||||
# Grant's workstation is not always on at 04:17. Without this a missed window
|
||||
# is simply skipped and the backup silently never runs.
|
||||
Persistent=true
|
||||
RandomizedDelaySec=600
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-ci-prune 7h --expect "ci storage [0-9]+G" --owner 13 -- /srv/windygit/src/deploy/runner/prune.sh
|
||||
14
deploy/systemd/windygit-state-backup.service
Normal file
14
deploy/systemd/windygit-state-backup.service
Normal file
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=Windy Git nightly STATE backup (Postgres + Gitea config + repos -> encrypted restic in R2)
|
||||
After=network-online.target docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/windygit/src
|
||||
# R2 credentials come from the .env; the restic password from /etc/windygit/restic.pass
|
||||
# (root 600; the same value lives in the lockbox as RESTIC_WINDYGIT_PASSWORD).
|
||||
EnvironmentFile=/srv/windygit/src/.env
|
||||
ExecStart=/bin/bash /srv/windygit/src/scripts/backup_state.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=3h
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-state-backup 26h --expect "ok — state backed up" --owner 13 -- /bin/bash /srv/windygit/src/scripts/backup_state.sh
|
||||
11
deploy/systemd/windygit-state-backup.timer
Normal file
11
deploy/systemd/windygit-state-backup.timer
Normal file
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Nightly Windy Git state backup
|
||||
|
||||
[Timer]
|
||||
# 03:07 local, clear of the git-bundle backup at 04:17.
|
||||
OnCalendar=*-*-* 03:07:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
12
deploy/systemd/windygit-sync.service
Normal file
12
deploy/systemd/windygit-sync.service
Normal file
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Sync GitHub -> Windy Git (Phase 1: GitHub is the source of truth)
|
||||
After=network-online.target docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/windygit/src
|
||||
EnvironmentFile=/srv/windygit/src/.env
|
||||
# GITHUB_TOKEN is set on the host only (root-only unit file / .env) — NEVER commit it.
|
||||
Environment=GITHUB_OWNER=sneakyfree
|
||||
ExecStart=/bin/bash /srv/windygit/src/scripts/sync_from_github.sh
|
||||
Nice=10
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
# Orchestrator 09-23: compute-guard BLOCKS lane-owned findings; Grant-owned (ci/grant-owned.yml) stay WARN.
|
||||
Environment=COMPUTE_GUARD_MODE=block
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
# Orchestrator 09-24: secret-guard BLOCKS lane-owned findings; Grant-owned stay WARN.
|
||||
Environment=SECRET_GUARD_MODE=block
|
||||
3
deploy/systemd/windygit-sync.service.d/windy-job.conf
Normal file
3
deploy/systemd/windygit-sync.service.d/windy-job.conf
Normal file
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-sync 20m --expect "all repos in step with GitHub" --owner 13 -- /bin/bash /srv/windygit/src/scripts/sync_from_github.sh
|
||||
10
deploy/systemd/windygit-sync.timer
Normal file
10
deploy/systemd/windygit-sync.timer
Normal file
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Keep Windy Git in step with GitHub every 5 minutes
|
||||
|
||||
[Timer]
|
||||
OnBootSec=3min
|
||||
OnUnitActiveSec=5min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
16
deploy/systemd/windygit-tunnel.service
Normal file
16
deploy/systemd/windygit-tunnel.service
Normal file
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Windy Git - Cloudflare Tunnel (the only ingress; no inbound port is opened)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
# G1.4 - bounded, so a misbehaving ingress can never starve Grant's workstation.
|
||||
MemoryMax=512M
|
||||
CPUQuota=100%
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
28
deploy/windy0/nightly-runner-guard-sweep.sh
Executable file
28
deploy/windy0/nightly-runner-guard-sweep.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Nightly (Boss 10-01): runner-guard over the default branch of EVERY public repo in Grant's
|
||||
# 5 GitHub accounts (runs on Veron: windy-git scripts/runner_guard.py report). Writes
|
||||
# ~/windy-orchestra/RUNNER_GUARD.md (repo, file:line, rule; no file content), appends ONE
|
||||
# BOARD line per NEW hit vs the last run, and prints "runner-guard sweep: N hit(s)" last, so
|
||||
# the windy-job heartbeat (--expect "runner-guard sweep: 0 hit") goes red while any hit exists.
|
||||
set -euo pipefail
|
||||
page=~/windy-orchestra/RUNNER_GUARD.md
|
||||
state=~/.local/state/runner-guard-sweep.txt
|
||||
mkdir -p "$(dirname "$state")"
|
||||
out=$(timeout 900 ssh -o BatchMode=yes -o ConnectTimeout=15 ts-veron \
|
||||
'cd /srv/windygit/src && timeout 850 python3 scripts/runner_guard.py report' || true)
|
||||
summary=$(grep '^# runner-guard sweep:' <<<"$out" || echo "# runner-guard sweep: ERROR (no summary)")
|
||||
hits=$(grep -v '^#' <<<"$out" | grep . || true)
|
||||
{
|
||||
echo "# Runner guard: stranger-code paths to self-hosted runners ($(date -u '+%Y-%m-%d %H:%MZ'))"
|
||||
echo "_Nightly; windy-git scripts/runner_guard.py. R1 pull_request_target · R2 fork PR on self-hosted without a same-repo/environment gate · R3 outsider events (issue_comment, workflow_run, ...) on self-hosted · R0 unparseable._"
|
||||
echo; echo "${summary#\# }"; echo
|
||||
echo "| repo | file:line | rule | fix |"; echo "|---|---|---|---|"
|
||||
while IFS=$'\t' read -r repo loc rule msg; do [[ -n $repo ]] && echo "| $repo | $loc | $rule | $msg |"; done <<<"$hits"
|
||||
} > "$page"
|
||||
new=$(comm -13 <(sort -u "$state" 2>/dev/null || true) <(cut -f1-3 <<<"$hits" | sort -u))
|
||||
cut -f1-3 <<<"$hits" | sort -u > "$state"
|
||||
if [[ -n "$new" ]]; then
|
||||
n=$(grep -c . <<<"$new")
|
||||
echo "$(date -u +%Y-%m-%dT%H:%MZ) Windy Git: 🚨 runner-guard: $n NEW stranger-code path(s) to a self-hosted runner in public repos; see ~/windy-orchestra/RUNNER_GUARD.md" >> ~/windy-orchestra/BOARD.md
|
||||
fi
|
||||
echo "${summary#\# }"
|
||||
7
deploy/windy0/windy-runner-guard-sweep.service
Normal file
7
deploy/windy0/windy-runner-guard-sweep.service
Normal file
@@ -0,0 +1,7 @@
|
||||
[Unit]
|
||||
Description=Nightly runner-guard sweep of every PUBLIC repo's workflows (Windy Git lane)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/windy-job windy-runner-guard-sweep 26h --expect "runner-guard sweep: 0 hit" --owner 13 -- %h/bin/nightly-runner-guard-sweep.sh
|
||||
TimeoutStartSec=1200
|
||||
9
deploy/windy0/windy-runner-guard-sweep.timer
Normal file
9
deploy/windy0/windy-runner-guard-sweep.timer
Normal file
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Nightly runner-guard sweep
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 09:40:00 UTC
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -16,7 +16,11 @@ services:
|
||||
# I-12: baked at build time. A runtime COMMIT_SHA override is ignored.
|
||||
COMMIT_SHA: ${COMMIT_SHA_BUILD:-}
|
||||
BUILT_AT: ${BUILT_AT:-}
|
||||
env_file: [.env]
|
||||
env_file:
|
||||
- .env
|
||||
# WINDYGIT_TELEMETRY_TOKEN (root-only on Veron). Optional: no file = no telemetry.
|
||||
- path: /etc/windygit/telemetry.env
|
||||
required: false
|
||||
environment:
|
||||
DATABASE_URL: postgresql+asyncpg://windygit:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@db:5432/windygit
|
||||
GITEA_BASE_URL: http://gitea:3000
|
||||
@@ -28,7 +32,7 @@ services:
|
||||
|
||||
gitea:
|
||||
# G2.1 — PIN AN EXACT VERSION. Never `latest`. Record it in SUBSTRATE.md.
|
||||
image: docker.io/gitea/gitea:1.24.6
|
||||
image: docker.io/gitea/gitea:1.24.7
|
||||
environment:
|
||||
GITEA__database__DB_TYPE: postgres
|
||||
GITEA__database__HOST: db:5432
|
||||
@@ -57,12 +61,32 @@ services:
|
||||
# G2.2 — OIDC only. No local password login, no self-registration.
|
||||
GITEA__service__DISABLE_REGISTRATION: "true"
|
||||
GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "true"
|
||||
# SSO #8 (2026-09-23): the password and passkey forms are OFF. windyadmin
|
||||
# is site admin with a local password; leaving the form up made that
|
||||
# password a second, phishable way into the whole forge. Break-glass is
|
||||
# the CLI on Veron (`docker exec -u git windy-git-gitea-1 gitea admin ...`).
|
||||
GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
|
||||
GITEA__service__ENABLE_PASSKEY_AUTHENTICATION: "false"
|
||||
# G3.1 — a Windy account IS the account. Signing in with Windy provisions
|
||||
# the Gitea user on first arrival; nobody is asked to invent a second
|
||||
# identity for the same person, and no local password ever exists.
|
||||
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "true"
|
||||
# 🔴 OFF (2026-09-23). With it on, ANY stranger with a Windy Word account
|
||||
# (public signup, not even email-verified) got a forge account on first
|
||||
# sign-in — and the CI runners were instance-wide, so their workflows
|
||||
# would run on Veron beside the R2 god token. Proven with a throwaway
|
||||
# account, then closed. Opening the forge to non-Grant users is a §7
|
||||
# Grant decision; until then new accounts are created deliberately.
|
||||
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "false"
|
||||
GITEA__oauth2_client__USERNAME: email
|
||||
GITEA__oauth2_client__ACCOUNT_LINKING: auto
|
||||
# 🔴 `login`, NOT `auto` (SSO #8). `auto` linked any hub login whose EMAIL
|
||||
# matched an existing account — and windyadmin (SITE ADMIN) carries Grant's
|
||||
# email, so the forge's admin rights rested on the hub never letting anyone
|
||||
# else hold that address. `login` makes an email match prove possession of
|
||||
# the existing account first. Grant is unaffected: his account is already
|
||||
# linked by the hub's stable `sub`, which is matched before email.
|
||||
# ⚠️ env-to-ini SETS but never UNSETS — this value must also be edited in
|
||||
# /srv/windygit/git/gitea/conf/app.ini if it is ever removed from here.
|
||||
GITEA__oauth2_client__ACCOUNT_LINKING: login
|
||||
# The email is asserted by account-server, which is the authority on it.
|
||||
# Asking the user to re-verify an address their identity provider already
|
||||
# verified is friction that buys nothing.
|
||||
|
||||
116
docs/AUDIT-fable-2026-08-13.md
Normal file
116
docs/AUDIT-fable-2026-08-13.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Second-auditor review of the Windy Git build (Fable, 2026-08-13)
|
||||
|
||||
A fresh-eyes trace of Opus's build, **verified against the live system** rather
|
||||
than against the transcript's own account. One critical finding was fixed during
|
||||
the review; the rest are recorded here with proportion.
|
||||
|
||||
## Fixed during this review
|
||||
|
||||
### 🔴 Agent authentication was not authentication (CRITICAL, was live+public)
|
||||
|
||||
`auth.py` read the passport out of a bearer token **without verifying the EPT
|
||||
signature**, asked Eternitas *"is this passport reputable?"*, and seated the
|
||||
caller on a yes. That answers reputation, not possession.
|
||||
|
||||
**Proven by exploit:** a forged `alg:none` token naming a passport lifted from
|
||||
the logs returned **HTTP 200** as that agent on the public API. Anyone who knows
|
||||
a passport number (they appear in logs, the lockbox, and revocation messages)
|
||||
could impersonate that agent.
|
||||
|
||||
The human path already failed closed for exactly this reason
|
||||
(`require_verified_jwt`) — but the gate sat *after* the agent branch returned, so
|
||||
it protected the safe path and skipped the exploitable one.
|
||||
|
||||
**Fix:** the agent path now fails closed in production, *before* the trust
|
||||
lookup, mirroring the human path. Reopens automatically when the ES256/JWKS
|
||||
verifier (G3.2/G9.1) is built. Re-tested live: forged token now `503`. Added a
|
||||
**behavioral** test (forged token through real `get_caller` must raise) and a
|
||||
**canary probe** (forged token must stay refused; a 2xx pages).
|
||||
|
||||
## Open findings (recorded, not yet fixed)
|
||||
|
||||
### 🟠 The EI throttle table is dead code (MEDIUM)
|
||||
`BAND_MULTIPLIER` and `rate_*_per_day` (G3.4) are defined and **read by
|
||||
nothing** — verified by grep. An authenticated agent has no rate limit at all.
|
||||
When the agent path reopens with real verification, wire the throttle before it
|
||||
does, or a single agent can hammer repo-create/token-mint unbounded.
|
||||
|
||||
### 🟠 The test suite is mostly source-string assertions (MEDIUM)
|
||||
56 invariant tests carried **~86 "does the source contain this string"**
|
||||
assertions and **zero** that exercised the auth decision. `make check` green
|
||||
means the code still *contains* the patterns, not that it *behaves*. The auth
|
||||
bypass passed every test. **Direction:** convert the top ~10 guards into
|
||||
behavioral tests against an ephemeral instance (this review added the first
|
||||
two). The live-curl proofs Opus ran were excellent but were never captured, so
|
||||
they don't defend against regression.
|
||||
|
||||
### 🟠 Privileged dind sits beside broad-scoped tokens (MEDIUM — Grant-aware)
|
||||
`dind privileged=true`, and the host `.env` holds the account-wide R2 token and
|
||||
a GitHub PAT, on the same box running untrusted CI. Escape from privileged dind
|
||||
→ host fs → both tokens. Grant waived *minting a new token* for the sandbox; the
|
||||
specific escalation path (privileged-dind-next-to-god-token) is a separate
|
||||
decision. Lowest-effort mitigations: rootless/sysbox runner, or move the two
|
||||
tokens out of the API container's env into a path the API reads but the runner
|
||||
host does not share.
|
||||
|
||||
### ✅ Revocation — was WORSE than flagged, now fixed (was CRITICAL once agent auth reopened)
|
||||
Re-examined after real agent auth went live, and the finding grew teeth. A
|
||||
revoked passport returns **HTTP 200, `status: revoked`, `band: unproven`,
|
||||
`allowed_actions: []`** (verified live). `resolve_passport` keyed refusal only on
|
||||
HTTP 4xx and `band=="untrusted"`, so it returned band `unproven` and **seated the
|
||||
revoked agent** — revocation was not enforced on the live path at all. The
|
||||
webhook everyone worried about was only ever cache-invalidation; the live trust
|
||||
lookup was the real gate, and it wasn't checking.
|
||||
|
||||
**Fixed:** `decide_trust` now allows only `status=="active"`; revoked / suspended
|
||||
/ frozen / unknown all refuse, fail-closed. Revocation now takes effect on the
|
||||
next request, no webhook required. Eternitas additionally refuses to mint EPTs
|
||||
for revoked bots, so the residual window was a pre-existing ~365-day token —
|
||||
exactly what the live check now stops. 79 tests green including the full wiring.
|
||||
|
||||
The webhook (`webhook_secret` lost to the 500) is now genuinely LOW: it only
|
||||
matters for locally-issued credentials/grants (G6.3, not built), and it is no
|
||||
longer the thing standing between a revoked agent and access.
|
||||
|
||||
## What is genuinely strong (and worth protecting)
|
||||
|
||||
- **Honesty engineering is real:** fail-closed providers, `/health` refusing to
|
||||
claim green it can't prove, I-12's baked-sha proven live against a hostile env
|
||||
var. This directly cures the parent ecosystem's #1 root cause.
|
||||
- **Incident response was first-rate:** the login outage was root-caused to a
|
||||
510-deep accept queue and a per-query node fork, with the "one function, not
|
||||
468 call sites" reframe that is correct and valuable.
|
||||
- **It caught its own mistakes** — the mirror direction and the push-triggered
|
||||
deploy workflows, the latter *before* they fired.
|
||||
- **The DR posture is right:** 131 read-only mirrors (no CI, zero deploy risk) +
|
||||
a rehearsed restore. Rehearsed restore is rare in this ecosystem; keep it.
|
||||
|
||||
## The one-line lesson
|
||||
|
||||
Opus aimed its considerable discipline at **honesty and documentation**
|
||||
(excellent) more than at **adversarial correctness** — so the property that
|
||||
mattered most, *is an agent really that agent*, shipped inverted and untested.
|
||||
The remedy is not more process; it is **behavioral tests and canary probes for
|
||||
the security-critical paths**, so verification persists instead of living in a
|
||||
transcript.
|
||||
|
||||
## Disposition update — 2026-09-23 (lane 13)
|
||||
|
||||
**Privileged dind beside the tokens — materially reduced, not closed.**
|
||||
- The account-wide R2 token is **gone from Veron**. `.env` now carries a token scoped
|
||||
to Workers R2 Bucket Item Read/Write on `windy-git-lfs` + `windy-git-backups` only,
|
||||
minted by API (verified: works on both buckets, refused on any other). A CI escape
|
||||
now reaches Windy Git's own two buckets, not every bucket and zone in the account.
|
||||
- Runners take jobs **only from windyadmin-owned repos** (`action_runner.owner_id`), and
|
||||
forge self-registration is off, so no stranger's workflow can run here.
|
||||
- A host egress filter (`deploy/runner/egress.sh`) stops job containers reaching Veron,
|
||||
the LAN, WireGuard or Tailscale.
|
||||
- Still open: dind runs `--privileged` (next: Sysbox); the host still holds a GitHub
|
||||
token and the Gitea admin token.
|
||||
|
||||
**Revocation / webhook secret** — `ETERNITAS_WEBHOOK_SECRET` recovered from the
|
||||
Eternitas platform row and set; signed deliveries verify.
|
||||
|
||||
**Tests are string asserts** — the security paths now have behavioural suites
|
||||
(`test_hub_jwt.py`, `test_webhooks_behavior.py`, `test_pr_status_bridge.py`);
|
||||
a mutation check showed the old grep invariant passing a broken HMAC prefix strip.
|
||||
31
docs/CUTOVER-PRIMARY.md
Normal file
31
docs/CUTOVER-PRIMARY.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Checklist: making Windy Git the PRIMARY home of one repo (after launch)
|
||||
|
||||
Status 2026-10-01: DRAFT, nothing flipped. GitHub stays the source of truth until Grant says otherwise.
|
||||
First candidate: `windy-git` itself (public, low blast radius). GitHub becomes the free off-site push-mirror.
|
||||
|
||||
## Preconditions (all must be true)
|
||||
- [x] Encrypted state backup (Postgres + Gitea config + repos) nightly, restore drill passed cross-host (10-01).
|
||||
- [x] Gitea on a patched release (1.24.7); upgrade path = snapshot first (docs/RESTORE-DRILL.md).
|
||||
- [ ] Heartbeat `windygit-state-backup` in heartbeats-veron expected list (asked Cloud/Super Admin 10-01).
|
||||
- [ ] A missed/failed backup PAGES (heartbeat 26h), tested once by skipping a night in a drill.
|
||||
- [ ] Boss/Grant capacity call on Veron (dedicated non-SMR volume for DB + git storage; root disk < 80%).
|
||||
- [ ] Post-launch freeze lifted (Hub). No cutover during store review or a launch window.
|
||||
|
||||
## Cutover for ONE repo (reversible at every step)
|
||||
1. Announce on BOARD; tell every consuming lane (no schema drift rule). Pause the sync for that repo only:
|
||||
remove it from `REPOS` in `scripts/sync_from_github.sh` FIRST (the sync force-overwrites Windy Git).
|
||||
2. Verify Windy Git main == GitHub main (sha equal), all branches/tags present, LFS (if any) in R2.
|
||||
3. Add a PUSH-mirror on the Windy Git repo -> GitHub (deploy key or scoped token, write on that repo only,
|
||||
interval 8h + on-commit), so GitHub keeps a current copy. Test with a throwaway branch.
|
||||
4. Flip the lanes' remote: `origin` = Windy Git (SSH/HTTPS via Windy SSO token), `github` = secondary.
|
||||
Lanes push to Windy Git only; the mirror carries it to GitHub.
|
||||
5. CI keeps running here (no change); remove the `pr_status_bridge` mirror-PR duplication for that repo
|
||||
(PRs are now native here; the bridge's GitHub status posting for it can stay for any open GitHub PRs).
|
||||
6. Watch 3 days: mirror lag, backup includes the new writes, no push rejected, no lane still pushing to GitHub.
|
||||
7. Rollback at any time: re-add the repo to `REPOS` (sync resumes GitHub->Windy Git, GitHub is intact via the
|
||||
push-mirror) and point lanes' remote back. Nothing is destroyed in either direction.
|
||||
|
||||
## NOT in scope for a first cutover
|
||||
Deploy workflows (stay disabled; deploys remain manual until a separate runner + scoped deploy keys exist),
|
||||
credential repos (soul/anima/kit-army-config), windy-pro (importer refuses by name), opening the forge to
|
||||
other members (Grant 09-23: registration closed; Hub 10-01: jit false, 4 conditions before any change).
|
||||
192
docs/CUTOVER.md
192
docs/CUTOVER.md
@@ -1,63 +1,163 @@
|
||||
# Windy Git is the daily driver — 2026-08-13
|
||||
# Migration plan — GitHub first, Windy Git second, flip per repo
|
||||
|
||||
Grant's call, 2026-08-12: **push to Windy Git; GitHub is the second copy.**
|
||||
**Superseded the 2026-08-13 "daily driver" cutover, which was premature.**
|
||||
|
||||
you ──push──▶ Windy Git (Veron 1) ──▶ CI on 24 cores
|
||||
│
|
||||
└──push-mirror on every commit──▶ GitHub
|
||||
## What went wrong, recorded so it is not repeated
|
||||
|
||||
## 🔴 The one rule this creates
|
||||
Nine repos were migrated writable with **push-mirrors pointed at GitHub**. At
|
||||
the same time a dozen agent sessions on the Mac mini were pushing to GitHub
|
||||
continuously — so GitHub, not Windy Git, was where the live work actually was.
|
||||
|
||||
**Do not push directly to GitHub for a migrated repo.**
|
||||
A push-mirror force-updates refs. On its 8-hour timer it would have pushed Windy
|
||||
Git's stale copy **over live work, silently, with no conflict to notice.**
|
||||
|
||||
A push mirror makes GitHub match Windy Git. Anything committed straight to
|
||||
GitHub is **overwritten on the next sync**, silently, with no conflict and no
|
||||
warning. That is the cost of having one writer, and one writer is the point —
|
||||
two writers with no reconciliation is how you lose work you thought was saved.
|
||||
All nine mirrors were removed before the first timer fired, and every GitHub
|
||||
repo was verified untouched (latest push predated the mirrors). **No work was
|
||||
lost.** The mistake was direction, and the lesson is: *the source of truth is
|
||||
wherever people are actually typing, not wherever the plan says it should be.*
|
||||
|
||||
If you must hotfix on GitHub: push there, then immediately pull that commit into
|
||||
Windy Git *before* anything triggers a mirror sync. Better: don't.
|
||||
## Phase 1 — now. Nothing changes for anyone.
|
||||
|
||||
## Migrated (9)
|
||||
Mac mini agents ──push──▶ GitHub ──sync every 15 min──▶ Windy Git ──▶ CI on Veron
|
||||
|
||||
`windy-calendar` · `windy-search` · `windy-registry` · `Windy-Clone` ·
|
||||
`WindyCloud` · `windy-cloud-sites` · `windy-mind` · `eternitas` · `windy-agent`
|
||||
- **You do not have to tell your agents anything.** No remote changes, no
|
||||
coordination, no "everyone stop pushing." They keep working exactly as they
|
||||
are.
|
||||
- `windygit-sync.timer` runs `scripts/sync_from_github.sh` every 15 minutes.
|
||||
- Windy Git is **force-updated** on purpose: it holds nothing anyone depends on,
|
||||
so GitHub always wins and there is **no merge to reconcile**. That is the
|
||||
whole point of not flipping until a repo is quiet.
|
||||
- CI runs on Veron 1 against current code, on the 36 workflows that already say
|
||||
`runs-on: [self-hosted, linux, x64]`.
|
||||
|
||||
All writable (`mirror=false`), all push-mirroring to GitHub with
|
||||
`sync_on_commit: true`. Clone from `https://app.windygit.com/windyadmin/<repo>.git`.
|
||||
Tracked repos live in `REPOS` in the script (currently 9 of 141).
|
||||
|
||||
**Not migrated on purpose:** `windy-pro`. Six checkouts exist, the build counter
|
||||
has forked three ways (main 12 / overnight 34 / wave-44 56), and two sessions
|
||||
recorded different HEADs hours apart. Resolve which is current and write it
|
||||
down first (G11.5). The import script refuses it by name.
|
||||
## Phase 2 — later, one repo at a time, only when that repo is idle
|
||||
|
||||
## CI
|
||||
For a single repo, when nobody is mid-work on it:
|
||||
|
||||
The runner advertises `veron-1`, `linux-x64`, `self-hosted`, `linux`, `x64`.
|
||||
**36 of 36 active workflows in the fleet already say
|
||||
`runs-on: [self-hosted, linux, x64]`** — they were written for the self-hosted
|
||||
runners that died when the repos went private, so they run **as-is, unedited**.
|
||||
1. Remove it from `REPOS` in `sync_from_github.sh` — **first**, or the sync will
|
||||
fight its authors and win.
|
||||
2. Point that repo's sessions at Windy Git:
|
||||
`git remote set-url origin https://app.windygit.com/windyadmin/<repo>.git`
|
||||
3. Add a push-mirror back to GitHub with `sync_on_commit: true`, so GitHub stays
|
||||
a current second copy.
|
||||
|
||||
Proven: `windy-calendar`'s existing `.github/workflows/ci.yml` ran on Veron 1
|
||||
and reported success with no changes.
|
||||
**Never flip more than one repo at a time, and never while an agent is working
|
||||
in it.** A dozen parallel sessions is exactly the situation where a big-bang
|
||||
cutover produces the dirty-branch mess this plan exists to avoid.
|
||||
|
||||
**Per-repo secrets are not imported.** A repo whose CI needs a database URL or
|
||||
an API key will fail until those are set in its Gitea repo settings. Set them as
|
||||
each repo needs them, not speculatively.
|
||||
## The whole account is on Windy Git — in two tiers
|
||||
|
||||
143 repos · 1.58 GB · 967 GB free (matches the GitHub archive exactly)
|
||||
|
||||
| tier | count | writable | runs CI | deploy risk |
|
||||
|---|---|---|---|---|
|
||||
| **read-only mirrors** | 131 | no | **no** | **none** |
|
||||
| **writable + CI** | 12 | yes | yes | deploys disabled |
|
||||
|
||||
**Why the bulk is mirrors, and why that is the safety decision:** sampling 40
|
||||
repos found **18 carrying deploy / release / publish workflows that trigger on
|
||||
`push:`** — roughly 63 across the account. Importing those writable with Actions
|
||||
enabled would have armed sixty-odd production deploy triggers on Veron 1, each
|
||||
needing disarming by hand. **A pull mirror cannot run Actions at all**, so the
|
||||
bulk import carries zero execution risk and Gitea syncs it with no script and no
|
||||
timer.
|
||||
|
||||
That splits the two things cleanly: **having a copy** (safe, do it for
|
||||
everything, now) and **running code** (needs judgement, do it per repo,
|
||||
deliberately).
|
||||
|
||||
Seven repos are empty here because they are empty on GitHub — 0 KB upstream,
|
||||
verified. Not failed imports.
|
||||
|
||||
`windy-pro` **is** present, as a mirror. That is safe: the G11.5 caution is
|
||||
about making it *writable* while six checkouts and a three-way-forked build
|
||||
counter disagree on HEAD. A read-only copy of whatever GitHub currently has
|
||||
carries none of that risk — and it means the DR copy is complete.
|
||||
|
||||
## Promoting a mirror to writable + CI
|
||||
|
||||
Per repo, deliberately, when that repo is quiet:
|
||||
|
||||
1. delete the mirror, re-import with `mirror=false`
|
||||
2. **review its workflows and disable every deploying one** (see the section
|
||||
above — this is the step that matters)
|
||||
3. add it to `REPOS` in `sync_from_github.sh` so it tracks GitHub
|
||||
4. later, when it flips to Windy-Git-first: remove it from `REPOS` *first*,
|
||||
repoint its sessions, add a push-mirror back to GitHub
|
||||
|
||||
## Private repos: Windy Git IS their CI (permanent, 2026-09-23)
|
||||
|
||||
The platform repos stay **private** on GitHub (Grant, 2026-09-23), and private
|
||||
repos cannot run GitHub Actions on this account at all. Windy Git is therefore
|
||||
their CI permanently, not a stopgap:
|
||||
|
||||
GitHub push ──sync (15 min)──▶ Windy Git ──runner──▶ Veron 1
|
||||
▲ │
|
||||
└──── commit status windy-git/<workflow>/<job> ◀───┘ scripts/pr_status_bridge.py
|
||||
|
||||
- `pr_status_bridge.py` runs at the end of every sync. It opens a `[GH#N]`
|
||||
mirror PR in Windy Git for every open **same-repo** GitHub PR (so
|
||||
`pull_request` workflows fire), closes it when the GitHub PR closes, and posts
|
||||
each job's result back to GitHub on PR heads and the default-branch head.
|
||||
**Never merge a `[GH#N]` PR here** — merge on GitHub.
|
||||
- Fork PRs are never run: their branch is never synced, and untrusted code
|
||||
beside the privileged dind is the open audit finding.
|
||||
- Covered repos: `BRIDGE_REPOS` in the script. Public repos are left out on
|
||||
purpose; they run real GitHub Actions and two verdicts per commit is noise.
|
||||
- `skipped` jobs post nothing — no green for a job nobody ran.
|
||||
- **Image-build jobs** (name matches `docker`) post nothing: job containers
|
||||
have no Docker daemon by design (I-5), so they are red on every commit. A
|
||||
rootless builder (BuildKit rootless / buildx in the capped dind) is the open
|
||||
decision that would bring them back. Jobs that need Docker but are named otherwise go in
|
||||
`BRIDGE_NO_DAEMON` (empty since eternitas converted to ci/smoke, #179). DECIDED 09-23 (orchestrator,
|
||||
option A): lanes convert these jobs to no-Docker smoke tests (job `services:` +
|
||||
start the app + curl /health); the real image build is the deploy step on the
|
||||
target host. ci-hygiene flags docker build/compose/run in workflows ("needs docker").
|
||||
No host Docker socket for CI without a separate decision.
|
||||
|
||||
**Onboarding another private repo** — the promotion steps below, then:
|
||||
|
||||
# on Veron 1, as root
|
||||
set -a; . /srv/windygit/src/.env; set +a
|
||||
python3 scripts/import_from_github.py <repo> # writable; aborts if the repo exists
|
||||
# disable EVERY deploying workflow before anything is pushed:
|
||||
curl -X PUT -H "Authorization: token $GITEA_ADMIN_TOKEN" \
|
||||
http://localhost:3080/api/v1/repos/windyadmin/<repo>/actions/workflows/deploy.yml/disable
|
||||
# add <repo> to REPOS in sync_from_github.sh AND BRIDGE_REPOS in pr_status_bridge.py
|
||||
|
||||
An import fires no push event, so `main` has no verdict until its next commit.
|
||||
To get one now: force Windy Git's `main` back one commit, then
|
||||
`systemctl start windygit-sync` — the sync pushes it forward and CI fires.
|
||||
|
||||
⚠️ **`/actions/tasks` lists only jobs a runner has PICKED UP.** Queued runs are
|
||||
invisible there, so a repo can read "0 runs" while work is waiting. The truth is
|
||||
`action_run` in the `gitea` database (status 1 success, 2 failure, 5 waiting,
|
||||
6 running).
|
||||
|
||||
## ⚠️ Deploy workflows are DISABLED on Windy Git, deliberately
|
||||
|
||||
Six workflows fire on `push:` and deploy to production:
|
||||
`windy-registry`, `Windy-Clone`, `WindyCloud`, `windy-mind`, `eternitas`
|
||||
(`deploy.yml`) and `windy-agent` (`release.yml`).
|
||||
|
||||
Windy Git now has a working runner, so the next synced commit to `main` would
|
||||
have attempted a **production deploy from Veron 1**. Their secrets
|
||||
(`DEPLOY_HOST` / `DEPLOY_KEY` / `VPS_SSH_KEY`) are unset here, so they would
|
||||
have failed — but they would have failed *loudly on every push*, and any step
|
||||
before the SSH step would still have run.
|
||||
|
||||
All six are now `disabled_manually`. Tests, lints and migration checks stay
|
||||
**active** — those need no secrets at all, which is why Phase 1 delivers real CI
|
||||
value immediately.
|
||||
|
||||
**Before re-enabling any deploy workflow here, decide deliberately whether
|
||||
production should be deployable from Windy Git at all.** Kit 0 deploys are
|
||||
currently manual runbooks; that is a feature, not a gap.
|
||||
|
||||
## Backups
|
||||
|
||||
Nightly `windygit-backup.timer` at 04:17 (`Persistent=true`, so a window missed
|
||||
while the workstation is off is caught up rather than skipped). Every repo is
|
||||
bundled `--all`, verified, and uploaded to R2 with the `windgit` schema.
|
||||
|
||||
**Restore is rehearsed, not assumed:** a bundle was pulled back from R2, cloned,
|
||||
and its HEAD matched live `origin/main` exactly.
|
||||
|
||||
## Verify the loop yourself
|
||||
|
||||
```bash
|
||||
git clone https://app.windygit.com/windyadmin/windy-calendar.git
|
||||
cd windy-calendar && git commit --allow-empty -m "probe" && git push
|
||||
# CI runs on Veron 1; GitHub receives the commit within ~20s
|
||||
```
|
||||
`windygit-backup.timer`, nightly 04:17, `git bundle --all` + verify + `windgit`
|
||||
schema dump to R2, 30-day retention. **Restore rehearsed:** a bundle was pulled
|
||||
from R2, cloned, and its HEAD matched live `origin/main` exactly.
|
||||
|
||||
@@ -15,6 +15,7 @@ Mirrored into `windy-cloud` and `eternitas` on change.
|
||||
| eternitas | `GET /api/v1/trust/{passport}` | band + allowed_actions |
|
||||
| eternitas | `GET /api/v1/registry/{passport}/integrity` | ⚠️ note the path — `windy-registry` calls `/api/v1/passports/{p}/status`, which 404s, which is why the integrity index has never been populated |
|
||||
| account-server | OIDC discovery + JWKS | human identity (G3.1) |
|
||||
| windy-admin ledger | `POST /v1/events` (admin.windyword.ai) | field telemetry: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed`. Shapes are declared with the ledger owner BEFORE shipping (the server quarantines undeclared keys). Codes, counts, route templates only |
|
||||
| windy-cloud-sites | `POST /api/v1/sites/{id}/versions` | publish docs from a repo |
|
||||
|
||||
## Calls IN
|
||||
|
||||
33
docs/RESTORE-DRILL.md
Normal file
33
docs/RESTORE-DRILL.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# Restoring Windy Git from the encrypted state backup
|
||||
|
||||
What is backed up (`scripts/backup_state.sh`, restic repo `s3:…/windy-git-backups/restic`, tag `windygit-state`):
|
||||
both Postgres databases (`gitea`, `windygit`, custom-format dumps + globals), the whole Gitea data root
|
||||
(`/srv/windygit/git`: config, jwt, attachments, avatars, templates AND the bare repositories),
|
||||
`/srv/windygit/src/.env`, `deploy/runner/.env`, `/etc/cloudflared`, the windygit systemd drop-ins.
|
||||
NOT in it: the restic password itself (lockbox `RESTIC_WINDYGIT_PASSWORD`) and the R2 access key
|
||||
(scoped token `windy-git-r2-scoped`, lockbox). Never print either: use `lockbox-get KEY FILE`.
|
||||
|
||||
## Drill (any machine with restic + docker; proven on Veron 2026-10-01, counts identical)
|
||||
export RESTIC_PASSWORD_FILE=<0600 file from lockbox-get RESTIC_WINDYGIT_PASSWORD>
|
||||
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… # from lockbox-get, into env, not echoed
|
||||
export RESTIC_REPOSITORY=s3:https://<R2 account>.r2.cloudflarestorage.com/windy-git-backups/restic
|
||||
restic snapshots --tag windygit-state
|
||||
restic restore latest --tag windygit-state --target /var/tmp/wg-drill
|
||||
docker run -d --name wg-drill-pg -e POSTGRES_PASSWORD=<random> -e POSTGRES_USER=drill postgres:16-alpine
|
||||
for db in gitea windygit; do
|
||||
docker exec wg-drill-pg psql -U drill -d postgres -c "create database $db"
|
||||
docker exec -i wg-drill-pg pg_restore -U drill -d $db --no-owner --no-privileges \
|
||||
< /var/tmp/wg-drill/var/backups/windygit-state/$db.dump
|
||||
done
|
||||
# compare row counts with live (or with the last known): repository, issue, pull_request, "user",
|
||||
# external_login_user, access_token, action_run, action_run_job
|
||||
docker rm -f wg-drill-pg; rm -rf /var/tmp/wg-drill
|
||||
|
||||
## Real disaster (Veron lost)
|
||||
1. New Linux host with Docker, a Cloudflare tunnel connector, the repo (`git clone` from GitHub: windy-git).
|
||||
2. `restic restore latest --tag windygit-state --target /` (puts /srv/windygit/git, the .env files, /etc/cloudflared back).
|
||||
3. `docker compose -p windy-git up -d db`, then pg_restore both dumps into it (as above, into the real db names/owner from `.env`).
|
||||
4. `docker compose -p windy-git up -d` + `deploy/runner` runners; re-register runners if the token changed.
|
||||
5. Verify: `/api/healthz`, Windy SSO login, `git ls-remote`, one CI run. GitHub is still the source of truth for code,
|
||||
so repo content can also be re-synced from there; the database is what only this backup holds.
|
||||
Retention: 14 daily / 8 weekly / 6 monthly (prune on Sundays). Integrity: every run does `restic check --read-data-subset=2%`.
|
||||
@@ -1,6 +1,10 @@
|
||||
# RUNBOOK — Windy Git on Veron 1 (rung R0)
|
||||
|
||||
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron`. Passwordless sudo.
|
||||
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron` (or `ts-veron`). Passwordless sudo.
|
||||
|
||||
**Checkouts (one-repo doctrine):** the ONE standing dev checkout is **OC5
|
||||
`~/windy-git`** (platform repos live on OC5). `/srv/windygit/src` on Veron is the
|
||||
*deploy* copy — it holds no local work. Nothing else should exist.
|
||||
|
||||
⛔ **Kit 0 is never a host for this service** (D-4). `api/app/main.py` refuses to
|
||||
boot in production if it finds itself on `72.60.118.54`.
|
||||
@@ -15,6 +19,9 @@ boot in production if it finds itself on `72.60.118.54`.
|
||||
| `/etc/cloudflared/config.yml` | tunnel ingress |
|
||||
| `/etc/cloudflared/windy-git.json` | tunnel credentials, mode 600 |
|
||||
| `/srv/windygit/src/.env` | secrets, mode 600, **never committed** |
|
||||
| `/srv/windygit/git/gitea/conf/app.ini` | Gitea's persisted config — env-to-ini SETS but never UNSETS; edit here when removing a `GITEA__*` var |
|
||||
| `/srv/windygit/sync/*.git` | bare staging copies the GitHub→Windy Git sync pushes from |
|
||||
| `/srv/windygit/src/deploy/runner/.env` | `RUNNER_TOKEN` — a **windyadmin user-level** registration token (not instance-level; see CI) |
|
||||
|
||||
## Ports — all loopback, on purpose
|
||||
|
||||
@@ -22,7 +29,7 @@ boot in production if it finds itself on `72.60.118.54`.
|
||||
|---|---|
|
||||
| `127.0.0.1:3080` | Gitea (host 3000 is a resident node dev server; 3300 is nginx — **do not fight them for a port**) |
|
||||
| `127.0.0.1:8600` | windy-git API |
|
||||
| `127.0.0.1:2000` | cloudflared metrics |
|
||||
| `127.0.0.1:2001` | cloudflared metrics (`metrics:` in `/etc/cloudflared/config.yml`) — **not 2000**, see Troubleshooting |
|
||||
|
||||
**No inbound port is opened.** cloudflared dials out, so the dynamic residential
|
||||
IP is irrelevant and there is no firewall hole to maintain.
|
||||
@@ -41,12 +48,26 @@ sudo systemctl status windygit-tunnel
|
||||
|
||||
```bash
|
||||
ssh wg-veron
|
||||
cd /srv/windygit/src && git pull
|
||||
cd /srv/windygit/src && git fetch origin && git merge --ff-only origin/main # READ the output
|
||||
export COMMIT_SHA_BUILD=$(git rev-parse HEAD) BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
sudo -E docker compose up -d --build
|
||||
sudo -E docker compose up -d --build --no-deps api # API only: no forge restart
|
||||
curl -s https://api.windygit.com/version # MUST equal git rev-parse HEAD
|
||||
```
|
||||
|
||||
A Gitea config change (compose `GITEA__*`) needs `sudo docker compose up -d --no-deps gitea`
|
||||
— a ~6 s forge outage; running CI jobs survive it. Check `app.ini` afterwards.
|
||||
|
||||
⚠️ **Never `git pull -q` in a deploy script.** `-q` hides *errors*, not just
|
||||
noise. On 2026-08-14 a divergent branch made `pull -q` fail silently and the
|
||||
"deploy" ran for 20 minutes against stale code while reporting success. Use
|
||||
`git fetch && git merge --ff-only` (or `reset --hard origin/main` on THIS
|
||||
checkout only, which holds no local work) and read the output.
|
||||
|
||||
⚠️ **Never force-push a branch a deploy checkout tracks.** An earlier
|
||||
`git commit --amend` + `--force-with-lease` rewrote history `/srv/windygit/src`
|
||||
was already sitting on, orphaning it. If you must amend, re-point the deploy
|
||||
checkout in the same breath.
|
||||
|
||||
⚠️ **Never put `COMMIT_SHA` in `.env`.** It does nothing here — the sha is baked
|
||||
into the image and a runtime override is ignored with a warning (I-12). That env
|
||||
pin is the documented root cause of nine sibling services misreporting their
|
||||
@@ -61,11 +82,54 @@ curl -sI https://app.windygit.com/ | head -1 # Gitea, 200
|
||||
sudo ss -tlnp | grep -E "3080|8600" # both must be 127.0.0.1
|
||||
```
|
||||
|
||||
## Timers (host systemd units — the sync timer is NOT in the repo)
|
||||
|
||||
| Unit | Cadence | Does |
|
||||
|---|---|---|
|
||||
| `windygit-sync.timer` | every 5 min (`OnUnitActiveSec`) | GitHub → Windy Git for `REPOS` in `scripts/sync_from_github.sh`, then `scripts/pr_status_bridge.py` (mirror PRs + GitHub commit statuses). A manual `systemctl start` RESETS the 5-min clock. |
|
||||
| `windygit-backup.timer` | nightly | `git bundle` + pg_dump → R2, 30-day retention |
|
||||
| `windygit-ci-prune.timer` | every 6 h | `deploy/runner/prune.sh` — CI dind storage, 60 GB cap |
|
||||
| `windygit-tunnel.service` | always | the only ingress |
|
||||
|
||||
## CI (Gitea Actions) — see `docs/CUTOVER.md` for onboarding a repo
|
||||
|
||||
- **Six runners × capacity 1** (`deploy/runner/docker-compose.yml`), one shared
|
||||
dind capped at 12 cores / 64 GB. Capacity >1 in one runner shares
|
||||
`/root/.cache/act` between jobs and races (`lstat …: no such file`).
|
||||
- **Runners are scoped to the `windyadmin` user** (`action_runner.owner_id=1`),
|
||||
so only first-party repos run. A repo owned by anyone else — a plane-created
|
||||
agent or `u-system` repo — gets NO runner. Re-registrations inherit this
|
||||
because `RUNNER_TOKEN` is user-level.
|
||||
- Job ceiling 90 min (`config.yaml` `runner.timeout`); a `config.yaml` change
|
||||
needs each runner restarted **while idle** — `compose up -d` won't recreate it.
|
||||
- `/actions/tasks` lists only PICKED-UP jobs. Queue truth is `action_run_job`
|
||||
in the `gitea` DB: `sudo docker exec -i windy-git-db-1 psql -U windygit -d gitea`
|
||||
(status 1 ok · 2 fail · 3 cancelled · 4 skipped · 5 waiting · 6 running).
|
||||
- Job logs are in R2, not on disk. `GET /api/v1/repos/{o}/{r}/actions/jobs/{JOB_ID}/logs`
|
||||
takes the `action_run_job` id, not the task id.
|
||||
|
||||
## Sign-in posture
|
||||
|
||||
- Windy SSO only: password + passkey forms OFF, `ACCOUNT_LINKING=login`,
|
||||
**auto-registration OFF** — opening the forge to non-Grant users is a §7
|
||||
Grant decision.
|
||||
- **Break-glass:** `sudo docker exec -u git windy-git-gitea-1 gitea admin user generate-access-token --username windyadmin --token-name <name> --scopes <scopes> --raw`
|
||||
(delete it after: `delete from access_token where name='<name>'` in the gitea DB —
|
||||
Gitea refuses token management over token auth).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**A hostname returns 530 or won't resolve** — the tunnel is down. `sudo systemctl
|
||||
restart windygit-tunnel`, then `journalctl -u windygit-tunnel -n 50`.
|
||||
|
||||
**`windygit-tunnel` crash-loops with `bind: address already in use` on the metrics
|
||||
port** — cloudflared exits if it cannot bind `metrics:`, taking ingress with it.
|
||||
Until 2026-09-23 this unit restarted ~91,000 times because another project's
|
||||
`cornercall-tunnel` held 127.0.0.1:2000; ingress only survived because a stray
|
||||
generic `cloudflared.service` ran the same config (now disabled). Windy Git's
|
||||
metrics port is **2001**. `sudo ss -ltnp | grep :2001` names any squatter.
|
||||
Keep exactly ONE unit running `/etc/cloudflared/config.yml`: `windygit-tunnel`.
|
||||
|
||||
**TLS handshake fails with `curl` exit 35 and no HTTP status at all** — someone
|
||||
added a **two-level** hostname. Free Universal SSL covers `windygit.com` and
|
||||
`*.windygit.com` only. The request dies before the tunnel is consulted, so it
|
||||
@@ -88,3 +152,16 @@ disqualifying the moment a stranger depends on it. **The trigger is not a date
|
||||
it is the first external push.** Move the control plane to a dedicated VPS (not
|
||||
Kit 0), keep Veron 1 as the runner. It is an rsync, a Postgres dump and three
|
||||
DNS record edits.
|
||||
|
||||
## Re-run a PR's CI (Gitea 1.24 has no rerun API)
|
||||
|
||||
```bash
|
||||
ssh wg-veron
|
||||
cd /srv/windygit/src && bash scripts/rerun_ci.sh <repo> <branch> <github-head-sha-prefix>
|
||||
```
|
||||
Moves the Windy Git branch back one commit; the next sync force-pushes the
|
||||
GitHub head again and Gitea re-fires every workflow for that event on the same
|
||||
commit. Guarded: refuses unless the branch is at the given sha, waits for a
|
||||
sync that starts AFTER the rewind, restores the branch itself on timeout.
|
||||
Don't use the web "Re-run" button: it needs a hub-SSO session as windyadmin,
|
||||
which is Grant's identity.
|
||||
|
||||
262
docs/TURNOVER-2026-08-14.md
Normal file
262
docs/TURNOVER-2026-08-14.md
Normal file
@@ -0,0 +1,262 @@
|
||||
# Windy Git — turnover, 2026-08-14
|
||||
|
||||
Paste the block at the bottom into a fresh terminal. Everything above is context
|
||||
for whoever reads this file directly.
|
||||
|
||||
## Where things stand
|
||||
|
||||
Windy Git is **live and in use**: `app.windygit.com` (forge), `api.windygit.com`
|
||||
(our plane), on **Veron 1** behind a Cloudflare Tunnel, zero inbound ports, $0/mo.
|
||||
143 repos, 85 tests green, health `ok` on all four checks.
|
||||
|
||||
Grant signs in with his existing Windy Word credentials — no second account.
|
||||
Agents authenticate with real Eternitas EPT signature verification and are
|
||||
rate-limited by integrity band.
|
||||
|
||||
## SOLVED — the CI failures were never about Postgres
|
||||
|
||||
The `localhost` → `postgres` fix was correct and is worth keeping, but it was
|
||||
**not** what was failing these jobs. They died at step 2, before Postgres was
|
||||
ever contacted.
|
||||
|
||||
**Root cause: `astral-sh/setup-uv@v4` asks the forge for uv's latest release.**
|
||||
|
||||
setup-uv v4 added "resolve latest version instead of downloading latest release"
|
||||
(astral-sh/setup-uv#178). Resolution goes through `@actions/github`, whose
|
||||
octokit reads **`GITHUB_API_URL`** — which act_runner points at *our forge*. So
|
||||
the action requested:
|
||||
|
||||
```
|
||||
GET https://app.windygit.com/api/v1/repos/astral-sh/uv/releases/latest → 404
|
||||
```
|
||||
|
||||
Gitea has no `astral-sh/uv`, so it answered its standard 404 body, *"The target
|
||||
couldn't be found."* setup-uv threw that string, act printed it as `::error::`,
|
||||
and every later step was skipped by `success()`.
|
||||
|
||||
**The fix (merged to GitHub, 11 repos):** pin an explicit `version:` on every
|
||||
`setup-uv@v4`/`@v5` step. `resolveVersion()` short-circuits on an explicit
|
||||
version *before* any API call, and the download URL is hardcoded to github.com —
|
||||
so the forge round-trip disappears. Pinned to `0.12.5`, which is what `latest`
|
||||
already resolved to.
|
||||
|
||||
windy-mind #101, WindyCloud #90, eternitas #150, then the sweep: Windy-Clone #77,
|
||||
windy-agent #355, windy-call #34, windy-cell #31, windy-hand #6, windy-mail #105,
|
||||
windy-search #77, windy-text #29. All merged and synced.
|
||||
|
||||
### Two things that made this hard to see, both worth keeping
|
||||
|
||||
- **act attributes the error to the wrong step.** `::error::The target couldn't
|
||||
be found.` is printed immediately after `actions/checkout`'s `::remove-matcher`,
|
||||
so it reads exactly like a checkout failure. It is not. What settled it was the
|
||||
**Gitea access log** — `sudo docker logs windy-git-gitea-1 | grep " 404 "` — which
|
||||
named the real URL at the same millisecond as the job error. When a job fails
|
||||
with an opaque forge-shaped message, go to the forge's access log, not the job log.
|
||||
|
||||
- **The natural experiment was sitting right there.** windy-registry and
|
||||
windy-drops use `setup-uv@v3` and always passed; every v4/v5 caller failed. A
|
||||
version skew across otherwise-identical repos is a diagnosis, not a coincidence.
|
||||
|
||||
### The jobs API "job not found" that blocked the last session
|
||||
|
||||
Not a bug. `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs` requires the
|
||||
job id to belong to **the repo in the path** — a valid id under the wrong owner/repo
|
||||
404s. The API works fine; the URLs were mismatched. Logs are readable this way and
|
||||
you do **not** need the web UI.
|
||||
|
||||
Note logs are **not on disk** — `[storage] STORAGE_TYPE = minio` sends action logs
|
||||
to R2, so `actions_log/` on the host is empty. Read them through the API.
|
||||
|
||||
## SOLVED — second root cause: the runner was four majors behind
|
||||
|
||||
Windy-Clone failed for a completely different reason: `The runs.using key in
|
||||
action.yml must be one of: [composite docker node12 node16 node20 go], got
|
||||
**node24**`. act_runner **0.2.11**'s bundled act predates node24, so any repo
|
||||
pinning a current action major (`actions/checkout@v5`, `actions/setup-python@v6`)
|
||||
died before its first step.
|
||||
|
||||
**Bumped to `gitea/act_runner:0.6.1`** (`cd7dd9b`). Verified beforehand that
|
||||
`node24` is absent from the 0.2.11 binary and present in 0.6.1, and that every
|
||||
key in `deploy/runner/config.yaml` still exists in 0.6.1's schema — 0.6.1 only
|
||||
*adds* keys, so the config carried over untouched. The runner re-declared with
|
||||
the same id and labels `[veron-1 linux-x64 self-hosted linux x64]`; the
|
||||
registration in the `runner-data` volume survived. Windy-Clone went 4/4 red →
|
||||
4/4 green. Rollback is re-pinning 0.2.11; registration is backed up at
|
||||
`/srv/windygit/runner-registration.bak`.
|
||||
|
||||
## What is still red, and why each one is real
|
||||
|
||||
The CI plane is healthy. windy-mind is fully green (`742 passed, 1 skipped`).
|
||||
What remains are genuine repo defects that were **invisible before**, because
|
||||
every job died at step 2:
|
||||
|
||||
| repo | job | cause |
|
||||
|---|---|---|
|
||||
| WindyCloud | `lint` | `ruff format --check` — 7 files would be reformatted |
|
||||
| eternitas | `py-sdk` | `uv run pytest` → `Failed to spawn: pytest`; pytest isn't a declared dep of that project |
|
||||
| eternitas | `test` | needs its own look |
|
||||
| windy-agent | `test (3.12/3.13/3.14)` | all three died **together** at 22:50:19 after ~43 min, mid-suite at 64%, with no verdict in the log. Not the 30m `runner.timeout` (that would have fired at 22:36) and not the runner bump (that was 23:00). Something bulk-killed them; unexplained. |
|
||||
| WindyCloud | `docker`, windy-search `Docker build` | **architectural** — see below |
|
||||
|
||||
`WindyCloud`'s `docker` job wants to build an image and gets `failed to connect
|
||||
to the docker API at unix:///var/run/docker.sock`. Job containers deliberately
|
||||
have **no** docker socket (I-5, and `deploy/runner/docker-compose.yml` says in
|
||||
so many words not to mount it). Mounting the host socket would hand every
|
||||
workflow root on Veron 1. This needs a decision — buildx-in-dind, a rootless
|
||||
builder, or "this job does not run on Windy Git" — not a quiet socket mount.
|
||||
|
||||
The WindyCloud `lint` and eternitas `py-sdk` rows are small fixes in their own
|
||||
repos, left alone on purpose: they are product defects, not forge defects.
|
||||
|
||||
## Second, smaller finding — act's action cache rots
|
||||
|
||||
act caches action repos at `/root/.cache/act/<hash>` inside the runner container
|
||||
and refreshes them with a go-git mirror fetch of `refs/*:refs/*`, unforced. That
|
||||
includes `refs/pull/*`, which GitHub **recomputes** whenever a base branch moves.
|
||||
Reproduced directly:
|
||||
|
||||
```
|
||||
! [rejected] refs/pull/1015/merge -> refs/pull/1015/merge (non-fast-forward)
|
||||
```
|
||||
|
||||
which surfaces as `Non-terminating error while running 'git clone': some refs
|
||||
were not updated`, after which the action does not report `Checked out <ref>`.
|
||||
|
||||
It **has** now failed a job on its own. After the runner bump, `windy-mind tests`
|
||||
died with:
|
||||
|
||||
```
|
||||
❌ Failure - Main Install uv
|
||||
lstat /root/.cache/act/d3e6…/.git-blame-ignore-revs: no such file or directory
|
||||
```
|
||||
|
||||
act tars the cached action directory into the job container, and a file vanished
|
||||
mid-walk. The cache dir had been created at 23:01 and mutated again at 23:03,
|
||||
with `.gitignore` showing as deleted — act removes it before `docker cp`. The
|
||||
likely mechanism is **concurrent jobs sharing one cache dir**: `capacity: 4`, and
|
||||
windy-mind fires four jobs at once that all use setup-uv. Wiping the cache and
|
||||
re-running the job alone made it pass (`742 passed, 1 skipped`). *Mechanism not
|
||||
isolated* — the wipe alone may have been sufficient.
|
||||
|
||||
One dead end worth not repeating: the cached worktree sits at `38f3f104` while
|
||||
`git rev-parse v4` says `e4db8464`. That is **not** a wrong checkout — `v4` is an
|
||||
*annotated tag*, and `v4^{commit}` is `38f3f104`. Don't chase it.
|
||||
|
||||
Wipe with `sudo docker exec windy-git-runner-runner-1 rm -rf /root/.cache/act`
|
||||
(safe — container layer, not a volume). It will rot again. A real fix is either
|
||||
lowering `capacity` or upstream act; neither was attempted.
|
||||
|
||||
## Traps that will waste your time
|
||||
|
||||
- **Gitea status codes are not what they look like.** `1 = success, 2 = failure`,
|
||||
3 cancelled, 4 skipped, 5 waiting, 6 running, 7 blocked. Reading 1/2 as
|
||||
waiting/running inverts every conclusion you draw from `action_run_job`.
|
||||
- **Gitea sets `Secure` cookies** (ROOT_URL is https), so a `curl` login against
|
||||
`http://127.0.0.1:3080` silently keeps no session — it 303s to `/` and you
|
||||
still get "Sign In". Log in through `https://app.windygit.com`.
|
||||
- **There is no rerun API in 1.24.6.** `POST /api/v1/.../runs/{n}/rerun` 404s.
|
||||
Use the web route `POST /{owner}/{repo}/actions/runs/{n}/rerun` with the session
|
||||
cookie plus an `X-Csrf-Token` header taken from the `_csrf` cookie.
|
||||
- **`git pull -q` hides errors.** A divergent branch once made a "deploy" run 20
|
||||
minutes against stale code while reporting success. Use `git fetch && git
|
||||
merge --ff-only` and read the output.
|
||||
- **Never force-push a branch a deploy checkout tracks** (`--amend` orphaned
|
||||
`/srv/windygit/src` once).
|
||||
- **Gitea's env-to-ini SETS but never UNSETS**, and sometimes *appends a
|
||||
duplicate*. `GITEA__DEFAULT__APP_NAME` does not work at all — Gitea reads
|
||||
`APP_NAME` from the **top level** of `app.ini`; the env var creates a literal
|
||||
`[default]` section it ignores. Edit `app.ini` on the host.
|
||||
- **Cloudflare caches `/assets/*` for 6h and no token in this stack can purge.**
|
||||
Version brand asset **filenames** (`theme-windy.v2.css`), not query strings.
|
||||
- **`base64` wraps at 76 chars** and corrupts long tokens in test commands →
|
||||
`curl (43)`, phantom HTTP 000. Use `base64 -w0`.
|
||||
- **Kit 0 is fragile.** 54 containers on 4 vCPU. Two production incidents in two
|
||||
days, both from *non-production* workloads. Check `uptime` before deploying
|
||||
anything there, and build before recreating so the swap is seconds.
|
||||
- **Service containers**: use the service NAME and its INTERNAL port (5432),
|
||||
never the mapped host port. The three repos did NOT share one pattern — a naive
|
||||
`localhost` → `postgres` swap would have left WindyCloud on port 15432 (it maps
|
||||
`15432:5432`) and windy-registry on a `job.services.postgres.ports[…]` expression.
|
||||
|
||||
## Open items, roughly by value
|
||||
|
||||
1. **Decide what the image-building jobs should do on Windy Git** — WindyCloud
|
||||
`docker` and windy-search `Docker build` (above). The only remaining *forge*
|
||||
question; it needs a decision, not code.
|
||||
2. **windy-agent's three `test` jobs were bulk-killed at 22:50:19** after ~43
|
||||
minutes, mid-suite, with no verdict. Unexplained and not the runner bump.
|
||||
3. The product-level test failures in the table above.
|
||||
4. **act's action cache race** — lower `capacity` below 4, or accept re-runs.
|
||||
5. **Get non-prod work off Kit 0.** 12 dev/demo containers on the box running
|
||||
identity, the CA, mail, Matrix and the broker. Cost two incidents already;
|
||||
the postgres-adapter fix would not have prevented either.
|
||||
6. **Login is ~4–6s** — `postgres-adapter.ts:114` forks a `node -e` process per
|
||||
query. Measured: node startup alone is 1.7s on Kit 0 vs 0.01s on Veron. The
|
||||
fix is **one function** (persistent worker + `pg.Pool`), not the "468 call
|
||||
sites" the SOTU scoped. See `docs/incidents/2026-08-12-login-latency-analysis.md`.
|
||||
7. **Privileged dind sits beside broad-scoped tokens** on the CI host — Grant's
|
||||
call, needs a decision not a code change.
|
||||
8. Push-velocity throttling is declared but unenforceable from our plane (git
|
||||
push never touches the API); needs a Gitea pre-receive hook.
|
||||
|
||||
## Read these first
|
||||
|
||||
- `~/.claude/.../memory/project_windy_git.md` — the full record, densest source
|
||||
- `DNA_STRAND_MASTER_PLAN.md` — D-1…D-9 locked decisions, I-1…I-13 invariants
|
||||
- `docs/AUDIT-fable-2026-08-13.md` — second-auditor findings and dispositions
|
||||
- `docs/CUTOVER.md` — the GitHub↔Windy Git migration plan and its one rule
|
||||
|
||||
---
|
||||
|
||||
## Copy-paste prompt
|
||||
|
||||
```
|
||||
Picking up Windy Git (agent-native code+model host on Veron 1, live at
|
||||
app.windygit.com). Read these before doing anything:
|
||||
|
||||
1. ~/.claude/projects/-home-grantwhitmer/memory/project_windy_git.md
|
||||
2. ~/windy-git/docs/TURNOVER-2026-08-14.md
|
||||
3. ~/windy-git/DNA_STRAND_MASTER_PLAN.md (D-1..D-9, I-1..I-13)
|
||||
|
||||
State: live and in use. Grant signs in with his existing Windy account. Agents
|
||||
authenticate with real EPT signature verification. 143 repos, 85 tests green.
|
||||
|
||||
TWO CI root causes are SOLVED and verified:
|
||||
(a) setup-uv v4+ resolved uv's "latest" through GITHUB_API_URL, which
|
||||
act_runner points at our own forge, so it 404'd ("The target couldn't be
|
||||
found.") and every job died at step 2. Fixed by pinning an explicit uv
|
||||
version across 11 repos.
|
||||
(b) act_runner 0.2.11 predates `runs.using: node24`, so any repo on
|
||||
actions/checkout@v5 died before its first step. Bumped to 0.6.1.
|
||||
windy-mind is fully green (742 passed). Windy-Clone went 4/4 red to 4/4 green.
|
||||
|
||||
TASK: one decision, then cleanup.
|
||||
1. DECIDE what the image-building CI jobs should do here — WindyCloud `docker`
|
||||
and windy-search `Docker build`. They need a Docker daemon; job containers
|
||||
deliberately have no socket (I-5 — mounting the host socket hands every
|
||||
workflow root on Veron 1). Options: buildx inside the existing dind, a
|
||||
rootless builder, or exclude the job. Do NOT mount the host socket.
|
||||
2. windy-agent's three `test` jobs were bulk-killed together at 22:50:19 after
|
||||
~43 min, mid-suite, with no verdict in the log. Not the 30m runner.timeout,
|
||||
not the runner bump. Unexplained — worth a look.
|
||||
3. Small product defects: WindyCloud `lint` (ruff format, 7 files), eternitas
|
||||
`py-sdk` (pytest not a declared dep), eternitas `test`.
|
||||
|
||||
Ground rules already paid for the hard way:
|
||||
- Gitea job status: 1=SUCCESS, 2=FAILURE, 5=waiting, 6=running. Not what you'd guess.
|
||||
- When a job fails with an opaque forge-shaped error, read the FORGE access log
|
||||
(`docker logs windy-git-gitea-1 | grep " 404 "`) — act misattributes the error
|
||||
to the previous step.
|
||||
- Job logs: `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs`. The job id
|
||||
must belong to the repo in the path or you get a misleading "job not found".
|
||||
Logs are in R2, not on disk.
|
||||
- Log into the forge over https://app.windygit.com — Gitea's cookies are Secure,
|
||||
so a curl login to http://127.0.0.1:3080 silently keeps no session.
|
||||
- verify the WHOLE flow, not the half that curls easily
|
||||
- never `git pull -q` in a deploy path; it hides errors
|
||||
- fixes go to GitHub, not Windy Git (sync is GitHub -> Windy Git, force-push)
|
||||
- if a job fails with `lstat .../<file>: no such file or directory` on an
|
||||
action, act's cache rotted: `docker exec windy-git-runner-runner-1 rm -rf
|
||||
/root/.cache/act`, then re-run. Safe; it is a container layer, not a volume.
|
||||
- check Kit 0's `uptime` before deploying there; two incidents in two days
|
||||
```
|
||||
@@ -25,10 +25,13 @@ dependencies = [
|
||||
"alembic>=1.14",
|
||||
"httpx>=0.27",
|
||||
"boto3>=1.35",
|
||||
# ES256 verification of Eternitas EPTs (G3.2/G9.1). Without crypto extras
|
||||
# PyJWT cannot verify EC signatures and silently offers no protection.
|
||||
"pyjwt[crypto]>=2.9",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8.3", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
|
||||
dev = ["pytest>=8.3", "pyyaml>=6.0", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
|
||||
@@ -23,6 +23,23 @@ BUCKET="${R2_BUCKET_BACKUPS:-windy-git-backups}"
|
||||
KEEP_DAYS="${BACKUP_KEEP_DAYS:-30}"
|
||||
FAILED=0
|
||||
|
||||
# NEVER bundle these to R2 (orchestrator decision 2026-09-23). They carry
|
||||
# credentials in plaintext — kit-army-config IS the lockbox, and the soul repos
|
||||
# hold agent memory with keys in it — and these bundles are unencrypted, so
|
||||
# anyone holding the R2 key could read every secret in the fleet. They are
|
||||
# backed up ENCRYPTED elsewhere (Windy Drops lane, restic, restore-tested) and
|
||||
# stay mirrored on Veron's own disk in Gitea. Extended globs, matched on name.
|
||||
EXCLUDE="${BACKUP_EXCLUDE:-kit-army-config anima *-soul}"
|
||||
excluded() {
|
||||
local n=$1 pat pats
|
||||
read -ra pats <<< "$EXCLUDE" # read never glob-expands; `for p in $EXCLUDE` would
|
||||
for pat in "${pats[@]}"; do
|
||||
# shellcheck disable=SC2053 # unquoted RHS: glob match is the point
|
||||
[[ "$n" == $pat ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
cleanup() { rm -rf "$WORK"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
@@ -44,6 +61,10 @@ count=0
|
||||
for repo in "$GIT_ROOT"/*/*.git; do
|
||||
owner="$(basename "$(dirname "$repo")")"
|
||||
name="$(basename "$repo" .git)"
|
||||
if excluded "$name"; then
|
||||
log "skip ${owner}/${name} (credential-bearing: never bundled to R2 in plaintext)"
|
||||
continue
|
||||
fi
|
||||
out="$WORK/${owner}__${name}.bundle"
|
||||
|
||||
# --all captures every ref, not just the default branch. A bundle of one
|
||||
|
||||
66
scripts/backup_state.sh
Executable file
66
scripts/backup_state.sh
Executable file
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env bash
|
||||
# Nightly STATE backup: everything git bundles do NOT hold (SOTU 10-01: 625 issues/PRs, users,
|
||||
# SSO links, CI history, settings lived on one unbacked-up host). Encrypted restic repo in R2.
|
||||
# - Postgres: every database (custom-format dump, restore-listable) + globals
|
||||
# - Gitea config/data (app.ini, jwt, attachments, avatars, templates) + the bare repositories
|
||||
# - the deploy .env files, systemd drop-ins and the cloudflared tunnel config (needed to rebuild)
|
||||
# The restic password lives in /etc/windygit/restic.pass (root 600) AND the lockbox
|
||||
# (RESTIC_WINDYGIT_PASSWORD): a lost Veron must not lose the backups. NEVER echo env/values here.
|
||||
# Restore: docs/RESTORE-DRILL.md. Bounded: every docker exec runs under `timeout` (a hung
|
||||
# runc exec in the IO stall wedged the sync on 09-23).
|
||||
set -euo pipefail
|
||||
log() { echo "[backup_state $(date -u +%FT%TZ)] $*"; }
|
||||
: "${R2_ACCOUNT_ID:?}" "${R2_ACCESS_KEY_ID:?}" "${R2_SECRET_ACCESS_KEY:?}"
|
||||
PASSFILE="${RESTIC_PASSWORD_FILE:-/etc/windygit/restic.pass}"
|
||||
[[ -s "$PASSFILE" ]] || { log "FATAL: $PASSFILE missing/empty: refusing to report a backup that did not happen"; exit 1; }
|
||||
export RESTIC_PASSWORD_FILE="$PASSFILE"
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export RESTIC_REPOSITORY="${RESTIC_REPOSITORY:-s3:https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com/${R2_BUCKET_BACKUPS:-windy-git-backups}/restic}"
|
||||
DB="${WG_DB_CONTAINER:-windy-git-db-1}"
|
||||
STAGE="${WG_STAGE:-/var/backups/windygit-state}"
|
||||
GIT_ROOT="${GIT_DATA_ROOT:-/srv/windygit/git}"
|
||||
umask 077
|
||||
mkdir -p "$STAGE"; chmod 700 "$STAGE"; rm -f "$STAGE"/*.dump "$STAGE"/globals.sql
|
||||
|
||||
# systemd gives units no $HOME, and restic wants a cache dir: pin one.
|
||||
export RESTIC_CACHE_DIR="${RESTIC_CACHE_DIR:-/var/cache/windygit-restic}"; mkdir -p "$RESTIC_CACHE_DIR"
|
||||
if ! err=$(restic cat config 2>&1 >/dev/null); then
|
||||
# only a MISSING repo may be initialised; any other error (auth, network, wrong password) must stop here
|
||||
if grep -qiE "does not exist|is there a repository|unable to open config file" <<<"$err"; then
|
||||
log "initialising restic repo"; restic init >/dev/null
|
||||
else
|
||||
log "FATAL: restic cannot open the repository: $(head -c 300 <<<"$err" | tr '\n' ' ')"; exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
PGU=$(timeout 30 docker exec "$DB" printenv POSTGRES_USER)
|
||||
[[ -n "$PGU" ]] || { log "FATAL: no POSTGRES_USER in $DB"; exit 1; }
|
||||
dbs=$(timeout 60 docker exec "$DB" psql -U "$PGU" -Atc "select datname from pg_database where not datistemplate and datname<>'postgres' order by 1")
|
||||
n=0
|
||||
for d in $dbs; do
|
||||
timeout 600 docker exec "$DB" pg_dump -U "$PGU" -Fc "$d" > "$STAGE/$d.dump"
|
||||
# a dump that cannot be listed is not a backup
|
||||
timeout 120 docker exec -i "$DB" pg_restore -l < "$STAGE/$d.dump" >/dev/null
|
||||
[[ $(stat -c%s "$STAGE/$d.dump") -gt 1000 ]] || { log "FATAL: $d dump suspiciously small"; exit 1; }
|
||||
n=$((n+1)); log "dumped $d ($(stat -c%s "$STAGE/$d.dump") bytes)"
|
||||
done
|
||||
[[ $n -ge 1 ]] || { log "FATAL: no databases dumped"; exit 1; }
|
||||
timeout 120 docker exec "$DB" pg_dumpall -U "$PGU" --globals-only > "$STAGE/globals.sql"
|
||||
|
||||
paths=("$STAGE" "$GIT_ROOT" /srv/windygit/src/.env /srv/windygit/src/deploy/runner/.env /etc/cloudflared)
|
||||
for p in /etc/systemd/system/windygit-*.service.d /etc/windygit; do [[ -e $p ]] && paths+=("$p"); done
|
||||
# restic.pass itself is excluded: the password never rides in its own backup
|
||||
snap=$(restic backup --tag windygit-state --host windygit-veron --quiet --json \
|
||||
--exclude "$GIT_ROOT/gitea/log" --exclude "$GIT_ROOT/gitea/queues" --exclude "$GIT_ROOT/gitea/tmp" \
|
||||
--exclude "$GIT_ROOT/gitea/indexers" --exclude "$GIT_ROOT/gitea/actions_log" --exclude /etc/windygit/restic.pass \
|
||||
"${paths[@]}" | python3 -c 'import sys,json
|
||||
for l in sys.stdin:
|
||||
d=json.loads(l)
|
||||
if d.get("message_type")=="summary": print(d["snapshot_id"][:8])')
|
||||
[[ -n "$snap" ]] || { log "FATAL: restic produced no snapshot"; exit 1; }
|
||||
rm -f "$STAGE"/*.dump "$STAGE"/globals.sql
|
||||
restic check --read-data-subset=2% --quiet >/dev/null || { log "FATAL: restic check failed"; exit 1; }
|
||||
if [[ $(date +%u) == 7 ]]; then
|
||||
restic forget --tag windygit-state --keep-daily 14 --keep-weekly 8 --keep-monthly 6 --prune --quiet >/dev/null
|
||||
fi
|
||||
echo "ok — state backed up ($n dbs, snapshot $snap)"
|
||||
@@ -33,6 +33,7 @@ import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
STATE_PATH = os.environ.get("CANARY_STATE", "canary-state.json")
|
||||
@@ -46,6 +47,16 @@ ALERT_FROM = os.environ.get("CANARY_ALERT_FROM", "office@thewindstorm.uk")
|
||||
LOGIN_WARN_SECONDS = float(os.environ.get("CANARY_LOGIN_WARN_S", "35"))
|
||||
TIMEOUT = float(os.environ.get("CANARY_TIMEOUT_S", "60"))
|
||||
|
||||
# Journey cleanup rule (orchestrator, 2026-09-23). The login probe creates a hub
|
||||
# session (access + refresh token) every run, so it must end it. The hub's
|
||||
# /auth/logout revokes the token AND every refresh token of the account
|
||||
# (verified live: access 401, refresh 401 after it). So the next successful
|
||||
# logout also heals anything a failed run left behind; no ledger needed.
|
||||
LOGOUT_URL = "https://account.windyword.ai/api/v1/auth/logout"
|
||||
LOGOUT_ATTEMPTS = 8 # retried on 5xx / no response only
|
||||
LOGOUT_GAP_S = 15.0
|
||||
LOGOUT_GONE = (401, 404, 410) # the session is already over = done
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
@@ -54,6 +65,7 @@ class Result:
|
||||
detail: str
|
||||
seconds: float = 0.0
|
||||
user_visible: str = ""
|
||||
followups: list[Result] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -65,11 +77,19 @@ class Check:
|
||||
body: dict | None = None
|
||||
headers: dict = field(default_factory=dict)
|
||||
warn_seconds: float | None = None
|
||||
# When True this check INVERTS: a 2xx is a critical failure (a security
|
||||
# control opened) and a 401/403/503 is the healthy, expected outcome.
|
||||
must_refuse: bool = False
|
||||
# Runs on a 2xx with the response body; returns follow-up results (cleanup).
|
||||
after: Callable[[bytes], list[Result]] | None = None
|
||||
|
||||
|
||||
def _probe(c: Check) -> Result:
|
||||
data = json.dumps(c.body).encode() if c.body else None
|
||||
headers = {"User-Agent": "windy-git-canary/1.0", **c.headers}
|
||||
# Our own probes are synthetic traffic (ecosystem convention, Telemetry
|
||||
# UPDATE 4): every service they touch labels the resulting rows.
|
||||
headers["X-Windy-Synthetic"] = "1"
|
||||
if data:
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(c.url, data=data, method=c.method, headers=headers)
|
||||
@@ -77,16 +97,30 @@ def _probe(c: Check) -> Result:
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
elapsed = time.monotonic() - start
|
||||
if c.must_refuse:
|
||||
# A 2xx here means a control that should reject accepted. That is
|
||||
# the alarm, not the absence of one.
|
||||
return Result(c.name, "down",
|
||||
f"ACCEPTED (HTTP {r.status}) — this MUST be refused",
|
||||
elapsed, c.what_it_proves)
|
||||
if r.status >= 400:
|
||||
return Result(c.name, "down", f"HTTP {r.status}", elapsed, c.what_it_proves)
|
||||
raw = r.read()
|
||||
warn = c.warn_seconds
|
||||
if warn and elapsed > warn:
|
||||
return Result(
|
||||
res = Result(
|
||||
c.name, "slow", f"HTTP {r.status} in {elapsed:.1f}s (warn >{warn:.0f}s)",
|
||||
elapsed, c.what_it_proves,
|
||||
)
|
||||
return Result(c.name, "ok", f"HTTP {r.status} in {elapsed:.1f}s", elapsed, c.what_it_proves)
|
||||
else:
|
||||
res = Result(c.name, "ok", f"HTTP {r.status} in {elapsed:.1f}s", elapsed, c.what_it_proves)
|
||||
if c.after:
|
||||
res.followups = c.after(raw)
|
||||
return res
|
||||
except urllib.error.HTTPError as e:
|
||||
if c.must_refuse and e.code in (401, 403, 503):
|
||||
return Result(c.name, "ok", f"correctly refused (HTTP {e.code})",
|
||||
time.monotonic() - start, c.what_it_proves)
|
||||
return Result(c.name, "down", f"HTTP {e.code}", time.monotonic() - start, c.what_it_proves)
|
||||
except Exception as e: # noqa: BLE001 — a probe must never raise upward
|
||||
return Result(
|
||||
@@ -95,6 +129,52 @@ def _probe(c: Check) -> Result:
|
||||
)
|
||||
|
||||
|
||||
def logout(token: str, *, attempts: int = LOGOUT_ATTEMPTS, gap: float = LOGOUT_GAP_S,
|
||||
sleep: Callable[[float], None] = time.sleep) -> Result:
|
||||
"""End the session the login probe opened. Honest: never ok unless proven."""
|
||||
what = "the canary leaves no live session behind (journey cleanup rule)"
|
||||
headers = {
|
||||
"User-Agent": "windy-git-canary/1.0",
|
||||
"X-Windy-Synthetic": "1",
|
||||
"Authorization": f"Bearer {token}",
|
||||
}
|
||||
start = time.monotonic()
|
||||
last = "no attempt"
|
||||
for i in range(attempts):
|
||||
if i:
|
||||
sleep(gap)
|
||||
req = urllib.request.Request(LOGOUT_URL, data=b"", method="POST", headers=headers)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
return Result("identity.logout", "ok", f"session ended (HTTP {r.status})",
|
||||
time.monotonic() - start, what)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code in LOGOUT_GONE:
|
||||
return Result("identity.logout", "ok", f"session already over (HTTP {e.code})",
|
||||
time.monotonic() - start, what)
|
||||
if e.code < 500: # a 4xx won't change on retry: fail fast
|
||||
return Result("identity.logout", "down", f"CLEANUP FAILED: HTTP {e.code}",
|
||||
time.monotonic() - start, what)
|
||||
last = f"HTTP {e.code}"
|
||||
except Exception as e: # noqa: BLE001 — no response / timeout: retry
|
||||
last = f"{type(e).__name__}"
|
||||
return Result("identity.logout", "down",
|
||||
f"CLEANUP FAILED after {attempts} tries: {last} (next run's logout heals it)",
|
||||
time.monotonic() - start, what)
|
||||
|
||||
|
||||
def _logout_after_login(raw: bytes) -> list[Result]:
|
||||
try:
|
||||
token = (json.loads(raw or b"{}") or {}).get("token")
|
||||
except ValueError:
|
||||
token = None
|
||||
if not token:
|
||||
return [Result("identity.logout", "down",
|
||||
"CLEANUP FAILED: login returned no token to log out with",
|
||||
0.0, "the canary leaves no live session behind (journey cleanup rule)")]
|
||||
return [logout(token)]
|
||||
|
||||
|
||||
def build_checks() -> list[Check]:
|
||||
checks = [
|
||||
Check(
|
||||
@@ -129,6 +209,36 @@ def build_checks() -> list[Check]:
|
||||
),
|
||||
]
|
||||
|
||||
# SECURITY REGRESSION GUARD. A forged, unsigned token naming a real passport
|
||||
# must be refused. On 2026-08-13 this returned HTTP 200 (full agent
|
||||
# impersonation). If it ever returns 2xx again, the bypass is back.
|
||||
import base64 as _b64
|
||||
import json as _j
|
||||
|
||||
def _seg(d: dict) -> str:
|
||||
return _b64.urlsafe_b64encode(_j.dumps(d).encode()).rstrip(b"=").decode()
|
||||
|
||||
# Two shapes, because they exercise two different gates. The EPT-shaped one
|
||||
# is the important one now: it is what real signature verification guards.
|
||||
for _label, _hdr in (
|
||||
("security.forged_agent_token", {"alg": "none", "typ": "JWT"}),
|
||||
("security.forged_ept", {"alg": "none", "typ": "EPT"}),
|
||||
):
|
||||
_tok = (
|
||||
f"{_seg(_hdr)}."
|
||||
f"{_seg({'sub': 'ET26-1EF9-VJAN', 'passport': 'ET26-1EF9-VJAN', 'iss': 'eternitas.ai', 'exp': 9999999999})}"
|
||||
".not-a-real-signature"
|
||||
)
|
||||
checks.append(
|
||||
Check(
|
||||
_label,
|
||||
"https://api.windygit.com/api/v1/repos",
|
||||
"an unsigned token cannot impersonate an agent",
|
||||
headers={"Authorization": f"Bearer {_tok}"},
|
||||
must_refuse=True,
|
||||
)
|
||||
)
|
||||
|
||||
# THE important one. /health was 200 for the entire 2026-08-12 outage while
|
||||
# this was timing out. A canary that skips it is decorative.
|
||||
pw = os.environ.get("CANARY_LOGIN_PASSWORD", "")
|
||||
@@ -142,6 +252,7 @@ def build_checks() -> list[Check]:
|
||||
method="POST",
|
||||
body={"email": email, "password": pw},
|
||||
warn_seconds=LOGIN_WARN_SECONDS,
|
||||
after=_logout_after_login,
|
||||
)
|
||||
)
|
||||
return checks
|
||||
@@ -218,7 +329,10 @@ def main() -> int:
|
||||
args = ap.parse_args()
|
||||
|
||||
previous = load_state()
|
||||
results = [_probe(c) for c in build_checks()]
|
||||
results = []
|
||||
for c in build_checks():
|
||||
r = _probe(c)
|
||||
results += [r, *r.followups]
|
||||
|
||||
print(f"windy canary — {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}\n")
|
||||
for r in results:
|
||||
|
||||
10
scripts/cancel_unrunnable.sh
Executable file
10
scripts/cancel_unrunnable.sh
Executable file
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Cancel jobs no runner can ever take (see cancel_unrunnable.sql). Run on Veron as root.
|
||||
set -euo pipefail
|
||||
SPOOL="${JANITOR_SPOOL:-/var/lib/windy-git/janitor-cancelled.jsonl}"
|
||||
mkdir -p "$(dirname "$SPOOL")"
|
||||
out=$(docker exec -i windy-git-db-1 sh -c 'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1' \
|
||||
< "$(dirname "$0")/cancel_unrunnable.sql")
|
||||
printf '%s\n' "$out" | grep '^{' >> "$SPOOL" || true
|
||||
n=$(printf '%s\n' "$out" | grep -c '^{' || true)
|
||||
echo "[janitor] cancelled ${n} unrunnable job(s)"
|
||||
53
scripts/cancel_unrunnable.sql
Normal file
53
scripts/cancel_unrunnable.sql
Normal file
@@ -0,0 +1,53 @@
|
||||
-- Cancel CI jobs that can never run (called by scripts/cancel_unrunnable.sh).
|
||||
--
|
||||
-- A job whose runs-on names a label no Windy Git runner offers (ubuntu-latest,
|
||||
-- macos-latest, windows-latest …) waits forever: Gitea evaluates a job's `if:`
|
||||
-- only when a runner picks it, so even `if: false` / tag-only jobs sit in the
|
||||
-- queue, invisible to /actions/tasks, and keep their run "waiting" for good.
|
||||
-- After 30 minutes they are cancelled here; the run's status is then recomputed
|
||||
-- (failure > still-active > cancelled > success), the same precedence Gitea uses.
|
||||
-- Keep RUNNER_LABELS in step with deploy/runner/config.yaml.
|
||||
BEGIN;
|
||||
WITH dead AS (
|
||||
UPDATE action_run_job j
|
||||
SET status = 3, stopped = extract(epoch from now())::bigint, updated = extract(epoch from now())::bigint
|
||||
WHERE j.status IN (5, 7)
|
||||
AND to_timestamp(j.created) < now() - interval '30 minutes'
|
||||
AND EXISTS (SELECT 1 FROM jsonb_array_elements_text(j.runs_on::jsonb) l
|
||||
WHERE l NOT IN ('veron-1', 'linux-x64', 'self-hosted', 'linux', 'x64'))
|
||||
RETURNING j.id, j.run_id, j.name, j.runs_on, j.created
|
||||
), runs AS (
|
||||
UPDATE action_run r
|
||||
SET status = CASE
|
||||
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status = 2) THEN 2
|
||||
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status IN (5, 6, 7)
|
||||
AND x.id NOT IN (SELECT id FROM dead)) THEN r.status
|
||||
ELSE 3 END,
|
||||
stopped = CASE WHEN r.stopped = 0 THEN extract(epoch from now())::bigint ELSE r.stopped END
|
||||
WHERE r.id IN (SELECT DISTINCT run_id FROM dead)
|
||||
RETURNING r.id
|
||||
)
|
||||
-- One JSON line per cancelled job: the telemetry emitter ships these as
|
||||
-- ci.job_cancelled (declared with Telemetry Boss, 2026-09-23).
|
||||
SELECT json_build_object(
|
||||
'repo', p.lower_name,
|
||||
'workflow', regexp_replace(r.workflow_id, '\.ya?ml$', ''),
|
||||
'job', d.name,
|
||||
'reason', 'unrunnable_label',
|
||||
'runs_on', (SELECT string_agg(l, ',') FROM jsonb_array_elements_text(d.runs_on::jsonb) l),
|
||||
'waited_s', (extract(epoch from now())::bigint - d.created))::text
|
||||
FROM dead d JOIN action_run r ON r.id = d.run_id JOIN repository p ON p.id = r.repo_id
|
||||
WHERE (SELECT count(*) FROM runs) >= 0;
|
||||
|
||||
-- Jobs BLOCKED on `needs:` inside a run that has already finished (a needed job
|
||||
-- failed): Gitea leaves them status 7 forever. They were never going to run;
|
||||
-- mark them skipped (4), which is what GitHub shows for the same situation.
|
||||
UPDATE action_run_job j
|
||||
SET status = 4, updated = extract(epoch from now())::bigint
|
||||
FROM action_run r
|
||||
WHERE r.id = j.run_id
|
||||
AND j.status = 7
|
||||
AND r.status IN (1, 2, 3)
|
||||
AND to_timestamp(j.created) < now() - interval '30 minutes'
|
||||
RETURNING j.run_id;
|
||||
COMMIT;
|
||||
258
scripts/ci_hygiene.py
Normal file
258
scripts/ci_hygiene.py
Normal file
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CI hygiene guard: installs come from a lockfile, never "latest" (house rule 6).
|
||||
|
||||
A floating install lets CI test different versions than prod ships, and a
|
||||
rebuild silently changes prod. Windy Cloud's OpenAPI test failed on exactly
|
||||
that (fastapi 0.141.1 in CI vs 0.136.0 on the dev box) and all three Cloud
|
||||
cells floated in prod. Also flags services that publish a HOST port: every
|
||||
CI job shares one dind daemon, so two jobs publishing 5432 collide ("port is
|
||||
already allocated", Windy Mind runs 147/176).
|
||||
|
||||
WARN-ONLY (`windy-git/ci-hygiene`, green + "⚠ WARN"); CI_HYGIENE_MODE=block
|
||||
turns it red once the lanes report clean. Scans CI workflow files and
|
||||
Dockerfiles only. PR heads: lines the PR adds. Default branch: every line.
|
||||
|
||||
OK (not flagged):
|
||||
pip / uv pip install -r FILE (with or without --require-hashes), --no-deps,
|
||||
exact pins (tool==1.2.3), pip/setuptools/wheel upgrades
|
||||
uv sync --locked | --frozen npm ci
|
||||
npm install pkg@1.2.3 (every package exact-pinned)
|
||||
yarn install --frozen-lockfile / --immutable pnpm install --frozen-lockfile
|
||||
Also flagged: `:latest` images (FROM / COPY --from / image: / docker://) and
|
||||
`COPY uv.lock* ...`-style globs that build without the lock (Windy Mail #147).
|
||||
Exceptions: ci/ci-hygiene-allow.yml, one reason per entry.
|
||||
|
||||
python3 scripts/ci_hygiene.py report [repo ...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import compute_guard as cg # noqa: E402 (shared walker, cache and allow-list loader)
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ALLOW_FILE = Path(os.environ.get("CI_HYGIENE_ALLOW", ROOT / "ci" / "ci-hygiene-allow.yml"))
|
||||
MODE = os.environ.get("CI_HYGIENE_MODE", "warn")
|
||||
|
||||
# CI workflow files and Dockerfiles; never vendored copies.
|
||||
INCLUDE = re.compile(r"(^|/)\.(github|gitea)/workflows/[^/]+\.ya?ml$|(^|/)(Dockerfile[^/]*|[^/]+\.Dockerfile)$")
|
||||
NEVER = re.compile(r"(^|/)(node_modules|vendor|third_party)/")
|
||||
PREFILTER = (r"pip3? install|pip install|uv sync|npm (install|i )|yarn install|pnpm install"
|
||||
r"|^\s*-\s*['\"]?[0-9]+:[0-9]+|:latest|lock[^ ]*\*"
|
||||
r"|docker[ -]compose|docker (build|buildx|run)|docker/build-push-action")
|
||||
|
||||
TOOLING = {"pip", "setuptools", "wheel"}
|
||||
NO_DOCKER_FIX = "use job services: + a no-Docker smoke test; the image builds at deploy"
|
||||
DOCKER_FILE = re.compile(r"(^|/)(Dockerfile[^/]*|[^/]+\.Dockerfile)$")
|
||||
LATEST = re.compile(r"(?:^\s*FROM\s+(?:--platform=\S+\s+)?|--from=|image:\s*['\"]?|docker://)([\w./-]+):latest\b", re.I)
|
||||
LOCKNAME = re.compile(r"(uv\.lock|poetry\.lock|package-lock\.json|pnpm-lock\.yaml|yarn\.lock|requirements[^ ]*\.(txt|lock))", re.I)
|
||||
EXACT_PY = re.compile(r"^[A-Za-z0-9._-]+(\[[^\]]*\])?==[A-Za-z0-9.+!-]+$")
|
||||
EXACT_NPM = re.compile(r"^(@[^/@]+/)?[^/@]+@\d+\.\d+\.\d+([-+][0-9A-Za-z.-]+)?$")
|
||||
HOST_PORT = re.compile(r"^\s*-\s*['\"]?(\d{2,5}):(\d{2,5})['\"]?\s*(#.*)?$")
|
||||
PIP_VALUE_FLAGS = {"-c", "--constraint", "-i", "--index-url", "--extra-index-url", "-f",
|
||||
"--find-links", "--target", "-t", "--python", "--prefix", "--root", "--platform",
|
||||
"--python-version", "--implementation", "--abi", "--only-binary", "--no-binary"}
|
||||
|
||||
|
||||
def path_ok(path: str) -> bool:
|
||||
return bool(INCLUDE.search(path)) and not NEVER.search(path)
|
||||
|
||||
|
||||
def _commands(text: str) -> list[list[str]]:
|
||||
"""Split a shell line into simple commands (&&, ||, ;, |), tokenized."""
|
||||
out = []
|
||||
for part in re.split(r"&&|\|\||;|\|", text):
|
||||
try:
|
||||
toks = shlex.split(part, comments=True)
|
||||
except ValueError:
|
||||
toks = part.split()
|
||||
# Dockerfile RUN prefix / sudo / env-prefixed assignments
|
||||
while toks and (toks[0] in ("RUN", "sudo", "exec", "-", "run:", "command:")
|
||||
or re.match(r"^[A-Z_][A-Z0-9_]*=", toks[0])):
|
||||
toks = toks[1:]
|
||||
if toks:
|
||||
out.append(toks)
|
||||
return out
|
||||
|
||||
|
||||
def _pip_problem(args: list[str]) -> str | None:
|
||||
if "-r" in args or "--requirement" in args or any(a.startswith("--requirement=") for a in args):
|
||||
return None
|
||||
if "--no-deps" in args:
|
||||
return None
|
||||
pkgs, skip = [], False
|
||||
for a in args:
|
||||
if skip:
|
||||
skip = False
|
||||
continue
|
||||
if a in PIP_VALUE_FLAGS:
|
||||
skip = True
|
||||
continue
|
||||
if a.startswith("-") and a not in ("-e", "--editable"):
|
||||
continue
|
||||
if a in ("-e", "--editable"):
|
||||
continue
|
||||
pkgs.append(a)
|
||||
loose = [p for p in pkgs if not EXACT_PY.match(p) and p.split("[")[0].lower() not in TOOLING]
|
||||
if loose:
|
||||
return f"floating pip install: {' '.join(loose)[:40]}"
|
||||
return None
|
||||
|
||||
|
||||
def scan_line(path: str, text: str) -> list[tuple[str, str]]:
|
||||
if cg.COMMENT.match(text):
|
||||
return []
|
||||
hits = []
|
||||
if "/workflows/" in path and HOST_PORT.match(text):
|
||||
hits.append(("host port", f"service publishes host port {HOST_PORT.match(text).group(1)} (shared dind)"))
|
||||
return hits
|
||||
# Windy Mail #147: a `:latest` build/tool image floats exactly like an
|
||||
# unpinned package, and `COPY uv.lock* ./` builds WITHOUT the lock when it
|
||||
# is missing instead of failing.
|
||||
m = LATEST.search(text)
|
||||
if m:
|
||||
hits.append(("floating image", f"{m.group(1)}:latest"))
|
||||
if DOCKER_FILE.search(path) and re.match(r"^\s*COPY\b", text, re.I):
|
||||
globbed = [t for t in text.split() if "*" in t and LOCKNAME.search(t)]
|
||||
if globbed:
|
||||
hits.append(("optional lock", f"COPY {globbed[0]} (must fail if the lock is missing)"))
|
||||
# Windy Git jobs get NO Docker daemon (I-5), so a docker build/compose/run
|
||||
# step in CI can never pass here (orchestrator 09-23, option A). The real
|
||||
# image build is the deploy step on the target host.
|
||||
if "/workflows/" in path and re.search(r"uses:\s*['\"]?docker/build-push-action", text):
|
||||
hits.append(("needs docker", "docker/build-push-action in CI (no Docker daemon on Windy Git; " + NO_DOCKER_FIX + ")"))
|
||||
for toks in _commands(text):
|
||||
low = [t.lower() for t in toks]
|
||||
if "/workflows/" in path and (
|
||||
low[:2] in (["docker", "build"], ["docker", "buildx"], ["docker", "run"], ["docker", "compose"])
|
||||
or low[:1] == ["docker-compose"]
|
||||
):
|
||||
hits.append(("needs docker", f"{' '.join(low[:2])} in CI (no Docker daemon on Windy Git; " + NO_DOCKER_FIX + ")"))
|
||||
continue
|
||||
# pip install / python -m pip install / uv pip install
|
||||
for i in range(len(low) - 1):
|
||||
if os.path.basename(low[i]) in ("pip", "pip3") and low[i + 1] == "install":
|
||||
prob = _pip_problem(toks[i + 2:])
|
||||
if prob:
|
||||
hits.append(("floating install", prob))
|
||||
break
|
||||
if low[:2] == ["uv", "sync"] and not ({"--locked", "--frozen"} & set(low)):
|
||||
hits.append(("floating install", "uv sync without --locked/--frozen"))
|
||||
if low[:1] == ["npm"] and len(low) > 1 and low[1] in ("install", "i", "add"):
|
||||
pkgs = [t for t in toks[2:] if not t.startswith("-")]
|
||||
if not pkgs or not all(EXACT_NPM.match(p) for p in pkgs):
|
||||
hits.append(("floating install", f"npm {low[1]} {' '.join(pkgs)[:30]}".strip() + " (use npm ci)"))
|
||||
if low[:2] == ["yarn", "install"] and not ({"--frozen-lockfile", "--immutable"} & set(low)):
|
||||
hits.append(("floating install", "yarn install without --frozen-lockfile"))
|
||||
if low[:2] == ["pnpm", "install"] and "--frozen-lockfile" not in low:
|
||||
hits.append(("floating install", "pnpm install without --frozen-lockfile"))
|
||||
return hits
|
||||
|
||||
|
||||
_DISABLED: dict[str, set[str]] | None = None
|
||||
|
||||
|
||||
def disabled_workflows() -> dict[str, set[str]]:
|
||||
"""Workflow file names Gitea has DISABLED per repo (lowercased repo name).
|
||||
|
||||
Deploy/release workflows are disabled on Windy Git: they run on the target
|
||||
host, where a Docker daemon really exists, so "needs docker" must not flag
|
||||
them. One bounded query per process; on any failure nothing is excused
|
||||
(flag rather than hide).
|
||||
"""
|
||||
global _DISABLED
|
||||
if _DISABLED is not None:
|
||||
return _DISABLED
|
||||
_DISABLED = {}
|
||||
query = ("select coalesce(json_object_agg(r.lower_name, u.config::json->'DisabledWorkflows'), '{}'::json)"
|
||||
" from repo_unit u join repository r on r.id = u.repo_id"
|
||||
" where u.type = 10 and u.config like '%DisabledWorkflows%';")
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["docker", "exec", "-i", "windy-git-db-1", "sh", "-c",
|
||||
'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1'],
|
||||
input=query, capture_output=True, text=True, check=True, timeout=30,
|
||||
).stdout.strip()
|
||||
_DISABLED = {k: set(v or []) for k, v in json.loads(out or "{}").items()}
|
||||
except (subprocess.SubprocessError, OSError, ValueError):
|
||||
pass
|
||||
return _DISABLED
|
||||
|
||||
|
||||
def _runs_here(repo: str, findings):
|
||||
"""Drop "needs docker" hits in workflows that never run on Windy Git."""
|
||||
if findings is None:
|
||||
return None
|
||||
off = disabled_workflows().get(repo.lower(), set())
|
||||
return [f for f in findings if not (f.kind == "needs docker" and Path(f.path).name in off)]
|
||||
|
||||
|
||||
def check(repo: str, sha: str, default_branch: str, is_default_head: bool):
|
||||
return _runs_here(repo, _check(repo, sha, default_branch, is_default_head))
|
||||
|
||||
|
||||
def _check(repo: str, sha: str, default_branch: str, is_default_head: bool):
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
if not bare.is_dir() or not cg.fetched(bare, sha): # pushed after the fetch: next cycle
|
||||
return None
|
||||
allow = cg.load_allow(ALLOW_FILE)
|
||||
rules = hashlib.sha256((PREFILTER + INCLUDE.pattern + EXACT_PY.pattern + EXACT_NPM.pattern).encode()).hexdigest()[:8]
|
||||
fp = cg._fingerprint(allow) + ":" + rules # hashlib, not hash(): hash() is per-process random
|
||||
kw = dict(line_fn=scan_line, path_ok=path_ok)
|
||||
if is_default_head:
|
||||
return cg.cached_scan(f"hyg-tree:{repo}:{sha}:{fp}",
|
||||
lambda: cg.scan_tree(repo, bare, sha, allow, prefilter=PREFILTER, **kw))
|
||||
return cg.cached_scan(f"hyg-pr:{repo}:{sha}:{fp}",
|
||||
lambda: cg.scan_added(repo, bare, f"refs/heads/{default_branch}", sha, allow, **kw))
|
||||
|
||||
|
||||
def status_for(findings, whole_tree: bool, grant=()):
|
||||
"""Same contract as compute_guard.status_for: `grant` findings never block."""
|
||||
scope = "in CI/Dockerfiles" if whole_tree else "added"
|
||||
if not findings and grant:
|
||||
g, n = grant[0], len(grant)
|
||||
desc = f"⚠ WARN (Grant-owned, not blocking): {n} CI hygiene issue{'s' if n > 1 else ''} {scope}, e.g. {g.path}:{g.line} {g.match}"
|
||||
return "success", desc[:140], g
|
||||
if not findings:
|
||||
return "success", f"OK: no floating install or host-port service {scope}", None
|
||||
f = findings[0]
|
||||
n = len(findings)
|
||||
state = "failure" if MODE == "block" else "success"
|
||||
lead = "BLOCKED" if MODE == "block" else "⚠ WARN (not blocking)"
|
||||
return state, f"{lead}: {n} CI hygiene issue{'s' if n > 1 else ''} {scope}, e.g. {f.path}:{f.line} {f.match}"[:140], f
|
||||
|
||||
|
||||
def report(repos: list[str]) -> int:
|
||||
allow = cg.load_allow(ALLOW_FILE)
|
||||
total = 0
|
||||
for repo in repos:
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
if not bare.is_dir():
|
||||
print(f"## {repo}: no sync clone, skipped")
|
||||
continue
|
||||
head = cg._git(bare, "symbolic-ref", "--short", "HEAD").strip()
|
||||
sha = cg._git(bare, "rev-parse", head).strip()
|
||||
fs = _runs_here(repo, cg.scan_tree(repo, bare, sha, allow, line_fn=scan_line, path_ok=path_ok, prefilter=PREFILTER))
|
||||
total += len(fs)
|
||||
print(f"## {repo} ({head} {sha[:7]}): {len(fs)} issue(s)")
|
||||
for f in fs:
|
||||
print(f" {f.path}:{f.line} [{f.kind}] {f.match}")
|
||||
print(f"TOTAL {total}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) >= 2 and sys.argv[1] == "report":
|
||||
default = os.environ.get("BRIDGE_REPOS", "").split() or sorted(
|
||||
p.name.removesuffix(".git") for p in cg.WORK.glob("*.git"))
|
||||
sys.exit(report(sys.argv[2:] or default))
|
||||
sys.exit(__doc__)
|
||||
323
scripts/compute_guard.py
Normal file
323
scripts/compute_guard.py
Normal file
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compute guard: Windy Mind is the ONLY door to AI compute (Grant, 2026-09-23).
|
||||
|
||||
Flags code that talks to an AI provider directly instead of through Windy Mind:
|
||||
a provider API host, a provider SDK import or dependency, or a raw provider key
|
||||
name. Direct calls skip Mind's metering, caps and live-model routing, and they
|
||||
spend whichever key happens to be lying around (the audit found Grant's personal
|
||||
Max OAuth token inside a platform container).
|
||||
|
||||
WARN-ONLY for now: the bridge posts `windy-git/compute-guard` as success with a
|
||||
"⚠ WARN" description, so nothing turns red. `COMPUTE_GUARD_MODE=block` flips
|
||||
findings to failure once the repos are clean (orchestrator's call).
|
||||
|
||||
- PR heads: only lines the PR ADDS (vs its merge-base with the default branch).
|
||||
- Default-branch head: the whole tree (the baseline, and what `report` prints).
|
||||
|
||||
Exceptions live in ONE file, ci/compute-guard-allow.yml, each with a reason.
|
||||
Tests, docs, lockfiles, vendored code and CI config are never scanned.
|
||||
Reads the sync's bare GitHub clones on Veron (no docker exec: IO-stall lesson).
|
||||
|
||||
python3 scripts/compute_guard.py report [repo ...] # whole-tree findings on each default branch
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ALLOW_FILE = Path(os.environ.get("COMPUTE_GUARD_ALLOW", ROOT / "ci" / "compute-guard-allow.yml"))
|
||||
WORK = Path(os.environ.get("SYNC_WORK", "/srv/windygit/sync"))
|
||||
CACHE = Path(os.environ.get("COMPUTE_GUARD_CACHE", "/var/lib/windy-git/compute-guard-cache.json"))
|
||||
MODE = os.environ.get("COMPUTE_GUARD_MODE", "warn") # warn | block
|
||||
|
||||
HOSTS = [
|
||||
"api.anthropic.com", "api.openai.com", "api.groq.com",
|
||||
"generativelanguage.googleapis.com", "api.mistral.ai", "api.perplexity.ai",
|
||||
"openrouter.ai", "api.together.xyz", "api.together.ai", "api.cerebras.ai",
|
||||
"api.sambanova.ai", "api.deepseek.com", "api.x.ai", "api.cohere.ai",
|
||||
"api.cohere.com", "api.fireworks.ai", "api.replicate.com",
|
||||
"api-inference.huggingface.co",
|
||||
]
|
||||
KEYS = [
|
||||
"ANTHROPIC_API_KEY", "ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_AUTH_TOKEN",
|
||||
"OPENAI_API_KEY", "GROQ_API_KEY", "GEMINI_API_KEY", "GOOGLE_GENERATIVE_AI_API_KEY",
|
||||
"GOOGLE_AI_API_KEY", "MISTRAL_API_KEY", "PERPLEXITY_API_KEY", "PPLX_API_KEY",
|
||||
"OPENROUTER_API_KEY", "TOGETHER_API_KEY", "CEREBRAS_API_KEY", "SAMBANOVA_API_KEY",
|
||||
"DEEPSEEK_API_KEY", "XAI_API_KEY", "COHERE_API_KEY", "FIREWORKS_API_KEY",
|
||||
"REPLICATE_API_TOKEN",
|
||||
]
|
||||
PY_SDKS = r"anthropic|openai|groq|mistralai|cohere|google\.generativeai|google\.genai|together|cerebras|litellm"
|
||||
JS_SDKS = (r"@anthropic-ai/sdk|openai|groq-sdk|@google/generative-ai|@google/genai|@mistralai/mistralai"
|
||||
r"|cohere-ai|together-ai|@ai-sdk/(?:anthropic|openai|groq|google|mistral)")
|
||||
|
||||
RULES: list[tuple[str, re.Pattern]] = [
|
||||
("provider host", re.compile("|".join(re.escape(h) for h in HOSTS))),
|
||||
# Grant via Boss 10-01: compute = Windy Mind. A NEW reference to an Ollama port (Veron's :11434) is a
|
||||
# direct call around Mind's metering/caps. WARN-only, never red, and only for lines a PR ADDS.
|
||||
("veron ollama", re.compile(r"(?::|%3[aA])11434(?![0-9])")),
|
||||
("provider key", re.compile(r"\b(?:" + "|".join(KEYS) + r")\b")),
|
||||
("provider SDK", re.compile(rf"^\s*(?:from|import)\s+(?:{PY_SDKS})(?:\s|\.|$|,)")),
|
||||
("provider SDK", re.compile(rf"""(?:from\s+|require\(\s*|import\(\s*)['"](?:{JS_SDKS})(?:/[^'"]*)?['"]""")),
|
||||
# dependency manifests: package.json keys, requirements / pyproject lines
|
||||
("provider SDK dep", re.compile(rf'''^\s*"(?:{JS_SDKS})"\s*:''')),
|
||||
("provider SDK dep", re.compile(rf'''^\s*["']?(?:{PY_SDKS.replace(chr(92) + ".", "-")})(?:\[[^\]]*\])?\s*(?:[<>=~!]=?|["',]|$)''')),
|
||||
]
|
||||
# Kinds that never block (even in MODE=block) and are only judged on ADDED lines, never the baseline tree.
|
||||
WARN_ONLY_KINDS = {"veron ollama"}
|
||||
OLLAMA_MSG = "compute = Windy Mind (endpoint + key); do not call Veron's Ollama directly"
|
||||
DEP_FILES = re.compile(r"(^|/)(package\.json|requirements[^/]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)$")
|
||||
|
||||
# Never scanned: tests, docs, lockfiles, vendored/built code, CI config.
|
||||
SKIP = re.compile(
|
||||
r"(^|/)(tests?|__tests__|spec|docs?|node_modules|vendor|dist|build|\.github|\.gitea)/"
|
||||
r"|(^|/)(test_[^/]*|[^/]*_test\.py|conftest\.py|[^/]*\.(test|spec)\.[cm]?[jt]sx?)$"
|
||||
r"|\.(md|mdx|rst|txt|lock|snap|svg|png|jpg|pdf)$"
|
||||
r"|(^|/)(package-lock\.json|pnpm-lock\.yaml|yarn\.lock|uv\.lock|poetry\.lock|Cargo\.lock)$"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Finding:
|
||||
path: str
|
||||
line: int
|
||||
kind: str
|
||||
match: str
|
||||
|
||||
|
||||
def load_allow(path: Path = ALLOW_FILE) -> list[dict]:
|
||||
data = yaml.safe_load(path.read_text()) or {}
|
||||
entries = data.get("allow") or []
|
||||
for e in entries: # a reason per entry is the whole point of the file
|
||||
if not (e.get("repo") and e.get("paths") and str(e.get("reason", "")).strip()):
|
||||
raise ValueError(f"allow entry needs repo, paths and a reason: {e}")
|
||||
return entries
|
||||
|
||||
|
||||
def allowed(repo: str, path: str, allow: list[dict], text: str | None = None) -> bool:
|
||||
"""An entry may carry `matches:` (regexes): then only lines matching one of
|
||||
them are allowed, so an allowed file can't smuggle in a NEW call (e.g. an
|
||||
OAuth sign-in endpoint is allowed, an inference endpoint in the same file
|
||||
still flags). Entries without `matches` cover the whole path."""
|
||||
for e in allow:
|
||||
if e["repo"] != repo or not any(fnmatch.fnmatch(path, g) for g in e["paths"]):
|
||||
continue
|
||||
pats = e.get("matches")
|
||||
if not pats:
|
||||
return True
|
||||
if text is not None and any(re.search(rx, text) for rx in pats):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
COMMENT = re.compile(r"^\s*(?:#|//|/\*|\*|<!--)")
|
||||
|
||||
|
||||
def scan_line(path: str, text: str) -> list[tuple[str, str]]:
|
||||
# A comment is not a call: "the ANTHROPIC_OAUTH_TOKEN setting was removed"
|
||||
# (windy-search) must not count, nor a commented-out `# OPENAI_API_KEY=`.
|
||||
if COMMENT.match(text):
|
||||
return []
|
||||
hits = []
|
||||
for kind, rx in RULES:
|
||||
if kind == "provider SDK dep" and not DEP_FILES.search(path):
|
||||
continue
|
||||
m = rx.search(text)
|
||||
if m:
|
||||
hits.append((kind, m.group(0).strip()[:60]))
|
||||
return hits
|
||||
|
||||
|
||||
def _git(bare: Path, *args: str) -> str:
|
||||
return subprocess.run(
|
||||
["git", "--git-dir", str(bare), *args],
|
||||
capture_output=True, text=True, check=True, timeout=120,
|
||||
).stdout
|
||||
|
||||
|
||||
def _default_path_ok(path: str) -> bool:
|
||||
return not SKIP.search(path)
|
||||
|
||||
|
||||
def scan_tree(repo: str, bare: Path, sha: str, allow: list[dict], *, line_fn=None,
|
||||
path_ok=None, prefilter: str | None = None) -> list[Finding]:
|
||||
"""Every line in the tree at `sha` (default branch: the baseline)."""
|
||||
# A cheap prefilter by git, then the real rules in Python.
|
||||
# Other guards (ci_hygiene) reuse this walker with their own line rules.
|
||||
line_fn = line_fn or scan_line
|
||||
path_ok = path_ok or _default_path_ok
|
||||
pre = prefilter or "|".join([re.escape(h) for h in HOSTS] + KEYS + [
|
||||
"anthropic", "openai", "groq", "mistral", "generativeai", "genai", "cohere",
|
||||
"together", "cerebras", "litellm"])
|
||||
try:
|
||||
out = _git(bare, "grep", "-nIE", "-e", pre, sha, "--", ".")
|
||||
except subprocess.CalledProcessError as e:
|
||||
if e.returncode == 1: # no matches
|
||||
return []
|
||||
raise
|
||||
found = []
|
||||
for raw in out.splitlines():
|
||||
# <sha>:<path>:<line>:<text>
|
||||
try:
|
||||
_, path, line, text = raw.split(":", 3)
|
||||
except ValueError:
|
||||
continue
|
||||
if not path_ok(path) or allowed(repo, path, allow, text):
|
||||
continue
|
||||
for kind, match in line_fn(path, text):
|
||||
found.append(Finding(path, int(line), kind, match))
|
||||
return found
|
||||
|
||||
|
||||
def scan_added(repo: str, bare: Path, base_ref: str, sha: str, allow: list[dict], **kw) -> list[Finding]:
|
||||
"""Only the lines a PR adds, vs its merge-base with the default branch."""
|
||||
mb = _git(bare, "merge-base", base_ref, sha).strip()
|
||||
diff = _git(bare, "diff", "-U0", "--no-color", "--no-ext-diff", mb, sha)
|
||||
return parse_added(repo, diff, allow, **kw)
|
||||
|
||||
|
||||
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")
|
||||
|
||||
|
||||
def parse_added(repo: str, diff: str, allow: list[dict], *, line_fn=None, path_ok=None) -> list[Finding]:
|
||||
line_fn = line_fn or scan_line
|
||||
path_ok = path_ok or _default_path_ok
|
||||
found, path, line = [], None, 0
|
||||
for raw in diff.splitlines():
|
||||
if raw.startswith("+++ "):
|
||||
p = raw[4:]
|
||||
path = None if p == "/dev/null" else p[2:] if p.startswith("b/") else p
|
||||
continue
|
||||
m = HUNK.match(raw)
|
||||
if m:
|
||||
line = int(m.group(1))
|
||||
continue
|
||||
if path is None or raw.startswith("--- "):
|
||||
continue
|
||||
if raw.startswith("+"):
|
||||
if path_ok(path) and not allowed(repo, path, allow, raw[1:]):
|
||||
for kind, match in line_fn(path, raw[1:]):
|
||||
found.append(Finding(path, line, kind, match))
|
||||
line += 1
|
||||
return found
|
||||
|
||||
|
||||
# ---- cache: a tree scan runs once per (repo, sha, rules+allow) --------------
|
||||
def _fingerprint(allow: list[dict]) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps([HOSTS, KEYS, PY_SDKS, JS_SDKS, SKIP.pattern, allow], sort_keys=True).encode()
|
||||
).hexdigest()[:16]
|
||||
|
||||
|
||||
def cached_scan(key: str, fn) -> list[Finding]:
|
||||
try:
|
||||
cache = json.loads(CACHE.read_text())
|
||||
except (OSError, ValueError):
|
||||
cache = {}
|
||||
if key in cache:
|
||||
return [Finding(**f) for f in cache[key]]
|
||||
result = fn()
|
||||
cache[key] = [f.__dict__ for f in result]
|
||||
if len(cache) > 2000: # keep it small: newest entries win
|
||||
cache = dict(list(cache.items())[-1000:])
|
||||
try:
|
||||
CACHE.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(cache))
|
||||
tmp.replace(CACHE)
|
||||
except OSError:
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def fetched(bare: Path, sha: str) -> bool:
|
||||
"""Is `sha` in the sync clone yet? The bridge learns PR / default heads from
|
||||
GitHub's API AFTER the sync fetched, so a push in between is simply not here
|
||||
until the next 5-min cycle. That is a race, not an error: skip quietly."""
|
||||
try:
|
||||
_git(bare, "cat-file", "-e", f"{sha}^{{commit}}")
|
||||
return True
|
||||
except subprocess.CalledProcessError:
|
||||
return False
|
||||
|
||||
|
||||
def check(repo: str, sha: str, default_branch: str, is_default_head: bool) -> list[Finding] | None:
|
||||
"""Findings for one commit, or None when the guard can't run (never a fake OK)."""
|
||||
bare = WORK / f"{repo}.git"
|
||||
if not bare.is_dir() or not fetched(bare, sha):
|
||||
return None
|
||||
allow = load_allow()
|
||||
fp = _fingerprint(allow)
|
||||
if is_default_head:
|
||||
return cached_scan(f"tree:{repo}:{sha}:{fp}",
|
||||
lambda: [f for f in scan_tree(repo, bare, sha, allow) if f.kind not in WARN_ONLY_KINDS])
|
||||
return cached_scan(
|
||||
f"pr:{repo}:{sha}:{fp}",
|
||||
lambda: scan_added(repo, bare, f"refs/heads/{default_branch}", sha, allow),
|
||||
)
|
||||
|
||||
|
||||
def status_for(findings: list[Finding], whole_tree: bool,
|
||||
grant: list[Finding] = ()) -> tuple[str, str, Finding | None]:
|
||||
"""(state, description, first finding) for the GitHub commit status.
|
||||
|
||||
`findings` = lane-owned (these block in MODE=block); `grant` = findings in
|
||||
Grant-owned code (ci/grant-owned.yml): always WARN, never red (orchestrator
|
||||
09-23: his desktop work is never blocked by us)."""
|
||||
scope = "in tree" if whole_tree else "added"
|
||||
soft = [f for f in findings if f.kind in WARN_ONLY_KINDS]
|
||||
findings = [f for f in findings if f.kind not in WARN_ONLY_KINDS]
|
||||
if not findings and not grant and soft:
|
||||
f = soft[0]
|
||||
return "success", f"⚠ WARN: new Veron Ollama ref {f.path}:{f.line}. {OLLAMA_MSG}"[:140], f
|
||||
if not findings and grant:
|
||||
g, n = grant[0], len(grant)
|
||||
desc = (f"⚠ WARN (Grant-owned, not blocking): {n} direct AI-provider use{'s' if n > 1 else ''} "
|
||||
f"{scope}, e.g. {g.path}:{g.line} {g.match}")
|
||||
return "success", desc[:140], g
|
||||
if not findings:
|
||||
what = "no direct AI-provider use in tree" if whole_tree else "no direct AI-provider use added"
|
||||
return "success", f"OK: {what} (Windy Mind is the only door)", None
|
||||
f = findings[0]
|
||||
n = len(findings)
|
||||
state = "failure" if MODE == "block" else "success"
|
||||
lead = "BLOCKED" if MODE == "block" else "⚠ WARN (not blocking)"
|
||||
desc = f"{lead}: {n} direct AI-provider use{'s' if n > 1 else ''} {scope}, e.g. {f.path}:{f.line} {f.match}"
|
||||
return state, desc[:140], f
|
||||
|
||||
|
||||
def report(repos: list[str]) -> int:
|
||||
allow = load_allow()
|
||||
total = 0
|
||||
for repo in repos:
|
||||
bare = WORK / f"{repo}.git"
|
||||
if not bare.is_dir():
|
||||
print(f"## {repo}: no sync clone, skipped")
|
||||
continue
|
||||
head = _git(bare, "symbolic-ref", "--short", "HEAD").strip()
|
||||
sha = _git(bare, "rev-parse", head).strip()
|
||||
fs = scan_tree(repo, bare, sha, allow)
|
||||
total += len(fs)
|
||||
print(f"## {repo} ({head} {sha[:7]}): {len(fs)} finding(s)")
|
||||
for f in fs:
|
||||
print(f" {f.path}:{f.line} [{f.kind}] {f.match}")
|
||||
print(f"TOTAL {total}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) >= 2 and sys.argv[1] == "report":
|
||||
default = os.environ.get("BRIDGE_REPOS", "").split() or sorted(
|
||||
p.name.removesuffix(".git") for p in WORK.glob("*.git"))
|
||||
sys.exit(report(sys.argv[2:] or default))
|
||||
sys.exit(__doc__)
|
||||
23
scripts/drill_cross_host.sh
Executable file
23
scripts/drill_cross_host.sh
Executable file
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Cross-host restore drill on Windy 0: ONLY lockbox + R2, nothing from Veron. Values never printed.
|
||||
# Usage: bash drill_cross_host.sh (needs lockbox keys RESTIC_WINDYGIT_PASSWORD, WINDYGIT_R2_ACCESS_KEY_ID, WINDYGIT_R2_SECRET_ACCESS_KEY, WINDYGIT_R2_ENDPOINT)
|
||||
set -euo pipefail
|
||||
umask 077; W=$(mktemp -d ~/.cache/wg-xdrill.XXXXXX)
|
||||
trap 'docker rm -f wg-xdrill-pg >/dev/null 2>&1 || true; rm -rf "$W"' EXIT
|
||||
lockbox-get RESTIC_WINDYGIT_PASSWORD "$W/pw" >/dev/null
|
||||
lockbox-get WINDYGIT_R2_ACCESS_KEY_ID "$W/ak" >/dev/null; lockbox-get WINDYGIT_R2_SECRET_ACCESS_KEY "$W/sk" >/dev/null; lockbox-get WINDYGIT_R2_ENDPOINT "$W/ep" >/dev/null
|
||||
export RESTIC_PASSWORD_FILE="$W/pw" AWS_ACCESS_KEY_ID="$(cat "$W/ak")" AWS_SECRET_ACCESS_KEY="$(cat "$W/sk")"
|
||||
export RESTIC_REPOSITORY="s3:$(cat "$W/ep")/windy-git-backups/restic"
|
||||
restic snapshots --tag windygit-state --compact | tail -3
|
||||
restic restore latest --tag windygit-state --target "$W/r" --include /var/backups/windygit-state --include /srv/windygit/git/gitea/conf --quiet
|
||||
ls -l "$W/r/var/backups/windygit-state" | awk 'NR>1{print $5, $NF}'
|
||||
docker run -d --name wg-xdrill-pg -e POSTGRES_PASSWORD="$(python3 -c 'import secrets;print(secrets.token_hex(12))')" -e POSTGRES_USER=drill postgres:16-alpine >/dev/null
|
||||
for i in $(seq 1 30); do docker exec wg-xdrill-pg pg_isready -U drill >/dev/null 2>&1 && break; sleep 2; done
|
||||
for db in gitea windygit; do
|
||||
docker exec wg-xdrill-pg psql -U drill -d postgres -qc "create database $db"
|
||||
docker exec -i wg-xdrill-pg pg_restore -U drill -d $db --no-owner --no-privileges < "$W/r/var/backups/windygit-state/$db.dump" 2>&1 | grep -v "already exists" | head -2 || true
|
||||
done
|
||||
for t in repository issue pull_request '"user"' external_login_user action_run action_run_job; do
|
||||
echo "$t restored=$(docker exec wg-xdrill-pg psql -U drill -d gitea -Atc "select count(*) from $t")"
|
||||
done
|
||||
echo "cross-host drill OK (cleaned up)"
|
||||
153
scripts/env_names.py
Normal file
153
scripts/env_names.py
Normal file
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""env-names: list environment variable NAMES only (Boss ruling 10-01, house rule 10).
|
||||
|
||||
env-names <docker container | systemd unit | env file> [--host H] [--sudo] [--hash]
|
||||
env-names A --compare B [--host H] [--host2 H2]
|
||||
|
||||
Prints NAME, set|empty, and value LENGTH. Never a value or fragment. --hash adds sha256[:8]
|
||||
(compare two places for equality; a hash of a weak value can be guessed, so use it for
|
||||
real secrets only). --compare prints SAME / DIFFERENT / only-in-A / only-in-B per name
|
||||
(equality by full-value hash, nothing else shown). Values live in memory only.
|
||||
Targets: an existing file (dotenv style) | a docker container name | a systemd unit
|
||||
(Environment= + EnvironmentFile=; --sudo to read root-only files). --host runs the docker /
|
||||
systemctl / file read over ssh (alias from ~/.ssh/config).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
KV = ("=",)
|
||||
|
||||
|
||||
def run(cmd: list[str], host: str | None, sudo: bool = False) -> tuple[int, str]:
|
||||
if sudo:
|
||||
cmd = ["sudo", "-n", *cmd]
|
||||
if host:
|
||||
cmd = ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", host, shlex.join(cmd)]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, errors="ignore", timeout=60)
|
||||
return r.returncode, r.stdout
|
||||
|
||||
|
||||
def parse_dotenv(text: str) -> dict[str, str]:
|
||||
out: dict[str, str] = {}
|
||||
for raw in text.splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
if line.startswith("export "):
|
||||
line = line[7:].lstrip()
|
||||
k, v = line.split("=", 1)
|
||||
k = k.strip()
|
||||
v = v.strip()
|
||||
if len(v) >= 2 and v[0] == v[-1] and v[0] in "\"'":
|
||||
v = v[1:-1]
|
||||
if k.replace("_", "").isalnum() and not k[0].isdigit():
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def from_file(path: str, host: str | None, sudo: bool) -> dict[str, str] | None:
|
||||
if host or sudo:
|
||||
rc, out = run(["cat", path], host, sudo)
|
||||
return parse_dotenv(out) if rc == 0 else None
|
||||
try:
|
||||
with open(path, errors="ignore") as fh:
|
||||
return parse_dotenv(fh.read())
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def from_docker(name: str, host: str | None, sudo: bool) -> dict[str, str] | None:
|
||||
rc, out = run(["docker", "inspect", "-f", "{{json .Config.Env}}", name], host, sudo)
|
||||
if rc != 0 or not out.strip():
|
||||
return None
|
||||
try:
|
||||
items = json.loads(out)
|
||||
except ValueError:
|
||||
return None
|
||||
return {k: v for k, _, v in (i.partition("=") for i in (items or []))}
|
||||
|
||||
|
||||
def from_systemd(unit: str, host: str | None, sudo: bool) -> dict[str, str] | None:
|
||||
rc, out = run(["systemctl", "show", unit, "-p", "Environment", "-p", "EnvironmentFiles"], host)
|
||||
if rc != 0 or "LoadState=not-found" in out:
|
||||
return None
|
||||
env: dict[str, str] = {}
|
||||
files: list[str] = []
|
||||
for line in out.splitlines():
|
||||
if line.startswith("Environment="):
|
||||
for tok in shlex.split(line[len("Environment="):]):
|
||||
k, _, v = tok.partition("=")
|
||||
env[k] = v
|
||||
elif line.startswith("EnvironmentFiles="):
|
||||
f = line[len("EnvironmentFiles="):].split(" (")[0].strip().lstrip("-")
|
||||
if f:
|
||||
files.append(f)
|
||||
for f in files: # later files override earlier, like systemd
|
||||
d = from_file(f, host, sudo)
|
||||
if d is None:
|
||||
print(f"# note: EnvironmentFile {f} unreadable (try --sudo)", file=sys.stderr)
|
||||
else:
|
||||
env.update(d)
|
||||
return env
|
||||
|
||||
|
||||
def load(target: str, host: str | None, sudo: bool) -> dict[str, str] | None:
|
||||
if (not host and os.path.isfile(target)) or target.startswith(("/", "./", "~")):
|
||||
return from_file(os.path.expanduser(target), host, sudo)
|
||||
if target.endswith((".service", ".timer", ".socket")):
|
||||
return from_systemd(target, host, sudo)
|
||||
return from_docker(target, host, sudo) or from_systemd(target, host, sudo)
|
||||
|
||||
|
||||
def sh(v: str) -> str:
|
||||
return hashlib.sha256(v.encode()).hexdigest()
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(prog="env-names", description="env var NAMES only")
|
||||
ap.add_argument("target")
|
||||
ap.add_argument("--host")
|
||||
ap.add_argument("--host2", help="ssh host for the --compare target")
|
||||
ap.add_argument("--sudo", action="store_true")
|
||||
ap.add_argument("--hash", action="store_true", help="add sha256[:8] per variable")
|
||||
ap.add_argument("--compare", metavar="TARGET2")
|
||||
a = ap.parse_args(argv)
|
||||
env = load(a.target, a.host, a.sudo)
|
||||
if env is None:
|
||||
print(f"error: could not read {a.target!r} (file, docker container or systemd unit)")
|
||||
return 2
|
||||
if a.compare:
|
||||
env2 = load(a.compare, a.host2 or a.host, a.sudo)
|
||||
if env2 is None:
|
||||
print(f"error: could not read {a.compare!r}")
|
||||
return 2
|
||||
for k in sorted(set(env) | set(env2)):
|
||||
if k not in env2:
|
||||
print(f"{k:<40} only-in-A")
|
||||
elif k not in env:
|
||||
print(f"{k:<40} only-in-B")
|
||||
else:
|
||||
print(f"{k:<40} {'SAME' if sh(env[k]) == sh(env2[k]) else 'DIFFERENT'}"
|
||||
f" (len {len(env[k])} vs {len(env2[k])})")
|
||||
return 0
|
||||
for k in sorted(env):
|
||||
v = env[k]
|
||||
extra = f" sha256:{sh(v)[:8]}" if a.hash and v else ""
|
||||
print(f"{k:<40} {'set ' if v else 'empty'} len={len(v)}{extra}")
|
||||
print(f"# {len(env)} variable(s); values never printed")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except Exception as e: # never a traceback
|
||||
print(f"error: {type(e).__name__}")
|
||||
sys.exit(2)
|
||||
161
scripts/guards_report.py
Normal file
161
scripts/guards_report.py
Normal file
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Live status of the repo guards (compute-guard + ci-hygiene + secret-guard) as one markdown page.
|
||||
|
||||
Scans every bridged repo's DEFAULT branch with both guards and renders what is
|
||||
left, per repo and owner lane. Findings in code Grant owns (ci/grant-owned.yml:
|
||||
windy-pro's desktop app and its build jobs) are listed in their OWN section and
|
||||
do not count against "ready to block": those are proposals for Grant, not a
|
||||
lane's fix (orchestrator, 09-23).
|
||||
|
||||
sudo python3 scripts/guards_report.py > GUARDS_STATUS.md
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import ci_hygiene as hy # noqa: E402
|
||||
import compute_guard as cg # noqa: E402
|
||||
import secret_guard as sgd # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OWNED = Path(os.environ.get("GRANT_OWNED", ROOT / "ci" / "grant-owned.yml"))
|
||||
REPOS = os.environ.get("BRIDGE_REPOS", "").split() or [
|
||||
"windy-chat", "windy-mail", "windy-calendar", "Windy-Clone", "WindyCloud", "windy-search",
|
||||
"windy-connect", "windy-drops", "windy-code-web", "windy-code", "windy-traveler",
|
||||
"windy-registry", "eternitas", "windy-translate", "windytranslate-site", "windytraveler-site",
|
||||
"windy-hand", "windy-cloud-sites", "windy-cloud-domains", "windy-cloud-vps", "windytalk",
|
||||
"windy-pro", "windy-mind", "windy-git", "windy-inbox"]
|
||||
# Owner lane per repo = the session name to message (orchestrator routing, 09-23 ~22:45Z:
|
||||
# hub/account-server/dashboard/site -> "Windy Hub"; windy-calendar only -> "Windy Calender";
|
||||
# windy-admin/telemetry -> "Windy Admin"). windy-pro here = its server/web side; the desktop
|
||||
# app is Grant-owned and listed in its own section below.
|
||||
OWNERS = {
|
||||
"windy-chat": "Windy Chat", "windy-mail": "Windy Mail", "windy-calendar": "Windy Calender",
|
||||
"Windy-Clone": "Windy Clone", "WindyCloud": "Windy Cloud", "windy-cloud-sites": "Windy Cloud",
|
||||
"windy-cloud-domains": "Windy Cloud", "windy-cloud-vps": "Windy Cloud", "windy-search": "Windy Search",
|
||||
"windy-connect": "Windy Connect", "windy-drops": "Windy Drops", "windy-registry": "Windy Drops",
|
||||
"windy-code-web": "Windy Code", "windy-code": "Windy Code", "windy-traveler": "Windy Traveler",
|
||||
"windytraveler-site": "Windy Traveler", "eternitas": "Eternitas", "windy-translate": "Windy Translate",
|
||||
"windytranslate-site": "Windy Translate", "windy-hand": "Windy Hand", "windytalk": "Windy Talk",
|
||||
"windy-pro": "Windy Hub", "windy-mind": "WIndy Mind", "windy-git": "Windy Git",
|
||||
"windy-inbox": "Windy Drops"} # Windy Inbox build lead (09-24)
|
||||
JOB = re.compile(r"^ ([A-Za-z0-9_-]+):\s*$")
|
||||
|
||||
|
||||
def job_of(text: str, line: int) -> str | None:
|
||||
"""The workflow job a line belongs to (2-space keys under `jobs:`)."""
|
||||
in_jobs, job = False, None
|
||||
for i, raw in enumerate(text.splitlines(), 1):
|
||||
if raw.startswith("jobs:"):
|
||||
in_jobs = True
|
||||
elif in_jobs and JOB.match(raw):
|
||||
job = JOB.match(raw).group(1)
|
||||
elif raw and not raw[0].isspace() and not raw.startswith("jobs:"):
|
||||
in_jobs = False
|
||||
if i == line:
|
||||
return job if in_jobs else None
|
||||
return None
|
||||
|
||||
|
||||
def grant_owned(repo: str, path: str, job: str | None, owned: list[dict]) -> bool:
|
||||
for e in owned:
|
||||
if e["repo"] != repo:
|
||||
continue
|
||||
if any(fnmatch.fnmatch(path, g) for g in e.get("paths") or []):
|
||||
return True
|
||||
if job and job in (e.get("jobs") or {}).get(path, []):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def split_grant(repo: str, sha: str, findings: list) -> tuple[list, list]:
|
||||
"""(lane-owned, Grant-owned) findings at `sha`, by ci/grant-owned.yml, with
|
||||
workflow lines attributed to their job exactly as the status page does."""
|
||||
owned = (yaml.safe_load(OWNED.read_text()) or {}).get("grant_owned") or []
|
||||
if not any(e["repo"] == repo for e in owned):
|
||||
return list(findings), []
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
texts: dict[str, str] = {}
|
||||
lane, grant = [], []
|
||||
for f in findings:
|
||||
job = None
|
||||
if "/workflows/" in f.path:
|
||||
if f.path not in texts:
|
||||
texts[f.path] = cg._git(bare, "show", f"{sha}:{f.path}")
|
||||
job = job_of(texts[f.path], f.line)
|
||||
(grant if grant_owned(repo, f.path, job, owned) else lane).append(f)
|
||||
return lane, grant
|
||||
|
||||
|
||||
def scan(repo: str, owned: list[dict]):
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
if not bare.is_dir():
|
||||
return None
|
||||
head = cg._git(bare, "symbolic-ref", "--short", "HEAD").strip()
|
||||
sha = cg._git(bare, "rev-parse", head).strip()
|
||||
out = {"sha": sha, "compute": [], "hygiene": [], "secrets": []}
|
||||
texts: dict[str, str] = {}
|
||||
for key, fs in (("compute", cg.check(repo, sha, head, True) or []),
|
||||
("hygiene", hy.check(repo, sha, head, True) or []),
|
||||
("secrets", sgd.check(repo, sha, head, True) or [])):
|
||||
for f in fs:
|
||||
job = None
|
||||
if "/workflows/" in f.path:
|
||||
if f.path not in texts:
|
||||
texts[f.path] = cg._git(bare, "show", f"{sha}:{f.path}")
|
||||
job = job_of(texts[f.path], f.line)
|
||||
out[key].append((f, job, grant_owned(repo, f.path, job, owned)))
|
||||
return out
|
||||
|
||||
|
||||
def render(results: dict) -> str:
|
||||
now = time.strftime("%Y-%m-%d %H:%MZ", time.gmtime())
|
||||
lane = {k: 0 for k in ("compute", "hygiene", "secrets")}
|
||||
grant = {k: 0 for k in ("compute", "hygiene", "secrets")}
|
||||
for r in results.values():
|
||||
for k in lane:
|
||||
lane[k] += sum(1 for _, _, g in r.get(k, []) if not g)
|
||||
grant[k] += sum(1 for _, _, g in r.get(k, []) if g)
|
||||
L = [f"# Repo guards: live status (generated {now}; windy-git scripts/guards_report.py)",
|
||||
"_Default branches only. WARN-only today; the orchestrator says \"block\" per guard when its LANE column is 0. "
|
||||
"Grant-owned code (ci/grant-owned.yml) is listed separately and never holds up a block._", "",
|
||||
"| Guard | Lane-owned findings | Grant-owned (proposals) | Ready to block? |", "|---|---|---|---|",
|
||||
f"| compute-guard (Mind is the only door) | {lane['compute']} | {grant['compute']} | {'✅ YES' if lane['compute'] == 0 else '❌ not yet'} |",
|
||||
f"| ci-hygiene (house rule 6) | {lane['hygiene']} | {grant['hygiene']} | {'✅ YES' if lane['hygiene'] == 0 else '❌ not yet'} |",
|
||||
f"| secret-guard (no credentials in repos; hash only) | {lane['secrets']} | {grant['secrets']} | {'✅ YES' if lane['secrets'] == 0 else '❌ not yet'} |",
|
||||
"", "## By repo (lane-owned)", "| Repo | owner | head | compute | hygiene | secrets | first items |", "|---|---|---|---|---|---|---|"]
|
||||
for repo, r in sorted(results.items()):
|
||||
c = [x for x in r["compute"] if not x[2]]
|
||||
h = [x for x in r["hygiene"] if not x[2]]
|
||||
s = [x for x in r.get("secrets", []) if not x[2]]
|
||||
items = "; ".join(f"`{f.path}:{f.line}` {f.match}" for f, _, _ in (s + c + h)[:3]) or "clean ✅"
|
||||
L.append(f"| {repo} | {OWNERS.get(repo, '?')} | {r['sha'][:7]} | {len(c)} | {len(h)} | {len(s)} | {items} |")
|
||||
L += ["", "## Grant-owned (windy-pro desktop app + its build jobs): proposals only, not blocking"]
|
||||
g = [(repo, f, job) for repo, r in sorted(results.items()) for k in ("compute", "hygiene", "secrets")
|
||||
for f, job, own in r.get(k, []) if own]
|
||||
L += [f"- {repo} `{f.path}:{f.line}`{f' (job {job})' if job else ''}: {f.match}" for repo, f, job in g] or ["- none"]
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
owned = (yaml.safe_load(OWNED.read_text()) or {}).get("grant_owned") or []
|
||||
results = {}
|
||||
for repo in REPOS:
|
||||
r = scan(repo, owned)
|
||||
if r is not None:
|
||||
results[repo] = r
|
||||
sys.stdout.write(render(results))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -43,7 +43,12 @@ import urllib.request
|
||||
#
|
||||
# Bulk import belongs on the host anyway: no hairpin through the edge, no
|
||||
# Cloudflare ~100s proxy ceiling (G4A.5) on a large clone. Run this on Veron 1.
|
||||
GITEA = os.environ.get("GITEA_BASE_URL", "http://localhost:3080")
|
||||
#
|
||||
# 🔴 Deliberately NOT `GITEA_BASE_URL`: the deploy `.env` sets that to
|
||||
# `http://gitea:3000` for the API container, and sourcing `.env` on the host
|
||||
# made this script die on DNS *after* a caller had already deleted the mirror it
|
||||
# was meant to replace (2026-09-23).
|
||||
GITEA = os.environ.get("IMPORT_GITEA_URL", "http://localhost:3080")
|
||||
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
|
||||
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
|
||||
GITHUB_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
|
||||
@@ -93,6 +98,55 @@ def _api(method: str, path: str, body: dict | None = None) -> tuple[int, dict]:
|
||||
return e.code, {"message": raw[:300]}
|
||||
|
||||
|
||||
def all_repo_names() -> list[str]:
|
||||
out = subprocess.run(
|
||||
["gh", "repo", "list", GITHUB_OWNER, "--limit", "300",
|
||||
"--json", "name,isArchived"],
|
||||
capture_output=True, text=True, check=True,
|
||||
)
|
||||
return sorted(r["name"] for r in json.loads(out.stdout) if not r["isArchived"])
|
||||
|
||||
|
||||
def import_everything_as_mirrors() -> int:
|
||||
"""Bulk DR copy: every repo, as a READ-ONLY pull mirror.
|
||||
|
||||
Mirrors are the right shape for the bulk, and the reason is safety rather
|
||||
than tidiness. Sampling 40 repos found **18 carrying deploy / release /
|
||||
publish workflows that trigger on `push:`** — roughly 63 across the account.
|
||||
Importing those as writable repos with Actions enabled would arm sixty-odd
|
||||
production deploy triggers on Veron 1, each of which would then have to be
|
||||
disarmed by hand.
|
||||
|
||||
A pull mirror cannot run Actions at all, so the bulk import carries **zero**
|
||||
deploy risk, and Gitea does the syncing itself with no script and no timer.
|
||||
What you get is a complete, current, second copy of the whole account.
|
||||
|
||||
Converting one to a writable CI repo is then a deliberate per-repo act:
|
||||
delete, re-import with `mirror=false`, review its workflows, disable the
|
||||
deploying ones. That is the moment to make that judgement — not in bulk,
|
||||
sixty times, by accident.
|
||||
"""
|
||||
names = all_repo_names()
|
||||
existing = 0
|
||||
done = 0
|
||||
failed = []
|
||||
print(f"{len(names)} active repos on GitHub. Importing missing ones as read-only mirrors.\n")
|
||||
for n in names:
|
||||
status, _ = _api("GET", f"/repos/{OWNER}/{n}")
|
||||
if status == 200:
|
||||
existing += 1
|
||||
continue
|
||||
if import_repo(n, mirror=True):
|
||||
done += 1
|
||||
else:
|
||||
failed.append(n)
|
||||
print(f"\n already present: {existing}")
|
||||
print(f" newly mirrored: {done}")
|
||||
if failed:
|
||||
print(f" FAILED ({len(failed)}): {', '.join(failed[:10])}")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
def list_candidates() -> None:
|
||||
"""Private repos whose CI cannot run on GitHub at all."""
|
||||
out = subprocess.run(
|
||||
@@ -158,23 +212,29 @@ def main() -> int:
|
||||
"--mirror", action="store_true",
|
||||
help="import read-only pull mirrors instead of writable repos. NOTE: mirrors CANNOT run CI.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--all-as-mirrors", action="store_true",
|
||||
help="bulk DR copy: every active repo on the account, read-only, no CI, zero deploy risk",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.list_candidates:
|
||||
list_candidates()
|
||||
return 0
|
||||
|
||||
if args.all_as_mirrors:
|
||||
return import_everything_as_mirrors()
|
||||
|
||||
targets = SAFE_ORDER if args.safe_batch else args.repos
|
||||
if not targets:
|
||||
ap.error("name a repo, or pass --safe-batch / --list-candidates")
|
||||
|
||||
if "windy-pro" in targets:
|
||||
sys.exit(
|
||||
"REFUSING windy-pro. Six checkouts exist, the build counter has forked "
|
||||
"three ways (main 12 / overnight 34 / wave-44 56), and two sessions "
|
||||
"recorded different HEADs hours apart. Resolve which is current and "
|
||||
"write it down BEFORE importing (G11.5)."
|
||||
)
|
||||
# G11.5 RESOLVED 2026-09-23 (lane 8c, ~/windy-orchestra/WINDYPRO_CHECKOUTS.md):
|
||||
# a read-only audit of all 14 windy-pro checkouts on 5 machines found GitHub
|
||||
# main is canonical (Kit 0 prod and Windy 0 sit exactly on it; the others are
|
||||
# stale, not divergent). Phase 1 keeps GitHub the source of truth anyway, so a
|
||||
# writable Windy Git copy is CI only. Its six deploy/release workflows must be
|
||||
# disabled on import — see docs/CUTOVER.md.
|
||||
|
||||
if args.mirror:
|
||||
print("mirror mode: repos will be read-only and will NOT run CI.\n")
|
||||
|
||||
14
scripts/install_secret_tools.sh
Executable file
14
scripts/install_secret_tools.sh
Executable file
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install the shared hash-only secret tools on THIS machine (Windy 0): secret-scan + env-names.
|
||||
# Source of truth is this repo (scripts/); re-run after a pull to update.
|
||||
set -euo pipefail
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
dest="$HOME/.local/share/secret-tools"
|
||||
mkdir -p "$dest" "$HOME/.local/bin"
|
||||
cp "$here/secret_shapes.py" "$here/secret_scan.py" "$here/env_names.py" "$here/lockbox_put.py" "$here/lockbox_names.py" "$dest/"
|
||||
for pair in "secret-scan:secret_scan.py" "env-names:env_names.py" "lockbox-put:lockbox_put.py" "lockbox-names:lockbox_names.py"; do
|
||||
n=${pair%%:*}; f=${pair##*:}
|
||||
printf '#!/usr/bin/env bash\nexec python3 "%s/%s" "$@"\n' "$dest" "$f" > "$HOME/.local/bin/$n"
|
||||
chmod 755 "$HOME/.local/bin/$n"
|
||||
done
|
||||
echo "installed secret-scan, env-names, lockbox-put and lockbox-names (shapes from secret_shapes.py, same as secret-guard)"
|
||||
134
scripts/lockbox_names.py
Normal file
134
scripts/lockbox_names.py
Normal file
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env python3
|
||||
"""lockbox-names: DISCOVER what the lockbox holds without reading it (Boss rule 10-01).
|
||||
|
||||
lockbox-names [REGEX] (case-insensitive; matches the section heading or the label)
|
||||
|
||||
Prints one row per entry: SECTION HEADING | LABEL | kind | resolvable
|
||||
kind env = `KEY=value` line md = `- **`KEY`**: `value`` line
|
||||
label = a bold prose label (`**Password (X):** ...`) file = secrets/**/*.env key
|
||||
resolvable yes = `lockbox-get LABEL FILE` returns exactly one value
|
||||
dup = defined with 2+ different values (lockbox-get refuses)
|
||||
no = a prose-only label, or not an exact key
|
||||
|
||||
NEVER prints a value or any prose after a label. A label or heading that itself looks
|
||||
like a secret (secret_shapes) is replaced by <secret-shaped>. Reads the COMMITTED lockbox at
|
||||
origin/main (like lockbox-get; LOCKBOX_REF=<ref> or WORKTREE overrides). Memory only, stdout only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import secret_shapes as ss # noqa: E402
|
||||
|
||||
REPO = os.environ.get("LOCKBOX_REPO", os.path.expanduser("~/kit-army-config"))
|
||||
REF = os.environ.get("LOCKBOX_REF", "origin/main")
|
||||
HEAD = re.compile(r"^#{1,6}\s+(.*\S)\s*$")
|
||||
ENV = re.compile(r"^([A-Z][A-Z0-9_]{2,})=(.*)$")
|
||||
MD = re.compile(r"^\s*[-*]?\s*\*\*`([A-Za-z0-9_]+)`\*\*\s*:\s*`([^`]+)`")
|
||||
LABEL = re.compile(r"\*\*([^*`]{2,70}?)\*\*")
|
||||
|
||||
|
||||
def git(*a: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["git", "-C", REPO, *a], capture_output=True, text=True, errors="ignore")
|
||||
|
||||
|
||||
def read_sources() -> dict[str, str]:
|
||||
"""{path: text} for ACCESS_LOCKBOX.md and secrets/**/*.env."""
|
||||
if REF == "WORKTREE":
|
||||
out = {}
|
||||
for p in [Path(REPO, "ACCESS_LOCKBOX.md"), *Path(REPO, "secrets").rglob("*.env")]:
|
||||
if p.is_file():
|
||||
out[str(p.relative_to(REPO))] = p.read_text(errors="ignore")
|
||||
return out
|
||||
if REF.startswith("origin/"):
|
||||
git("fetch", "-q", "origin", REF.split("/", 1)[1])
|
||||
names = ["ACCESS_LOCKBOX.md"] + [
|
||||
p for p in git("ls-tree", "-r", "--name-only", REF, "--", "secrets").stdout.splitlines() if p.endswith(".env")]
|
||||
out = {}
|
||||
for n in names:
|
||||
r = git("show", f"{REF}:{n}")
|
||||
if r.returncode == 0:
|
||||
out[n] = r.stdout
|
||||
return out
|
||||
|
||||
|
||||
def safe(text: str, limit: int = 70) -> str:
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
return "<secret-shaped>" if ss.find(text) else text[:limit]
|
||||
|
||||
|
||||
def h(v: str) -> str:
|
||||
return hashlib.sha256(v.strip().strip('"').strip("'").encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def collect(src: dict[str, str]):
|
||||
"""(rows, values) where values[KEY] = {hash,...} for resolvability; nothing printed from it."""
|
||||
rows, values = [], {}
|
||||
for path, text in src.items():
|
||||
section = path
|
||||
for line in text.splitlines():
|
||||
m = HEAD.match(line) if path.endswith(".md") else None
|
||||
if m:
|
||||
section = safe(m.group(1), 90)
|
||||
continue
|
||||
m = ENV.match(line)
|
||||
if m:
|
||||
values.setdefault(m.group(1), set()).add(h(m.group(2)))
|
||||
rows.append((section, m.group(1), "file" if path.startswith("secrets/") else "env"))
|
||||
continue
|
||||
m = MD.match(line)
|
||||
if m:
|
||||
values.setdefault(m.group(1), set()).add(h(m.group(2)))
|
||||
rows.append((section, m.group(1), "md"))
|
||||
continue
|
||||
if path.endswith(".md"):
|
||||
mm = LABEL.search(line)
|
||||
if mm and not mm.group(1).startswith("http"):
|
||||
rows.append((section, safe(mm.group(1)), "label"))
|
||||
return rows, values
|
||||
|
||||
|
||||
def resolvable(label: str, kind: str, values) -> str:
|
||||
if kind not in ("env", "md", "file"):
|
||||
return "no"
|
||||
n = len(values.get(label, ()))
|
||||
return "yes" if n == 1 else "dup" if n > 1 else "no"
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
argv = list(sys.argv[1:] if argv is None else argv)
|
||||
rx = re.compile(argv[0], re.I) if argv else None
|
||||
src = read_sources()
|
||||
if not src:
|
||||
print("lockbox-names: cannot read the lockbox")
|
||||
return 2
|
||||
rows, values = collect(src)
|
||||
seen, n = set(), 0
|
||||
for section, label, kind in rows:
|
||||
if rx and not (rx.search(section) or rx.search(label)):
|
||||
continue
|
||||
key = (section, label, kind)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
n += 1
|
||||
print(f"{section} | {label} | {kind} | {resolvable(label, kind, values)}")
|
||||
print(f"# {n} entr{'y' if n == 1 else 'ies'}; names only, values never printed")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except re.error:
|
||||
print("lockbox-names: bad regex")
|
||||
sys.exit(2)
|
||||
except Exception as e: # never a traceback
|
||||
print(f"lockbox-names: error: {type(e).__name__}")
|
||||
sys.exit(2)
|
||||
141
scripts/lockbox_put.py
Normal file
141
scripts/lockbox_put.py
Normal file
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""lockbox-put: add ONE secret to the lockbox by PR, without anyone reading, printing or
|
||||
grepping the lockbox (Boss rule 10-01; Windy Hub ruling).
|
||||
|
||||
lockbox-put KEY FILE [--lane NAME] [--note TEXT]
|
||||
|
||||
KEY exact name, ^[A-Z][A-Z0-9_]{2,63}$ . FILE a 0600 file you own (not a symlink) whose
|
||||
content is the value (one line). Appends ONE line `- **`KEY`**: `<value>`` (the format
|
||||
lockbox-get reads) under a new heading at the END of ACCESS_LOCKBOX.md in a fresh temp clone,
|
||||
on a new branch, and opens a kit-army-config PR. Append-only: the diff is verified to be one
|
||||
file, additions only, before pushing. REFUSES if KEY already exists (a bool computed in
|
||||
memory; no line, value or location is ever printed). Reviewers see the KEY NAME + lane only
|
||||
if they look at the diff; the PR body never carries the value. Never echoes the value.
|
||||
Env (tests): LOCKBOX_PUT_REPO=<clone url/path>, LOCKBOX_PUT_NO_PR=1.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
REPO = os.environ.get("LOCKBOX_PUT_REPO", "https://github.com/sneakyfree/kit-army-config.git")
|
||||
SLUG = "sneakyfree/kit-army-config"
|
||||
KEY_RE = re.compile(r"^[A-Z][A-Z0-9_]{2,63}$")
|
||||
VAL_RE = re.compile(r"^[A-Za-z0-9._~+/=:@%,-]{8,512}$") # no backtick, quote, space or newline
|
||||
|
||||
|
||||
def die(msg: str, code: int = 2):
|
||||
print(f"lockbox-put: {msg}")
|
||||
sys.exit(code)
|
||||
|
||||
|
||||
def git(cwd: str, *a: str, quiet=True) -> subprocess.CompletedProcess:
|
||||
# stderr is dropped: git/gh errors can echo URLs; stdout only when we need it.
|
||||
return subprocess.run(["git", "-C", cwd, *a], capture_output=True, text=True, errors="ignore")
|
||||
|
||||
|
||||
def key_exists(clone: str, key: str) -> bool:
|
||||
"""True if KEY is already defined anywhere lockbox-get reads. Bool only, nothing printed."""
|
||||
pat_env = re.compile(r"^" + re.escape(key) + r"=")
|
||||
pat_md = re.compile(r"^\s*[-*]?\s*\*\*`" + re.escape(key) + r"`\*\*\s*:")
|
||||
paths = [Path(clone, "ACCESS_LOCKBOX.md")] + [
|
||||
Path(dp, f) for dp, _d, fs in os.walk(Path(clone, "secrets")) for f in fs if f.endswith(".env")]
|
||||
for p in paths:
|
||||
try:
|
||||
with open(p, errors="ignore") as fh:
|
||||
for line in fh:
|
||||
if pat_env.match(line) or pat_md.match(line):
|
||||
return True
|
||||
except OSError:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(prog="lockbox-put")
|
||||
ap.add_argument("key")
|
||||
ap.add_argument("file")
|
||||
ap.add_argument("--lane", default=os.environ.get("LOCKBOX_LANE", "a lane"))
|
||||
ap.add_argument("--note", default="")
|
||||
a = ap.parse_args(argv)
|
||||
if not KEY_RE.match(a.key):
|
||||
die("KEY must match ^[A-Z][A-Z0-9_]{2,63}$")
|
||||
try:
|
||||
st = os.lstat(a.file)
|
||||
except OSError:
|
||||
die("FILE not found")
|
||||
if stat.S_ISLNK(st.st_mode) or not stat.S_ISREG(st.st_mode):
|
||||
die("FILE must be a regular file (not a symlink)")
|
||||
if st.st_uid != os.getuid() or (st.st_mode & 0o077):
|
||||
die("FILE must be owned by you and mode 0600")
|
||||
with open(a.file) as fh:
|
||||
value = fh.read().strip()
|
||||
if not VAL_RE.match(value):
|
||||
die("value must be one line of 8-512 chars from [A-Za-z0-9._~+/=:@%,-] (no spaces, quotes, backticks)")
|
||||
note = re.sub(r"[`\n\r]", " ", a.note)[:160]
|
||||
lane = re.sub(r"[^A-Za-z0-9 ._-]", "", a.lane)[:40]
|
||||
tmp = tempfile.mkdtemp(prefix="lockbox-put-", dir=str(Path.home() / ".cache") if (Path.home() / ".cache").is_dir() else None)
|
||||
os.chmod(tmp, 0o700)
|
||||
clone = os.path.join(tmp, "k")
|
||||
try:
|
||||
if subprocess.run(["git", "clone", "-q", "--depth", "1", REPO, clone],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode != 0:
|
||||
die("clone failed (details withheld: URLs can carry tokens)")
|
||||
if key_exists(clone, a.key):
|
||||
die(f"{a.key} already exists: refusing to overwrite (append-only; pick a new KEY)", 3)
|
||||
lb = Path(clone, "ACCESS_LOCKBOX.md")
|
||||
if not lb.is_file():
|
||||
die("ACCESS_LOCKBOX.md not found in the repo")
|
||||
today = dt.date.today().isoformat()
|
||||
stamp = dt.datetime.now(dt.UTC).strftime("%Y%m%d%H%M")
|
||||
branch = f"lockbox-put/{a.key.lower()}-{stamp}"
|
||||
with open(lb, "a") as fh:
|
||||
fh.write(f"\n## 🗝️ {a.key} (added {today} by {lane} via lockbox-put)\n")
|
||||
fh.write(f"- **`{a.key}`**: `{value}`\n")
|
||||
if note:
|
||||
fh.write(f"- **Note:** {note}\n")
|
||||
git(clone, "checkout", "-q", "-b", branch)
|
||||
git(clone, "add", "ACCESS_LOCKBOX.md")
|
||||
ns = git(clone, "diff", "--cached", "--numstat").stdout.split()
|
||||
# numstat: <added> <deleted> <path>; exactly one file, no deletions
|
||||
if len(ns) != 3 or ns[1] != "0" or ns[2] != "ACCESS_LOCKBOX.md":
|
||||
die("diff is not a pure append to ACCESS_LOCKBOX.md: aborting, nothing pushed")
|
||||
msg = f"lockbox: add {a.key} (via lockbox-put, {lane})\n\nCo-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>"
|
||||
if git(clone, "-c", "user.name=lockbox-put", "-c", "user.email=lockbox-put@windy.invalid",
|
||||
"commit", "-q", "-m", msg).returncode != 0:
|
||||
die("commit failed")
|
||||
if git(clone, "push", "-q", "origin", branch).returncode != 0:
|
||||
die("push failed (details withheld)")
|
||||
if os.environ.get("LOCKBOX_PUT_NO_PR"):
|
||||
print(f"ok: pushed branch {branch} ({a.key}); PR skipped")
|
||||
return 0
|
||||
body = (f"Adds exactly one key: `{a.key}` (by {lane}). Append-only, one file, no deletions "
|
||||
f"(verified before push). Review by KEY NAME only; do not paste the value anywhere.\n\n"
|
||||
f"{note}\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)")
|
||||
r = subprocess.run(["gh", "pr", "create", "-R", SLUG, "--head", branch, "--base", "main",
|
||||
"--title", f"lockbox: add {a.key} ({lane})", "--body", body],
|
||||
capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
die("branch pushed but `gh pr create` failed; open the PR for the branch by hand")
|
||||
print(f"ok: {a.key} added via PR {r.stdout.strip().splitlines()[-1]}")
|
||||
return 0
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception as e: # never a traceback: it could carry data
|
||||
print(f"lockbox-put: error: {type(e).__name__}")
|
||||
sys.exit(2)
|
||||
455
scripts/pr_status_bridge.py
Executable file
455
scripts/pr_status_bridge.py
Executable file
@@ -0,0 +1,455 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Give private GitHub repos a CI signal from Windy Git (P1, 2026-09-23).
|
||||
|
||||
GitHub Actions cannot run on private repos on this account — not even on
|
||||
self-hosted runners ([[reference-github-actions-billing-lock]]). The code is
|
||||
already synced into Windy Git every 15 min and CI runs there, so the only thing
|
||||
missing is the *signal on GitHub*, where people and agents actually read PRs.
|
||||
|
||||
Two jobs, run after every sync:
|
||||
|
||||
1. **Mirror open PRs.** The sync carries branches, not PRs, and the workflows
|
||||
trigger on `pull_request` — so a PR branch alone fires nothing. For each open
|
||||
same-repo GitHub PR we keep one open Windy Git PR with the same head/base.
|
||||
Gitea then fires `pull_request` on open and `synchronize` whenever the sync
|
||||
moves the branch. Windy Git PRs whose GitHub PR closed are closed here too.
|
||||
Fork PRs are ignored: their head branch is never synced, and untrusted fork
|
||||
code on this runner is exactly the blast radius the audit warned about.
|
||||
|
||||
2. **Post results back** as GitHub commit statuses (context
|
||||
`windy-git/<workflow>/<job>`) on each PR head and on the default-branch head.
|
||||
Only posts when a context's state changed, so a 15-min loop doesn't pile
|
||||
hundreds of identical statuses onto one commit.
|
||||
|
||||
Runs ON Veron 1 (localhost Gitea; no Cloudflare hairpin). Needs
|
||||
GITEA_ADMIN_TOKEN and a GITHUB_TOKEN with `repo` scope. Nothing here executes
|
||||
repo code, and no secret is handed to any repo.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import yaml
|
||||
|
||||
GITEA = os.environ.get("BRIDGE_GITEA_URL", "http://localhost:3080").rstrip("/")
|
||||
PUBLIC = "https://app.windygit.com"
|
||||
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
|
||||
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
|
||||
GH_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
|
||||
WG_OWNER = os.environ.get("WINDYGIT_OWNER", "windyadmin")
|
||||
# Private repos only. Public repos run real GitHub Actions on veron1's GitHub
|
||||
# runner; bridging those too would put two competing verdicts on every commit.
|
||||
REPOS = os.environ.get(
|
||||
"BRIDGE_REPOS",
|
||||
"windy-chat windy-mail windy-calendar Windy-Clone WindyCloud windy-search windy-connect"
|
||||
" windy-drops windy-code-web windy-code windy-traveler windy-registry eternitas"
|
||||
" windy-translate windytranslate-site windytraveler-site windy-hand"
|
||||
" windy-cloud-sites windy-cloud-domains windy-cloud-vps windytalk windy-pro windy-mind"
|
||||
" windy-inbox windy-text windy-call windy-cell windy-hand-site windy-calendar-site",
|
||||
).split()
|
||||
|
||||
# Gitea run status -> GitHub status state. `skipped` is deliberately absent: a
|
||||
# job skipped by its own `if:` (e.g. substrate-drift's no-secrets path) has no
|
||||
# verdict, and painting it green would be a claim nobody tested.
|
||||
STATE = {
|
||||
"success": "success",
|
||||
"failure": "failure",
|
||||
"cancelled": "error",
|
||||
"running": "pending",
|
||||
"waiting": "pending",
|
||||
"blocked": "pending",
|
||||
}
|
||||
MIRROR_TAG = "[GH#"
|
||||
|
||||
# Image-build jobs cannot pass here BY DESIGN: job containers get no Docker
|
||||
# daemon (I-5 — the host socket would hand every workflow root on Veron 1).
|
||||
# Posting them would put a permanent red X on every commit, and a signal that is
|
||||
# always red trains everyone to ignore red. Not posted until a rootless builder
|
||||
# exists; that is a decision, recorded in docs/CUTOVER.md, not a failure.
|
||||
# ...but NOT the no-Docker smoke jobs that replaced them (option A, 09-23):
|
||||
# windy-search's "Boot smoke (no Docker)" matched plain `docker` and was hidden.
|
||||
NO_DAEMON_JOB = re.compile(r"(?<!no )(?<!no-)(?<!without )docker", re.IGNORECASE)
|
||||
# Image-build jobs whose NAME doesn't say docker (orchestrator 09-23, option A:
|
||||
# each lane converts the job to a no-Docker smoke test; until then it is not
|
||||
# posted). Format: "repo:workflow/job,...;repo2:...".
|
||||
NO_DAEMON_NAMED: dict[str, set[str]] = {}
|
||||
# eternitas:ci/build dropped 09-23: converted to a no-Docker ci/smoke (#179).
|
||||
for _entry in os.environ.get("BRIDGE_NO_DAEMON", "").split(";"):
|
||||
if ":" in _entry:
|
||||
_repo, _jobs = _entry.split(":", 1)
|
||||
NO_DAEMON_NAMED[_repo.strip()] = {j.strip() for j in _jobs.split(",") if j.strip()}
|
||||
|
||||
|
||||
def needs_daemon(repo: str, wf: str, job: str) -> bool:
|
||||
"""True for image-build jobs, which cannot run here (no Docker daemon, I-5)."""
|
||||
return bool(NO_DAEMON_JOB.search(job)) or f"{wf}/{job}" in NO_DAEMON_NAMED.get(repo, ())
|
||||
|
||||
# Jobs Grant ruled NON-BLOCKING (GRANT_DECISIONS_2026-09-23): still run on
|
||||
# Windy Git and visible there, but not posted to GitHub, so they cannot turn a
|
||||
# commit's combined status red. Format: "repo:workflow/job,workflow/job;repo2:..."
|
||||
# windy-pro's desktop/installer jobs belong to Grant's desktop side (fixed from
|
||||
# his Mac mini), not to any lane's merge gate.
|
||||
NON_BLOCKING: dict[str, set[str]] = {}
|
||||
for _entry in os.environ.get(
|
||||
"BRIDGE_NON_BLOCKING", "windy-pro:ci/build-desktop,ci/test-installer,ci/reality-check"
|
||||
).split(";"):
|
||||
if ":" in _entry:
|
||||
_repo, _jobs = _entry.split(":", 1)
|
||||
NON_BLOCKING[_repo.strip()] = {j.strip() for j in _jobs.split(",") if j.strip()}
|
||||
|
||||
|
||||
# Gitea reads the FIRST of these dirs that has workflow files at a commit (1.24).
|
||||
WORKFLOW_DIRS = (".gitea/workflows", ".github/workflows")
|
||||
|
||||
|
||||
def workflow_problem(text: str) -> str | None:
|
||||
"""Why Gitea would drop this workflow file, or None if it looks runnable.
|
||||
|
||||
Gitea skips an invalid workflow with one log line and fires no run at all,
|
||||
so on GitHub the PR just shows nothing, and people wait for CI that is never
|
||||
coming. These are the shapes we have actually hit, not a full schema.
|
||||
"""
|
||||
try:
|
||||
doc = yaml.safe_load(text)
|
||||
except yaml.YAMLError as e:
|
||||
mark = getattr(e, "problem_mark", None)
|
||||
return f"invalid YAML at line {mark.line + 1}" if mark else "invalid YAML"
|
||||
if not isinstance(doc, dict):
|
||||
return "not a YAML mapping"
|
||||
if "on" not in doc and True not in doc: # YAML 1.1 reads a bare `on` as True
|
||||
return "no `on:` trigger"
|
||||
jobs = doc.get("jobs")
|
||||
if not isinstance(jobs, dict) or not jobs:
|
||||
return "no `jobs:`"
|
||||
for name, job in jobs.items():
|
||||
if not isinstance(job, dict):
|
||||
return f"job `{name}` is not a mapping"
|
||||
if "runs-on" not in job and "uses" not in job:
|
||||
return f"job `{name}` has no `runs-on:`"
|
||||
return None
|
||||
|
||||
|
||||
def invalid_workflows(repo: str, sha: str) -> dict[str, tuple[str, str]]:
|
||||
"""{context: (path, problem)} for each workflow file at `sha` that won't run."""
|
||||
for d in WORKFLOW_DIRS:
|
||||
st, entries = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{d}?ref={sha}")
|
||||
if st == 404:
|
||||
continue
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git {d}@{sha[:7]} -> {st}")
|
||||
files = [e for e in entries or [] if e.get("type") == "file"
|
||||
and e["name"].endswith((".yml", ".yaml"))]
|
||||
if not files:
|
||||
continue
|
||||
bad = {}
|
||||
for e in files:
|
||||
st, f = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{e['path']}?ref={sha}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git {e['path']}@{sha[:7]} -> {st}")
|
||||
problem = workflow_problem(base64.b64decode(f["content"]).decode("utf-8", "replace"))
|
||||
if problem:
|
||||
stem = re.sub(r"\.ya?ml$", "", e["name"])
|
||||
bad[f"windy-git/{stem}/workflow"] = (e["path"], problem)
|
||||
return bad
|
||||
return {}
|
||||
|
||||
|
||||
def _call(base: str, token_header: str, method: str, path: str, body=None):
|
||||
req = urllib.request.Request(
|
||||
base + path,
|
||||
data=json.dumps(body).encode() if body is not None else None,
|
||||
method=method,
|
||||
headers={
|
||||
"Authorization": token_header,
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
# urllib's default UA is 403'd as a bot by GitHub's edge and CF.
|
||||
"User-Agent": "windy-git-pr-bridge/1",
|
||||
},
|
||||
)
|
||||
# Transport errors (TLS handshake timeout, reset) are retried: one GitHub
|
||||
# blip used to fail the whole sync, flip its heartbeat to ok:false and page
|
||||
# someone for nothing. HTTP errors are answers, not blips — never retried.
|
||||
for attempt in range(3):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
raw = r.read()
|
||||
return r.status, (json.loads(raw) if raw else None)
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, None
|
||||
except (urllib.error.URLError, TimeoutError, ConnectionError):
|
||||
if attempt == 2:
|
||||
raise
|
||||
time.sleep(2 * (attempt + 1))
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def gitea(method, path, body=None):
|
||||
return _call(GITEA + "/api/v1", f"token {GITEA_TOKEN}", method, path, body)
|
||||
|
||||
|
||||
def github(method, path, body=None):
|
||||
return _call("https://api.github.com", f"Bearer {GITHUB_TOKEN}", method, path, body)
|
||||
|
||||
|
||||
def sync_prs(repo: str) -> list[str]:
|
||||
"""Mirror open same-repo GitHub PRs into Windy Git. Returns their head shas."""
|
||||
st, gh_prs = github("GET", f"/repos/{GH_OWNER}/{repo}/pulls?state=open&per_page=100")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: GitHub PR list -> {st}")
|
||||
st, wg_prs = gitea("GET", f"/repos/{WG_OWNER}/{repo}/pulls?state=open&limit=50")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git PR list -> {st}")
|
||||
ours = {p["title"].split("]")[0] + "]": p for p in wg_prs if p["title"].startswith(MIRROR_TAG)}
|
||||
|
||||
heads, wanted = [], set()
|
||||
for pr in gh_prs:
|
||||
if pr["head"]["repo"] is None or pr["head"]["repo"]["full_name"] != f"{GH_OWNER}/{repo}":
|
||||
continue # fork PR — never synced, never run here
|
||||
tag = f"{MIRROR_TAG}{pr['number']}]"
|
||||
wanted.add(tag)
|
||||
heads.append(pr["head"]["sha"])
|
||||
if tag in ours:
|
||||
cur = (ours[tag].get("base") or {}).get("ref")
|
||||
if cur is None or cur == pr["base"]["ref"]: # unknown base: never guess, leave it
|
||||
continue
|
||||
# Retargeted on GitHub (e.g. a stacked PR moved to main after its
|
||||
# parent merged). The mirror kept the OLD base, so workflows filtered
|
||||
# on the base (`pull_request: branches: [main]`) silently stopped
|
||||
# running: eternitas #167, 09-23. Replace the mirror: an "edited"
|
||||
# event triggers nothing, a freshly opened PR runs CI at once.
|
||||
gitea("PATCH", f"/repos/{WG_OWNER}/{repo}/pulls/{ours[tag]['number']}", {"state": "closed"})
|
||||
print(f" {repo}: GH#{pr['number']} retargeted {cur} -> "
|
||||
f"{pr['base']['ref']}: replacing its mirror PR")
|
||||
st, _ = gitea(
|
||||
"POST",
|
||||
f"/repos/{WG_OWNER}/{repo}/pulls",
|
||||
{
|
||||
"head": pr["head"]["ref"],
|
||||
"base": pr["base"]["ref"],
|
||||
"title": f"{tag} {pr['title']}"[:250],
|
||||
"body": f"Mirror of {pr['html_url']} so CI runs here. Do not merge in Windy Git — "
|
||||
"GitHub is the source of truth; merge there.",
|
||||
},
|
||||
)
|
||||
print(f" {repo}: opened mirror PR for GH#{pr['number']} -> {st}")
|
||||
|
||||
for tag, p in ours.items():
|
||||
if tag not in wanted:
|
||||
gitea("PATCH", f"/repos/{WG_OWNER}/{repo}/pulls/{p['number']}", {"state": "closed"})
|
||||
print(f" {repo}: closed mirror PR {tag} (closed on GitHub)")
|
||||
return heads
|
||||
|
||||
|
||||
SAFE_NAME = re.compile(r"^[A-Za-z0-9._-]+$")
|
||||
SAFE_SHA = re.compile(r"^[0-9a-f]{40}$")
|
||||
|
||||
|
||||
def queued_jobs(repo: str, sha: str) -> list[dict]:
|
||||
"""Jobs at `sha` that are waiting for a runner and that a runner CAN take.
|
||||
|
||||
Gitea 1.24's API lists only PICKED-UP jobs (/actions/tasks), so a queued PR
|
||||
showed nothing on GitHub and people asked whether the push was lost. The
|
||||
truth is in the gitea DB. Bounded + non-fatal: during the 09-23 IO stall
|
||||
`docker exec` hung for an hour and must never wedge the bridge again.
|
||||
|
||||
Only status 5 (waiting) with labels some live runner has. A `pending` we
|
||||
post must end in a verdict we will also see, or it sits yellow forever:
|
||||
blocked jobs (7) often end SKIPPED, and jobs for labels no runner has
|
||||
(macos-/windows-/ubuntu-latest) are cancelled by the janitor unpicked;
|
||||
neither ever appears in /actions/tasks.
|
||||
"""
|
||||
if not (SAFE_NAME.match(repo) and SAFE_NAME.match(WG_OWNER) and SAFE_SHA.match(sha)):
|
||||
return []
|
||||
query = (
|
||||
"select json_build_object("
|
||||
" 'jobs', (select coalesce(json_agg(t), '[]'::json) from ("
|
||||
" select ar.index as run_number, ar.workflow_id, j.name, j.runs_on"
|
||||
" from action_run_job j join action_run ar on ar.id = j.run_id"
|
||||
" join repository r on r.id = j.repo_id join \"user\" o on o.id = r.owner_id"
|
||||
f" where o.lower_name = '{WG_OWNER.lower()}' and r.lower_name = '{repo.lower()}'"
|
||||
f" and ar.commit_sha = '{sha}' and j.status = 5) t),"
|
||||
" 'labels', (select coalesce(json_agg(agent_labels), '[]'::json)"
|
||||
" from action_runner where coalesce(deleted, 0) = 0));"
|
||||
)
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["docker", "exec", "-i", "windy-git-db-1", "sh", "-c",
|
||||
'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1'],
|
||||
input=query, capture_output=True, text=True, check=True, timeout=30,
|
||||
).stdout.strip()
|
||||
got = json.loads(out or "{}")
|
||||
runners = [set(json.loads(x or "[]")) for x in got.get("labels") or []]
|
||||
return [
|
||||
j for j in got.get("jobs") or []
|
||||
if any(set(json.loads(j.get("runs_on") or "[]")) <= r for r in runners)
|
||||
]
|
||||
except (subprocess.SubprocessError, OSError, ValueError) as e:
|
||||
print(f" {repo}: queued-job lookup skipped ({type(e).__name__})")
|
||||
return []
|
||||
|
||||
|
||||
def post_statuses(repo: str, sha: str) -> None:
|
||||
# Gitea caps a page at 50 (MAX_RESPONSE_ITEMS) whatever `limit` says, and a
|
||||
# daily scheduled workflow can push a quiet main's runs off page 1.
|
||||
runs = []
|
||||
for page in range(1, 6):
|
||||
st, body = gitea("GET", f"/repos/{WG_OWNER}/{repo}/actions/tasks?limit=50&page={page}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git runs -> {st}")
|
||||
runs += body.get("workflow_runs", [])
|
||||
if len(body.get("workflow_runs", [])) < 50:
|
||||
break
|
||||
latest: dict[str, dict] = {}
|
||||
for r in runs:
|
||||
if r["head_sha"] != sha or needs_daemon(repo, r["workflow_id"].removesuffix(".yml"), r["name"]):
|
||||
continue
|
||||
if f"{r['workflow_id'].removesuffix('.yml')}/{r['name']}" in NON_BLOCKING.get(repo, ()):
|
||||
continue
|
||||
ctx = f"windy-git/{r['workflow_id'].removesuffix('.yml')}/{r['name']}"
|
||||
if ctx not in latest or r["id"] > latest[ctx]["id"]:
|
||||
latest[ctx] = r
|
||||
# Queued jobs: `pending` where nothing newer has been picked up. A re-run
|
||||
# queued behind an old failure must read pending, not the stale red.
|
||||
for q in queued_jobs(repo, sha):
|
||||
wf = q["workflow_id"].removesuffix(".yml")
|
||||
if needs_daemon(repo, wf, q["name"]):
|
||||
continue
|
||||
if f"{wf}/{q['name']}" in NON_BLOCKING.get(repo, ()):
|
||||
continue
|
||||
ctx = f"windy-git/{wf}/{q['name']}"
|
||||
if ctx not in latest or q["run_number"] > latest[ctx]["run_number"]:
|
||||
latest[ctx] = {"id": 0, "status": "waiting", "run_number": q["run_number"]}
|
||||
bad = invalid_workflows(repo, sha)
|
||||
if not (latest or bad):
|
||||
return
|
||||
|
||||
st, existing = github("GET", f"/repos/{GH_OWNER}/{repo}/commits/{sha}/statuses?per_page=100")
|
||||
current: dict[str, str] = {}
|
||||
for s in existing or []: # newest first
|
||||
current.setdefault(s["context"], s["state"])
|
||||
|
||||
for ctx, (path, problem) in sorted(bad.items()):
|
||||
if current.get(ctx) == "error":
|
||||
continue
|
||||
st, _ = github(
|
||||
"POST",
|
||||
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
|
||||
{
|
||||
"state": "error",
|
||||
"context": ctx,
|
||||
"description": f"Windy Git ignored this workflow, no CI ran: {problem}"[:140],
|
||||
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}/{path}",
|
||||
},
|
||||
)
|
||||
print(f" {repo}@{sha[:7]} {ctx} = error ({problem}) -> {st}")
|
||||
|
||||
for ctx, r in sorted(latest.items()):
|
||||
state = STATE.get(r["status"])
|
||||
if state is None or current.get(ctx) == state:
|
||||
continue
|
||||
st, _ = github(
|
||||
"POST",
|
||||
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
|
||||
{
|
||||
"state": state,
|
||||
"context": ctx,
|
||||
"description": f"Windy Git CI on Veron 1: {r['status']}"[:140],
|
||||
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/actions/runs/{r['run_number']}",
|
||||
},
|
||||
)
|
||||
print(f" {repo}@{sha[:7]} {ctx} = {state} -> {st}")
|
||||
|
||||
|
||||
GUARD_CTX = "windy-git/compute-guard"
|
||||
HYGIENE_CTX = "windy-git/ci-hygiene"
|
||||
SECRET_CTX = "windy-git/secret-guard"
|
||||
|
||||
|
||||
def post_compute_guard(repo: str, sha: str, default_branch: str, is_default_head: bool) -> None:
|
||||
"""Windy Mind is the only door to AI compute: flag direct provider use (warn-only)."""
|
||||
_post_guard("compute_guard", GUARD_CTX, repo, sha, default_branch, is_default_head)
|
||||
|
||||
|
||||
def post_secret_guard(repo: str, sha: str, default_branch: str, is_default_head: bool) -> None:
|
||||
"""No live credential in a bridged repo (leak hunt 09-24). Findings carry sha256[:8] only."""
|
||||
_post_guard("secret_guard", SECRET_CTX, repo, sha, default_branch, is_default_head)
|
||||
|
||||
|
||||
def post_ci_hygiene(repo: str, sha: str, default_branch: str, is_default_head: bool) -> None:
|
||||
"""House rule 6: lockfile-only installs, pinned images, no host-port services (warn-only)."""
|
||||
_post_guard("ci_hygiene", HYGIENE_CTX, repo, sha, default_branch, is_default_head)
|
||||
|
||||
|
||||
def _post_guard(modname: str, ctx: str, repo: str, sha: str, default_branch: str,
|
||||
is_default_head: bool) -> None:
|
||||
"""One code path for every repo-scanning guard. Non-fatal and never a fake OK:
|
||||
if the guard can't run, nothing is posted."""
|
||||
try:
|
||||
import importlib
|
||||
|
||||
g = importlib.import_module(modname) # same directory; lazy so the bridge never depends on it
|
||||
findings = g.check(repo, sha, default_branch, is_default_head)
|
||||
except Exception as e: # noqa: BLE001 — a guard must never break CI signals
|
||||
print(f" {repo}@{sha[:7]} {ctx} skipped ({type(e).__name__}: {str(e)[:80]})")
|
||||
return
|
||||
if findings is None:
|
||||
return
|
||||
try: # Grant-owned code never blocks (orchestrator 09-23); lazy like the guards
|
||||
import importlib
|
||||
|
||||
lane, grant = importlib.import_module("guards_report").split_grant(repo, sha, findings)
|
||||
except Exception as e: # noqa: BLE001 — can't tell whose code: warn, never block
|
||||
print(f" {repo}@{sha[:7]} {ctx}: Grant-owned split failed ({type(e).__name__}); WARN only")
|
||||
lane, grant = [], list(findings)
|
||||
state, desc, first = g.status_for(lane, whole_tree=is_default_head, grant=grant)
|
||||
st, existing = github("GET", f"/repos/{GH_OWNER}/{repo}/commits/{sha}/statuses?per_page=100")
|
||||
for s in existing or []: # newest first: compare the latest guard status only
|
||||
if s["context"] == ctx:
|
||||
if (s["state"], s.get("description")) == (state, desc):
|
||||
return
|
||||
break
|
||||
url = (f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}/{first.path}#L{first.line}"
|
||||
if first else f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}")
|
||||
st, _ = github("POST", f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
|
||||
{"state": state, "context": ctx, "description": desc, "target_url": url})
|
||||
print(f" {repo}@{sha[:7]} {ctx} = {state} ({len(findings)} finding(s)) -> {st}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not (GITEA_TOKEN and GITHUB_TOKEN):
|
||||
sys.exit("GITEA_ADMIN_TOKEN and GITHUB_TOKEN are required")
|
||||
failed = 0
|
||||
for repo in REPOS:
|
||||
try:
|
||||
shas = sync_prs(repo)
|
||||
default_branch, default_head = "main", None
|
||||
st, br = github("GET", f"/repos/{GH_OWNER}/{repo}")
|
||||
if st == 200:
|
||||
default_branch = br["default_branch"]
|
||||
st, b = github("GET", f"/repos/{GH_OWNER}/{repo}/branches/{default_branch}")
|
||||
if st == 200:
|
||||
default_head = b["commit"]["sha"]
|
||||
shas.append(default_head)
|
||||
for sha in dict.fromkeys(shas):
|
||||
post_statuses(repo, sha)
|
||||
post_compute_guard(repo, sha, default_branch, sha == default_head)
|
||||
post_ci_hygiene(repo, sha, default_branch, sha == default_head)
|
||||
post_secret_guard(repo, sha, default_branch, sha == default_head)
|
||||
except Exception as e: # one repo's failure must not hide the others'
|
||||
print(f" FAILED {repo}: {e}")
|
||||
failed = 1
|
||||
return failed
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
24
scripts/promote_to_ci.sh
Executable file
24
scripts/promote_to_ci.sh
Executable file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# promote_to_ci.sh <repo> [workflow-to-disable ...] — pull mirror -> writable CI repo.
|
||||
# Run ON Veron as root. See docs/CUTOVER.md "Onboarding another private repo".
|
||||
#
|
||||
# ⚠️ It DELETES the mirror before importing (Gitea's migrate refuses an existing
|
||||
# name). If the import then fails, the Windy Git copy is gone until you re-run —
|
||||
# GitHub and the nightly R2 bundles still hold everything, but check first that
|
||||
# scripts/import_from_github.py will accept the repo. (2026-09-23: windy-pro was
|
||||
# deleted this way while the importer still refused it by name.)
|
||||
set -euo pipefail
|
||||
set -a; . /srv/windygit/src/.env; set +a
|
||||
export IMPORT_GITEA_URL=http://localhost:3080
|
||||
A=http://localhost:3080/api/v1; H="Authorization: token $GITEA_ADMIN_TOKEN"; r=$1; shift
|
||||
info=$(curl -s -H "$H" $A/repos/windyadmin/$r)
|
||||
m=$(echo "$info" | python3 -c 'import json,sys;print(json.load(sys.stdin).get("mirror"))')
|
||||
if [ "$m" = True ]; then
|
||||
curl -sf -o /dev/null -X DELETE -H "$H" $A/repos/windyadmin/$r
|
||||
(cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1)
|
||||
elif [ "$m" = False ]; then echo "$r already writable"; else echo "$r absent -> importing"; (cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1); fi
|
||||
db=$(curl -s -H "$H" $A/repos/windyadmin/$r | python3 -c 'import json,sys;print(json.load(sys.stdin).get("default_branch","main"))')
|
||||
for i in $(seq 1 120); do curl -sf -o /dev/null -H "$H" $A/repos/windyadmin/$r/branches/$db && break; sleep 5; done
|
||||
for w in "$@"; do printf " disable %s: " "$w"; curl -s -o /dev/null -w '%{http_code}\n' -X PUT -H "$H" $A/repos/windyadmin/$r/actions/workflows/$w/disable; done
|
||||
curl -s -H "$H" $A/repos/windyadmin/$r/actions/workflows | python3 -c 'import json,sys,os;print(" "+os.environ.get("R",""),[(w["path"].split("/")[-1],w["state"]) for w in json.load(sys.stdin).get("workflows",[])])'
|
||||
echo " default=$db"
|
||||
118
scripts/public_secret_scan.py
Normal file
118
scripts/public_secret_scan.py
Normal file
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Weekly: every PUBLIC repo in Grant's GitHub accounts, full history (every object,
|
||||
reachable or not, incl. PR refs), for secret-shaped strings. Leak hunt 09-24: a
|
||||
real bot token sat in a public test fixture for five months.
|
||||
|
||||
Output is hash + location only, never a value (house rule 10). Known fakes are
|
||||
excused by ci/secret-guard-allow.yml (same file as secret-guard; the repo NAME is
|
||||
matched across accounts). Runs on Veron as user1-gpu (gh is logged in there):
|
||||
|
||||
python3 scripts/public_secret_scan.py [--out FILE] [--owners a,b,...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import fnmatch
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import secret_shapes as ss # noqa: E402
|
||||
|
||||
OWNERS = ["sneakyfree", "VERONTECH", "Windstorm-Institute", "Windstorm-Labs", "Public-Streamer"]
|
||||
WORK = Path.home() / "leakscan" / "public"
|
||||
MAX_BLOB = 20_000_000
|
||||
|
||||
|
||||
def _run(*a: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(a, capture_output=True)
|
||||
|
||||
|
||||
def blob_hits(bare: Path) -> dict[str, list[tuple[str, str]]]:
|
||||
"""{blob: [(kind, hash8)]} over every blob object in the repo."""
|
||||
chk = _run("git", "-C", str(bare), "cat-file", "--batch-all-objects",
|
||||
"--batch-check=%(objectname) %(objecttype) %(objectsize)")
|
||||
blobs = [p[0] for p in (ln.split() for ln in chk.stdout.decode().splitlines())
|
||||
if len(p) == 3 and p[1] == "blob" and int(p[2]) < MAX_BLOB]
|
||||
if not blobs:
|
||||
return {}
|
||||
import threading
|
||||
p = subprocess.Popen(["git", "-C", str(bare), "cat-file", "--batch"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
|
||||
|
||||
def feed() -> None: # separate thread: writing everything first deadlocks (both pipes fill)
|
||||
p.stdin.write(("\n".join(blobs) + "\n").encode())
|
||||
p.stdin.close()
|
||||
threading.Thread(target=feed, daemon=True).start()
|
||||
out: dict[str, list[tuple[str, str]]] = {}
|
||||
for _ in blobs:
|
||||
hdr = p.stdout.readline().split()
|
||||
data = p.stdout.read(int(hdr[2]))
|
||||
p.stdout.read(1)
|
||||
found = ss.find(data.decode("utf-8", "ignore"))
|
||||
if found:
|
||||
out[hdr[0].decode()] = found
|
||||
p.wait()
|
||||
return out
|
||||
|
||||
|
||||
def locate(bare: Path, blob: str) -> dict:
|
||||
lg = _run("git", "-C", str(bare), "log", "--all", "--format=@@%H %cI", "--name-only",
|
||||
f"--find-object={blob}").stdout.decode()
|
||||
commits, paths = [], set()
|
||||
for line in lg.splitlines():
|
||||
if line.startswith("@@"):
|
||||
commits.append(line[2:].split())
|
||||
elif line.strip():
|
||||
paths.add(line.strip())
|
||||
head = _run("git", "-C", str(bare), "ls-tree", "-r", "HEAD").stdout.decode()
|
||||
first = commits[-1] if commits else ["unreachable", "-"]
|
||||
return {"paths": sorted(paths), "first_commit": first[0][:10], "first_date": first[1],
|
||||
"in_head": blob in head}
|
||||
|
||||
|
||||
def excused(repo: str, kind: str, h: str, paths: list[str], allow: dict) -> bool:
|
||||
a = allow.get(repo) or {"hashes": set(), "paths": []}
|
||||
if kind in {"private key block"}:
|
||||
return bool(paths) and all(any(kind in k and fnmatch.fnmatch(p, g) for g, k in a["paths"]) for p in paths)
|
||||
return h in a["hashes"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--out", default=str(WORK / "latest.json"))
|
||||
ap.add_argument("--owners", default=",".join(OWNERS))
|
||||
args = ap.parse_args()
|
||||
import secret_guard as sg
|
||||
allow = sg.load_allow()
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
rows, scanned, errors = [], [], []
|
||||
for owner in args.owners.split(","):
|
||||
lst = _run("gh", "repo", "list", owner, "--limit", "1000", "--visibility", "public", "--json", "name")
|
||||
for r in json.loads(lst.stdout or b"[]"):
|
||||
name = r["name"]
|
||||
bare = WORK / owner / f"{name}.git"
|
||||
if bare.exists():
|
||||
c = _run("git", "-C", str(bare), "remote", "update", "--prune")
|
||||
else:
|
||||
bare.parent.mkdir(parents=True, exist_ok=True)
|
||||
c = _run("gh", "repo", "clone", f"{owner}/{name}", str(bare), "--", "--mirror", "-q")
|
||||
if c.returncode:
|
||||
errors.append(f"{owner}/{name}")
|
||||
continue
|
||||
scanned.append(f"{owner}/{name}")
|
||||
for blob, found in blob_hits(bare).items():
|
||||
loc = locate(bare, blob)
|
||||
for kind, h in sorted(set(found)):
|
||||
rows.append({"repo": f"{owner}/{name}", "kind": kind, "hash8": h, **loc,
|
||||
"excused": excused(name, kind, h, loc["paths"], allow)})
|
||||
Path(args.out).write_text(json.dumps({"scanned": scanned, "errors": errors, "rows": rows}, indent=1))
|
||||
new = [r for r in rows if not r["excused"]]
|
||||
print(f"scanned {len(scanned)} public repos, {len(errors)} errors, {len(rows)} secret-shaped, {len(new)} NOT excused")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
52
scripts/rerun_ci.sh
Executable file
52
scripts/rerun_ci.sh
Executable file
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-run a PR's (or branch's) CI on Windy Git. Runs ON Veron 1.
|
||||
#
|
||||
# bash scripts/rerun_ci.sh <repo> <branch> <sha-prefix>
|
||||
#
|
||||
# Gitea 1.24 has NO rerun API; the web button needs a hub-SSO session as
|
||||
# windyadmin, which is Grant's identity, so we don't use it. Instead: move the
|
||||
# Windy Git branch back one commit, let the next sync force-push the GitHub head
|
||||
# again, and Gitea fires an ordinary push / pull_request_sync event on the SAME
|
||||
# commit. Every workflow on that event re-runs, not only the failed one.
|
||||
#
|
||||
# Safety: refuses unless the branch is exactly at <sha-prefix> (GitHub's head),
|
||||
# never rewinds while a sync is running (a run already past this repo would
|
||||
# not push it back), waits for a sync that STARTS after the rewind, and if the
|
||||
# branch is not verifiably back at <sha-prefix> by the deadline, restores it
|
||||
# itself, so Windy Git is never left behind GitHub.
|
||||
set -euo pipefail
|
||||
repo="${1:?repo}"; branch="${2:?branch}"; want="${3:?sha prefix}"
|
||||
# wg-q lives in the INVOKING user's ~/bin; under sudo, ~ is /root.
|
||||
WGQ="${WGQ:-$(getent passwd "${SUDO_USER:-$USER}" | cut -d: -f6)/bin/wg-q}"
|
||||
# Gitea stores repositories LOWERCASED on disk (WindyCloud -> windycloud.git).
|
||||
G="sudo docker exec -u git windy-git-gitea-1 git -C /data/git/repositories/windyadmin/${repo,,}.git"
|
||||
|
||||
head=$($G rev-parse "refs/heads/${branch}")
|
||||
[[ "$head" == "$want"* ]] || { echo "refusing: ${branch} is at ${head:0:7}, not ${want}"; exit 1; }
|
||||
parent=$($G rev-parse "${head}^")
|
||||
|
||||
# NOT `systemctl is-active`: the sync is Type=oneshot, which reads "activating"
|
||||
# (exit 3) for its whole run, so is-active says "idle" mid-run.
|
||||
busy() { case "$(systemctl show windygit-sync -p ActiveState --value)" in
|
||||
activating|active|deactivating|reloading) return 0;; esac; return 1; }
|
||||
while busy; do sleep 5; done
|
||||
$G update-ref "refs/heads/${branch}" "$parent" "$head"
|
||||
mark=$(awk '{print int($1*1000000)}' /proc/uptime)
|
||||
echo "rewound ${repo}:${branch} ${head:0:7} -> ${parent:0:7}"
|
||||
|
||||
deadline=$(( $(date +%s) + 900 ))
|
||||
until [ "$(systemctl show windygit-sync -p ExecMainStartTimestampMonotonic --value)" -gt "$mark" ] \
|
||||
&& [ "$($G rev-parse "refs/heads/${branch}")" = "$head" ]; do
|
||||
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||
$G update-ref "refs/heads/${branch}" "$head" "$($G rev-parse "refs/heads/${branch}")" || true
|
||||
echo "TIMEOUT: restored ${branch} to ${head:0:7} by hand; NO new run fired"; exit 1
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
echo "restored by sync: ${branch} = ${head:0:7}"
|
||||
sleep 5
|
||||
"$WGQ" <<SQL
|
||||
select ar.index, ar.workflow_id, ar.event, ar.status, to_char(to_timestamp(ar.created),'HH24:MI:SS')
|
||||
from action_run ar join repository r on r.id = ar.repo_id
|
||||
where r.name = '${repo}' and ar.commit_sha = '${head}' order by ar.id desc limit 6;
|
||||
SQL
|
||||
209
scripts/runner_guard.py
Normal file
209
scripts/runner_guard.py
Normal file
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Runner guard: no workflow may let a STRANGER's code reach a self-hosted runner (Boss 10-01).
|
||||
|
||||
Our self-hosted GitHub runners run on Veron as `github-runner`, which is in the docker group
|
||||
(= root on Veron). Until ephemeral containerised runners exist (after launch), the cheap
|
||||
guard is to refuse the workflow shapes that hand a self-hosted runner to outsiders:
|
||||
|
||||
R1 pull_request_target : runs with secrets/write token in the BASE repo context
|
||||
R2 pull_request on self-hosted : fork PRs run their own code, unless the job is gated to
|
||||
same-repo heads (`if:` on head.repo.full_name == github.repository
|
||||
or head.repo.fork == false) or an `environment:`
|
||||
R3 issue_comment / workflow_run / issues / discussion* / pull_request_review* / fork / watch
|
||||
: anyone can fire these; never on self-hosted without an environment gate
|
||||
(push, tags, schedule, workflow_dispatch, repository_dispatch, workflow_call: writers only, fine)
|
||||
|
||||
A job counts as self-hosted when its runs-on names `self-hosted`, or is an expression we
|
||||
can't resolve (conservative). Findings carry file:line, never file content beyond that.
|
||||
|
||||
python3 scripts/runner_guard.py lint FILE... # local files
|
||||
python3 scripts/runner_guard.py report [--owners a,b] # default branch of every PUBLIC repo
|
||||
python3 scripts/runner_guard.py pr [--owners a,b] [--post] # open PRs on public repos: changed workflows
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
OWNERS = ["sneakyfree", "VERONTECH", "Windstorm-Institute", "Windstorm-Labs", "Public-Streamer"]
|
||||
CTX = "windy-git/runner-guard"
|
||||
OUTSIDE = {"issue_comment", "workflow_run", "issues", "discussion", "discussion_comment",
|
||||
"pull_request_review", "pull_request_review_comment", "fork", "watch"}
|
||||
SAME_REPO_GATES = ("head.repo.full_name == github.repository", "github.repository == github.event.pull_request.head.repo.full_name",
|
||||
"head.repo.fork == false", "!github.event.pull_request.head.repo.fork")
|
||||
|
||||
|
||||
def _node_map(node):
|
||||
"""{key: (value_node, line)} for a YAML mapping node."""
|
||||
if not isinstance(node, yaml.MappingNode):
|
||||
return {}
|
||||
return {k.value: (v, k.start_mark.line + 1) for k, v in node.value if isinstance(k, yaml.ScalarNode)}
|
||||
|
||||
|
||||
def _triggers(on_node) -> dict[str, int]:
|
||||
"""{event: line}."""
|
||||
if isinstance(on_node, yaml.ScalarNode):
|
||||
return {on_node.value: on_node.start_mark.line + 1}
|
||||
if isinstance(on_node, yaml.SequenceNode):
|
||||
return {n.value: n.start_mark.line + 1 for n in on_node.value if isinstance(n, yaml.ScalarNode)}
|
||||
return {k: line for k, (_v, line) in _node_map(on_node).items()}
|
||||
|
||||
|
||||
def _self_hosted(runs_on) -> bool:
|
||||
if runs_on is None:
|
||||
return False
|
||||
text = yaml.serialize(runs_on) if isinstance(runs_on, yaml.Node) else str(runs_on)
|
||||
return "self-hosted" in text or "${{" in text
|
||||
|
||||
|
||||
def lint_text(path: str, text: str) -> list[tuple[str, int, str, str]]:
|
||||
"""[(path, line, rule, message)]. Unparseable YAML is a finding (it cannot be reviewed)."""
|
||||
try:
|
||||
root = yaml.compose(text)
|
||||
except yaml.YAMLError as e:
|
||||
line = getattr(getattr(e, "problem_mark", None), "line", 0) + 1
|
||||
return [(path, line, "R0", "workflow YAML does not parse; cannot be checked")]
|
||||
top = _node_map(root)
|
||||
if "on" not in top or "jobs" not in top:
|
||||
return []
|
||||
trig = _triggers(top["on"][0])
|
||||
jobs = _node_map(top["jobs"][0])
|
||||
out = []
|
||||
if "pull_request_target" in trig:
|
||||
out.append((path, trig["pull_request_target"], "R1",
|
||||
"pull_request_target runs fork code with base-repo secrets; not allowed"))
|
||||
for name, (jnode, jline) in jobs.items():
|
||||
j = _node_map(jnode)
|
||||
ro = j.get("runs-on", (None, jline))
|
||||
if not _self_hosted(ro[0]):
|
||||
continue
|
||||
has_env = "environment" in j
|
||||
cond = j["if"][0].value if "if" in j and isinstance(j["if"][0], yaml.ScalarNode) else ""
|
||||
same_repo = any(g in cond.replace(" ", " ") for g in SAME_REPO_GATES)
|
||||
if "pull_request" in trig and not (has_env or same_repo):
|
||||
out.append((path, ro[1], "R2", f"job '{name}' runs fork PR code on a self-hosted runner "
|
||||
"(gate it: if: github.event.pull_request.head.repo.full_name == github.repository, or an environment)"))
|
||||
for ev in sorted(OUTSIDE & trig.keys()):
|
||||
if not has_env:
|
||||
out.append((path, trig[ev], "R3", f"'{ev}' can be fired by anyone and job '{name}' is self-hosted "
|
||||
"without an environment gate"))
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- GitHub side
|
||||
def gh(*args: str, check=True) -> str:
|
||||
r = subprocess.run(["gh", "api", *args], capture_output=True, text=True, timeout=60)
|
||||
if check and r.returncode != 0:
|
||||
raise RuntimeError(f"gh api {args[0]} failed")
|
||||
return r.stdout
|
||||
|
||||
|
||||
def public_repos(owners) -> list[tuple[str, str]]:
|
||||
out = []
|
||||
for o in owners:
|
||||
txt = gh(f"users/{o}/repos?per_page=100&type=owner", "--paginate",
|
||||
"--jq", '.[]|select(.private==false and .archived==false)|.full_name+" "+.default_branch', check=False)
|
||||
out += [tuple(row.split()) for row in txt.splitlines() if row.strip()]
|
||||
return out
|
||||
|
||||
|
||||
def workflows_at(full: str, ref: str) -> list[tuple[str, str]]:
|
||||
txt = gh(f"repos/{full}/contents/.github/workflows?ref={ref}", "--jq",
|
||||
'.[]|select(.type=="file")|.path', check=False)
|
||||
files = [p for p in txt.splitlines() if p.endswith((".yml", ".yaml"))]
|
||||
return [(p, file_at(full, p, ref)) for p in files]
|
||||
|
||||
|
||||
def file_at(full: str, path: str, ref: str) -> str:
|
||||
raw = gh(f"repos/{full}/contents/{path}?ref={ref}", "--jq", ".content", check=False).strip()
|
||||
return base64.b64decode(raw).decode("utf-8", "replace") if raw else ""
|
||||
|
||||
|
||||
def cmd_report(owners) -> int:
|
||||
hits = 0
|
||||
repos = public_repos(owners)
|
||||
for full, branch in repos:
|
||||
for path, text in workflows_at(full, branch):
|
||||
for p, line, rule, msg in lint_text(path, text):
|
||||
hits += 1
|
||||
print(f"{full}\t{p}:{line}\t{rule}\t{msg}")
|
||||
print(f"# runner-guard sweep: {hits} hit(s) in {len(repos)} public repos")
|
||||
return 1 if hits else 0
|
||||
|
||||
|
||||
STATE = os.environ.get("RUNNER_GUARD_STATE", "/var/lib/windy-git/runner-guard-posted.json")
|
||||
|
||||
|
||||
def cmd_pr(owners, post: bool) -> int:
|
||||
# Post each (repo, sha, state, description) ONCE: the sync runs every 5 min and GitHub caps
|
||||
# statuses per sha+context at 1000.
|
||||
try:
|
||||
with open(STATE) as fh:
|
||||
posted = set(json.load(fh))
|
||||
except (OSError, ValueError):
|
||||
posted = set()
|
||||
seen = set()
|
||||
for full, _branch in public_repos(owners):
|
||||
prs = gh(f"repos/{full}/pulls?state=open&per_page=50", "--jq",
|
||||
'.[]|(.number|tostring)+" "+.head.sha+" "+.head.repo.full_name', check=False)
|
||||
for line in prs.splitlines():
|
||||
num, sha, head_repo = line.split(" ", 2)
|
||||
files = gh(f"repos/{full}/pulls/{num}/files?per_page=100", "--jq",
|
||||
'.[]|select(.status!="removed")|.filename', check=False).split()
|
||||
wf = [f for f in files if f.startswith(".github/workflows/") and f.endswith((".yml", ".yaml"))]
|
||||
# a fork's own content is read from the head repo at the head sha
|
||||
src = head_repo if head_repo and head_repo != "null" else full
|
||||
found = [h for f in wf for h in lint_text(f, file_at(src, f, sha))]
|
||||
if found:
|
||||
p, ln, rule, msg = found[0]
|
||||
state, desc = "failure", f"BLOCKED: {rule} {p}:{ln}: {msg}"[:140]
|
||||
else:
|
||||
state, desc = "success", ("OK: no workflow changes" if not wf else
|
||||
"OK: no stranger-code path to a self-hosted runner")
|
||||
key = f"{full}@{sha}:{state}:{desc}"
|
||||
seen.add(key)
|
||||
if key in posted:
|
||||
continue
|
||||
print(f"{full}#{num}@{sha[:7]} {state} {desc}")
|
||||
if post:
|
||||
gh(f"repos/{full}/statuses/{sha}", "-f", f"state={state}", "-f", f"context={CTX}",
|
||||
"-f", f"description={desc}", check=False)
|
||||
posted.add(key)
|
||||
if post: # keep only keys for PRs still open, so the file never grows without bound
|
||||
os.makedirs(os.path.dirname(STATE), exist_ok=True)
|
||||
with open(STATE, "w") as fh:
|
||||
json.dump(sorted(posted & seen), fh)
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(prog="runner_guard")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
lint_p = sub.add_parser("lint")
|
||||
lint_p.add_argument("files", nargs="+")
|
||||
rep = sub.add_parser("report")
|
||||
rep.add_argument("--owners", default=",".join(OWNERS))
|
||||
prp = sub.add_parser("pr")
|
||||
prp.add_argument("--owners", default="sneakyfree") # self-hosted runners exist only there
|
||||
prp.add_argument("--post", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
if a.cmd == "lint":
|
||||
hits = []
|
||||
for f in a.files:
|
||||
with open(f, errors="replace") as fh:
|
||||
hits += lint_text(f, fh.read())
|
||||
for p_, ln, rule, msg in hits:
|
||||
print(f"{p_}:{ln}\t{rule}\t{msg}")
|
||||
return 1 if hits else 0
|
||||
owners = a.owners.split(",")
|
||||
return cmd_report(owners) if a.cmd == "report" else cmd_pr(owners, a.post)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
131
scripts/secret_guard.py
Normal file
131
scripts/secret_guard.py
Normal file
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Secret guard: no live credential lands in a bridged repo (leak hunt 09-24).
|
||||
|
||||
Same walker, cache and GitHub posting as compute_guard / ci_hygiene
|
||||
(`windy-git/secret-guard`), but over EVERY text file, and a finding carries only
|
||||
"<kind> #<sha256[:8]>", never the value (house rule 10). Known fakes are allowed
|
||||
BY HASH in ci/secret-guard-allow.yml (repo + hashes + reason).
|
||||
|
||||
sudo python3 scripts/secret_guard.py report [repo ...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import compute_guard as cg # noqa: E402 (shared walker, cache)
|
||||
import secret_shapes as ss # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ALLOW_FILE = Path(os.environ.get("SECRET_GUARD_ALLOW", ROOT / "ci" / "secret-guard-allow.yml"))
|
||||
MODE = os.environ.get("SECRET_GUARD_MODE", "warn")
|
||||
# Kinds that only WARN (rolled out warn-first); empty = every kind blocks in block mode.
|
||||
WARN_KINDS = {k for k in os.environ.get("SECRET_GUARD_WARN_KINDS", "").split(",") if k}
|
||||
NEVER = re.compile(r"(^|/)(node_modules|vendor|third_party)/")
|
||||
|
||||
|
||||
def path_ok(path: str) -> bool:
|
||||
return not NEVER.search(path)
|
||||
|
||||
|
||||
# A private-key match is only its BEGIN line, so its hash is the same everywhere:
|
||||
# those are allowed by PATH (entries with `paths` + `kinds`), everything else by HASH.
|
||||
PATH_ONLY_KINDS = {"private key block"}
|
||||
|
||||
|
||||
def load_allow(path: Path = ALLOW_FILE) -> dict[str, dict]:
|
||||
"""{repo: {"hashes": {hash8}, "paths": [(glob, {kind})]}}; every entry needs a reason."""
|
||||
data = yaml.safe_load(path.read_text()) if path.exists() else {}
|
||||
out: dict[str, dict] = {}
|
||||
for e in (data or {}).get("allow") or []:
|
||||
if not (e.get("repo") and (e.get("hashes") or (e.get("paths") and e.get("kinds")))
|
||||
and str(e.get("reason", "")).strip()):
|
||||
raise ValueError(f"allow entry needs repo, hashes (or paths + kinds) and a reason: {e}")
|
||||
if e.get("paths") and not set(e["kinds"]) <= PATH_ONLY_KINDS:
|
||||
raise ValueError(f"path allows are only for {sorted(PATH_ONLY_KINDS)}: {e}")
|
||||
r = out.setdefault(e["repo"], {"hashes": set(), "paths": []})
|
||||
r["hashes"].update(str(h) for h in e.get("hashes") or [])
|
||||
r["paths"] += [(g, set(e["kinds"])) for g in e.get("paths") or []]
|
||||
return out
|
||||
|
||||
|
||||
def scan_line(path: str, text: str) -> list[tuple[str, str]]:
|
||||
return [(kind, f"{kind} #{h}") for kind, h in ss.find(text)]
|
||||
|
||||
|
||||
def _drop_allowed(repo: str, findings, allow: dict[str, dict]):
|
||||
a = allow.get(repo) or {"hashes": set(), "paths": []}
|
||||
|
||||
def ok(f) -> bool:
|
||||
if f.kind in PATH_ONLY_KINDS:
|
||||
return any(f.kind in kinds and fnmatch.fnmatch(f.path, g) for g, kinds in a["paths"])
|
||||
return f.match.rsplit("#", 1)[-1] in a["hashes"]
|
||||
return [f for f in findings if not ok(f)]
|
||||
|
||||
|
||||
def check(repo: str, sha: str, default_branch: str, is_default_head: bool):
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
if not bare.is_dir() or not cg.fetched(bare, sha): # pushed after the fetch: next cycle
|
||||
return None
|
||||
allow = load_allow()
|
||||
rules = hashlib.sha256(("|".join(rx.pattern for _, rx in ss.PATTERNS) + ss.PREFILTER).encode()).hexdigest()[:8]
|
||||
kw = dict(line_fn=scan_line, path_ok=path_ok)
|
||||
if is_default_head:
|
||||
fs = cg.cached_scan(f"sec-tree:{repo}:{sha}:{rules}",
|
||||
lambda: cg.scan_tree(repo, bare, sha, [], prefilter=ss.PREFILTER, **kw))
|
||||
else:
|
||||
fs = cg.cached_scan(f"sec-pr:{repo}:{sha}:{rules}",
|
||||
lambda: cg.scan_added(repo, bare, f"refs/heads/{default_branch}", sha, [], **kw))
|
||||
return _drop_allowed(repo, fs, allow)
|
||||
|
||||
|
||||
def status_for(findings, whole_tree: bool, grant=()):
|
||||
"""Same contract as the other guards. `grant` findings never block."""
|
||||
scope = "in tree" if whole_tree else "added"
|
||||
if not findings and grant:
|
||||
g, n = grant[0], len(grant)
|
||||
return "success", f"⚠ WARN (Grant-owned, not blocking): {n} secret-shaped string{'s' if n > 1 else ''} {scope}, e.g. {g.path}:{g.line} {g.match}"[:140], g
|
||||
if not findings:
|
||||
return "success", f"OK: no secret-shaped strings {scope}", None
|
||||
f, n = findings[0], len(findings)
|
||||
soft = MODE != "block" or all(x.kind in WARN_KINDS for x in findings)
|
||||
state = "success" if soft else "failure"
|
||||
lead = "⚠ WARN (not blocking)" if soft else "BLOCKED"
|
||||
if not soft:
|
||||
f = next(x for x in findings if x.kind not in WARN_KINDS)
|
||||
return state, f"{lead}: {n} secret-shaped string{'s' if n > 1 else ''} {scope}, e.g. {f.path}:{f.line} {f.match}"[:140], f
|
||||
|
||||
|
||||
def report(repos: list[str]) -> int:
|
||||
allow = load_allow()
|
||||
total = 0
|
||||
for repo in repos:
|
||||
bare = cg.WORK / f"{repo}.git"
|
||||
if not bare.is_dir():
|
||||
continue
|
||||
head = cg._git(bare, "symbolic-ref", "--short", "HEAD").strip()
|
||||
sha = cg._git(bare, "rev-parse", head).strip()
|
||||
fs = _drop_allowed(repo, cg.scan_tree(repo, bare, sha, [], line_fn=scan_line, path_ok=path_ok,
|
||||
prefilter=ss.PREFILTER), allow)
|
||||
total += len(fs)
|
||||
print(f"## {repo} ({head} {sha[:7]}): {len(fs)} finding(s)")
|
||||
for f in fs:
|
||||
print(f" {f.path}:{f.line} {f.match}")
|
||||
print(f"TOTAL {total}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) >= 2 and sys.argv[1] == "report":
|
||||
default = os.environ.get("BRIDGE_REPOS", "").split() or sorted(
|
||||
p.name.removesuffix(".git") for p in cg.WORK.glob("*.git"))
|
||||
sys.exit(report(sys.argv[2:] or default))
|
||||
sys.exit(__doc__)
|
||||
210
scripts/secret_scan.py
Normal file
210
scripts/secret_scan.py
Normal file
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env python3
|
||||
"""secret-scan: hash-only secret finder. Boss ruling 10-01 after three lanes printed secrets
|
||||
into their own transcripts while hunting secrets (house rule 10).
|
||||
|
||||
secret-scan <path> [--history] [--repo <git url or path>] [--no-lockbox]
|
||||
|
||||
Reports `file:line` (and the commit with --history), WHICH lockbox entry matched (the KEY
|
||||
NAME only) or which secret SHAPE matched (twilio, zai, aws, ...), plus a sha256[:8] of the
|
||||
token for allow-listing. It NEVER prints, logs or writes a value or any fragment of one
|
||||
(no context line, no masking). The lockbox is loaded in memory only. stdout only.
|
||||
Exit 0 = clean, 1 = findings, 2 = usage/error.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import secret_shapes as ss # noqa: E402 (the SAME shapes as secret-guard)
|
||||
|
||||
HOME = Path.home()
|
||||
LOCKBOX_PATHS = [p for p in os.environ.get("SECRET_SCAN_LOCKBOX_PATHS", "").split(":") if p] or [
|
||||
str(HOME / "kit-army-config" / "secrets"), str(HOME / "kit-army-config" / "ACCESS_LOCKBOX.md")]
|
||||
# Extra shapes that are scan-only (not in the blocking guard): label, regex.
|
||||
EXTRA = [("zai key", re.compile(r"(?<![0-9a-f])[0-9a-f]{32}\.[A-Za-z0-9]{16}(?![A-Za-z0-9])"))]
|
||||
SKIP_DIRS = {".git", "node_modules", "vendor", "third_party", "__pycache__", ".venv"}
|
||||
MAX_BYTES = 5_000_000
|
||||
|
||||
RUN = re.compile(r"[A-Za-z0-9][A-Za-z0-9_\-]{15,199}")
|
||||
UUID = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
|
||||
NAME = re.compile(r"[\s>*`|-]*([A-Za-z][A-Za-z0-9_]{2,60})\s*[=:|]")
|
||||
|
||||
|
||||
def h16(t: str) -> str:
|
||||
return hashlib.sha256(t.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def cands(line: str):
|
||||
"""Token candidates: runs >=16 chars with a digit and a letter; git shas/uuids excluded."""
|
||||
for chunk in re.split(r"[^A-Za-z0-9_\-.]+", line):
|
||||
parts = [chunk, *chunk.split(".")] if "." in chunk else [chunk]
|
||||
for p in parts:
|
||||
for m in RUN.finditer(p):
|
||||
t = m.group(0)
|
||||
if not (re.search(r"\d", t) and re.search(r"[A-Za-z]", t)):
|
||||
continue
|
||||
if re.fullmatch(r"[0-9a-fA-F]{40}|[0-9a-fA-F]{64}", t) or UUID.match(t.lower()):
|
||||
continue
|
||||
yield t
|
||||
|
||||
|
||||
def load_lockbox() -> dict[str, set[str]]:
|
||||
"""{hash16: {key-name labels}}; values never leave this dict."""
|
||||
out: dict[str, set[str]] = {}
|
||||
files: list[str] = []
|
||||
for p in LOCKBOX_PATHS:
|
||||
if os.path.isdir(p):
|
||||
for root, _d, fs in os.walk(p):
|
||||
files += [os.path.join(root, f) for f in fs]
|
||||
elif os.path.isfile(p):
|
||||
files.append(p)
|
||||
for f in files:
|
||||
try:
|
||||
with open(f, errors="ignore") as fh:
|
||||
for line in fh:
|
||||
m = NAME.match(line)
|
||||
label = m.group(1) if m else "?"
|
||||
for t in cands(line):
|
||||
out.setdefault(h16(t), set()).add(label)
|
||||
except OSError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def scan_line(line: str, lockbox: dict[str, set[str]]) -> list[tuple[str, str]]:
|
||||
"""[(label, hash8)]: `shape:<kind>` and/or `lockbox:<NAMES>`. No value escapes."""
|
||||
res: list[tuple[str, str]] = []
|
||||
for kind, h in ss.find(line):
|
||||
res.append((f"shape:{kind}", h))
|
||||
for kind, rx in EXTRA:
|
||||
for m in rx.finditer(line):
|
||||
res.append((f"shape:{kind}", ss.h8(m.group(0))))
|
||||
for t in cands(line):
|
||||
k = h16(t)
|
||||
if k in lockbox:
|
||||
names = sorted(n for n in lockbox[k])
|
||||
res.append(("lockbox:" + ",".join(names)[:70], k[:8]))
|
||||
return sorted(set(res))
|
||||
|
||||
|
||||
def is_text(path: Path) -> bool:
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
return b"\0" not in fh.read(4096)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def scan_tree(root: Path, lockbox):
|
||||
files = [root] if root.is_file() else [
|
||||
Path(dp) / f for dp, dn, fn in os.walk(root) for f in fn
|
||||
if not set(Path(dp).relative_to(root).parts) & SKIP_DIRS]
|
||||
for p in sorted(files):
|
||||
try:
|
||||
if p.stat().st_size > MAX_BYTES or not is_text(p):
|
||||
continue
|
||||
with open(p, errors="ignore") as fh:
|
||||
for n, line in enumerate(fh, 1):
|
||||
for label, h in scan_line(line, lockbox):
|
||||
yield (str(p), n, None, label, h)
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
|
||||
def git(repo: str, *a: str) -> subprocess.Popen:
|
||||
return subprocess.Popen(["git", "--git-dir", repo, *a], stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL, text=True, errors="ignore")
|
||||
|
||||
|
||||
def scan_history(gitdir: str, lockbox):
|
||||
"""Every ADDED line on every ref (incl. PR refs). Oldest commit per (hash, file, line)."""
|
||||
seen: dict[tuple, str] = {}
|
||||
p = git(gitdir, "log", "--all", "-p", "-U0", "--no-color", "--format=@@C %h", "-a")
|
||||
commit = path = None
|
||||
ln = 0
|
||||
for row in p.stdout: # type: ignore[union-attr]
|
||||
if row.startswith("@@C "):
|
||||
commit = row[4:].strip()
|
||||
elif row.startswith("+++ "):
|
||||
path = row[6:].strip() if row.startswith("+++ b/") else None
|
||||
elif row.startswith("@@ "):
|
||||
m = re.search(r"\+(\d+)", row)
|
||||
ln = int(m.group(1)) - 1 if m else 0
|
||||
elif row.startswith("+") and path:
|
||||
ln += 1
|
||||
for label, h in scan_line(row[1:], lockbox):
|
||||
seen[(label, h, path, ln)] = commit or "?"
|
||||
p.wait()
|
||||
for (label, h, path, ln), c in sorted(seen.items(), key=lambda x: (x[0][2], x[0][3])):
|
||||
yield (path, ln, c, label, h)
|
||||
|
||||
|
||||
def resolve_gitdir(p: Path) -> str | None:
|
||||
for cand in (p / ".git", p):
|
||||
if (cand / "HEAD").exists() and ((cand / "objects").exists()):
|
||||
return str(cand)
|
||||
return None
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(prog="secret-scan", description=__doc__.split("\n\n")[1] if __doc__ else "")
|
||||
ap.add_argument("path", nargs="?", help="file, directory, or git repo (with --history)")
|
||||
ap.add_argument("--history", action="store_true", help="scan every added line in all git history")
|
||||
ap.add_argument("--repo", help="git URL or path to scan (mirror-cloned to a temp dir, then deleted)")
|
||||
ap.add_argument("--no-lockbox", action="store_true", help="shapes only")
|
||||
a = ap.parse_args(argv)
|
||||
if not (a.path or a.repo):
|
||||
ap.print_usage()
|
||||
return 2
|
||||
lockbox = {} if a.no_lockbox else load_lockbox()
|
||||
print(f"# secret-scan: {len(lockbox)} lockbox tokens in memory, values never printed", flush=True)
|
||||
tmp = None
|
||||
findings = 0
|
||||
try:
|
||||
if a.repo:
|
||||
tmp = tempfile.mkdtemp(prefix="secret-scan-", dir=str(HOME / ".cache") if (HOME / ".cache").is_dir() else None)
|
||||
os.chmod(tmp, 0o700)
|
||||
target = os.path.join(tmp, "r.git")
|
||||
rc = subprocess.run(["git", "clone", "-q", "--mirror", a.repo, target],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode
|
||||
if rc != 0:
|
||||
print("error: clone failed (details withheld: URLs can carry tokens)")
|
||||
return 2
|
||||
a.history = True
|
||||
gitdir = target
|
||||
elif a.history:
|
||||
gitdir = resolve_gitdir(Path(a.path))
|
||||
if not gitdir:
|
||||
print("error: --history needs a git repo path")
|
||||
return 2
|
||||
if a.history:
|
||||
for path, ln, c, label, h in scan_history(gitdir, lockbox):
|
||||
findings += 1
|
||||
print(f"{path}:{ln} commit={c} {label} #{h}")
|
||||
else:
|
||||
for path, ln, _c, label, h in scan_tree(Path(a.path), lockbox):
|
||||
findings += 1
|
||||
print(f"{path}:{ln} {label} #{h}")
|
||||
finally:
|
||||
if tmp:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
print(f"# {findings} finding(s)")
|
||||
return 1 if findings else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
except Exception as e: # never a traceback: it could carry data
|
||||
print(f"error: {type(e).__name__}")
|
||||
sys.exit(2)
|
||||
46
scripts/secret_shapes.py
Normal file
46
scripts/secret_shapes.py
Normal file
@@ -0,0 +1,46 @@
|
||||
"""Secret-shaped strings, shared by secret_guard (bridged repos) and
|
||||
public_secret_scan (weekly, every public repo). A finding NEVER carries the value:
|
||||
only its kind and sha256[:8] (house rule 10). Leak hunt 09-24: @Windy_0_bot's
|
||||
token sat in a public repo's test fixture for five months."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
|
||||
# (kind, regex). Order matters only for readability; each match is reported once.
|
||||
PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||
("telegram bot token", re.compile(r"(?<![0-9])[0-9]{8,10}:[A-Za-z0-9_-]{35}(?![A-Za-z0-9_-])")),
|
||||
("github token", re.compile(r"\b(?:gh[pousr]_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{82})\b")),
|
||||
("aws access key", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")),
|
||||
("slack token", re.compile(r"\bxox[abprs]-[A-Za-z0-9-]{10,}")),
|
||||
("anthropic key", re.compile(r"\bsk-ant-[A-Za-z0-9_-]{20,}")),
|
||||
("openai key", re.compile(r"\bsk-(?:proj-|svcacct-)?(?!ant-)[A-Za-z0-9_-]{32,}")),
|
||||
("stripe live key", re.compile(r"\b[rs]k_live_[A-Za-z0-9]{20,}")),
|
||||
("google api key", re.compile(r"\bAIza[0-9A-Za-z_-]{35}(?![0-9A-Za-z_-])")),
|
||||
# Twilio (Windy Text 10-01: a live auth token sat in test files for months). An auth token
|
||||
# is a bare 32-hex with no prefix, so it is only caught when ASSIGNED to a secret-ish name.
|
||||
("twilio sid/api key", re.compile(r"\b(?:AC|SK)[0-9a-f]{32}\b")),
|
||||
("32-hex secret assignment", re.compile(
|
||||
r"(?i)\b[a-z0-9_.-]*(?:token|secret|key|password)[a-z0-9_.-]*[\"']?\s*[:=]\s*[\"']?(?P<v>(?<![0-9a-f])[0-9a-f]{32}(?![0-9a-f]))")),
|
||||
("pypi token", re.compile(r"\bpypi-AgE[A-Za-z0-9_-]{50,}")),
|
||||
("private key block", re.compile(r"-----BEGIN (?:RSA |EC |DSA |OPENSSH |ENCRYPTED )?PRIVATE KEY-----")),
|
||||
]
|
||||
|
||||
# git grep -E (POSIX ERE) prefilter: cheap superset of PATTERNS.
|
||||
PREFILTER = ("[0-9]{8,10}:[A-Za-z0-9_-]{35}|gh[pousr]_[A-Za-z0-9]{36}|github_pat_|(AKIA|ASIA)[0-9A-Z]{16}"
|
||||
"|xox[abprs]-|sk-ant-|sk-[A-Za-z0-9_-]{32}|sk-proj-|[rs]k_live_|AIza[0-9A-Za-z_-]{35}"
|
||||
"|-----BEGIN [A-Z ]*PRIVATE KEY-----|(AC|SK)[0-9a-f]{32}|[0-9a-fA-F]{32}|pypi-AgE")
|
||||
|
||||
|
||||
def h8(value: str | bytes) -> str:
|
||||
return hashlib.sha256(value.encode() if isinstance(value, str) else value).hexdigest()[:8]
|
||||
|
||||
|
||||
def find(text: str) -> list[tuple[str, str]]:
|
||||
"""[(kind, hash8)] for every secret-shaped string in `text`. Values never leave."""
|
||||
out = []
|
||||
for kind, rx in PATTERNS:
|
||||
for m in rx.finditer(text):
|
||||
out.append((kind, h8(m.group('v') if 'v' in rx.groupindex else m.group(0))))
|
||||
return out
|
||||
107
scripts/sync_from_github.sh
Executable file
107
scripts/sync_from_github.sh
Executable file
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 1 sync: GitHub is the source of truth, Windy Git follows.
|
||||
#
|
||||
# ── Why this direction, and why the other one was wrong ────────────────────
|
||||
#
|
||||
# On 2026-08-13 nine repos were migrated writable with push-mirrors pointed AT
|
||||
# GitHub. That was premature: a dozen agent sessions on the Mac mini are pushing
|
||||
# to GitHub continuously, so GitHub — not Windy Git — is where the current work
|
||||
# actually lives. A push-mirror force-updates refs, so on its 8-hour timer it
|
||||
# would have pushed Windy Git's stale copy over live work, silently, with no
|
||||
# conflict to notice. The mirrors were removed before the first timer fired.
|
||||
#
|
||||
# This script is the correct Phase 1: pull from GitHub, push into Windy Git.
|
||||
#
|
||||
# Mac mini agents ──push──▶ GitHub ──this script──▶ Windy Git ──▶ CI on Veron
|
||||
#
|
||||
# **It requires nothing from anyone.** No remote changes, no coordination, no
|
||||
# "everybody stop pushing for a minute." Agents keep working exactly as they are
|
||||
# and CI starts running on 24 cores.
|
||||
#
|
||||
# Windy Git is force-updated on purpose. In Phase 1 it holds nothing anyone
|
||||
# depends on, so GitHub always wins and there is no merge to reconcile — which
|
||||
# is the entire point of not flipping direction until a repo is quiet.
|
||||
#
|
||||
# Phase 2, per repo, only when that repo is idle: point its agents at Windy Git,
|
||||
# drop it from REPOS here, and add a push-mirror back to GitHub. One repo at a
|
||||
# time. Never a big-bang cutover across a dozen live sessions.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
: "${GITHUB_TOKEN:?GITHUB_TOKEN required}"
|
||||
: "${GITEA_ADMIN_TOKEN:?GITEA_ADMIN_TOKEN required}"
|
||||
GH_OWNER="${GITHUB_OWNER:-sneakyfree}"
|
||||
WG="${WG_HOST:-app.windygit.com}"
|
||||
WG_OWNER="${WINDYGIT_OWNER:-windyadmin}"
|
||||
WORK="${SYNC_WORK:-/srv/windygit/sync}"
|
||||
FAILED=0
|
||||
|
||||
# Repos Windy Git tracks FROM GitHub. Remove a repo from this list at the moment
|
||||
# it flips to Windy-Git-first, or the sync will fight its authors and win.
|
||||
REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent windy-git windy-chat windy-mail windy-connect windy-drops windy-code-web windy-code windy-traveler windy-translate windytranslate-site windytraveler-site windy-hand windy-cloud-domains windy-cloud-vps windytalk windy-pro windy-inbox windy-text windy-call windy-cell windy-hand-site windy-calendar-site}"
|
||||
|
||||
# Repos whose TAGS must not reach Windy Git. A tag push fires `on: push: tags`
|
||||
# workflows; windy-pro's build-electron is a matrix over ubuntu/macos/windows-
|
||||
# latest, labels no runner here has, so every leg would queue forever (and
|
||||
# queued jobs are invisible in /actions/tasks). Releases are built elsewhere.
|
||||
NO_TAGS="${SYNC_NO_TAGS:-windy-pro}"
|
||||
|
||||
# `archive/*` branches never reach Windy Git (negative refspec, git >= 2.29).
|
||||
# They are off-machine safety copies of unpushed work (one-repo doctrine), not
|
||||
# work in progress: GitHub holds them, and CI time on them is waste.
|
||||
|
||||
mkdir -p "$WORK"
|
||||
log() { printf '[sync %s] %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }
|
||||
|
||||
for r in $REPOS; do
|
||||
bare="$WORK/${r}.git"
|
||||
if [[ ! -d "$bare" ]]; then
|
||||
git clone --quiet --bare "https://x-access-token:${GITHUB_TOKEN}@github.com/${GH_OWNER}/${r}.git" "$bare" 2>/dev/null \
|
||||
|| { log "FAILED initial clone of $r"; FAILED=1; continue; }
|
||||
fi
|
||||
|
||||
# +refs/heads/* — branches only, deliberately.
|
||||
#
|
||||
# `--mirror` would also carry refs/pull/* (GitHub's read-only PR refs, which
|
||||
# Gitea rejects) and every remote-tracking ref, turning a working sync into a
|
||||
# wall of errors that hides the one that matters.
|
||||
if ! git --git-dir="$bare" fetch --quiet --prune origin '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*' 2>/dev/null; then
|
||||
log "FAILED fetch $r"; FAILED=1; continue
|
||||
fi
|
||||
|
||||
before="$(git --git-dir="$bare" rev-parse HEAD 2>/dev/null || echo none)"
|
||||
|
||||
if git --git-dir="$bare" push --quiet --force \
|
||||
"https://${WG_OWNER}:${GITEA_ADMIN_TOKEN}@${WG}/${WG_OWNER}/${r}.git" \
|
||||
'+refs/heads/*:refs/heads/*' '^refs/heads/archive/*' $([[ " $NO_TAGS " == *" $r "* ]] || echo '+refs/tags/*:refs/tags/*') 2>/dev/null; then
|
||||
log "$r ok (${before:0:7})"
|
||||
else
|
||||
log "FAILED push $r -> windy git"; FAILED=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Jobs that name labels no runner has (ubuntu/macos/windows-latest) would wait
|
||||
# forever and invisibly; cancel them after 30 min. Never fails the sync.
|
||||
# Both DB steps go through `docker exec`, which hangs outright while the host
|
||||
# is in an IO stall (09-23: data2 SMR cliff wedged this sync for 10+ min and
|
||||
# stopped mirroring + the bridge for every lane). They are optional; mirroring
|
||||
# and the bridge are not. Bound them so a stuck exec costs one step, not the run.
|
||||
timeout -k 10 120 bash "$(dirname "$0")/cancel_unrunnable.sh" || log "janitor failed or timed out (non-fatal)"
|
||||
|
||||
# Private repos can't run GitHub Actions; mirror their open PRs here so CI
|
||||
# fires, and post the verdicts back to GitHub as commit statuses.
|
||||
if ! python3 "$(dirname "$0")/pr_status_bridge.py"; then
|
||||
log "FAILED pr status bridge"; FAILED=1
|
||||
fi
|
||||
|
||||
# Runner guard (Boss 10-01): PUBLIC sneakyfree repos have self-hosted GitHub runners on Veron.
|
||||
# A PR that changes a workflow so a stranger's code could reach one gets a red
|
||||
# windy-git/runner-guard status. Each status is posted once; never fails the sync.
|
||||
timeout -k 10 120 python3 "$(dirname "$0")/runner_guard.py" pr --post || log "runner-guard failed or timed out (non-fatal)"
|
||||
|
||||
# CI telemetry -> admin.windyword.ai (shapes declared with Windy Telemetry 40).
|
||||
# Sends nothing until WINDYGIT_TELEMETRY_TOKEN is set; never fails the sync.
|
||||
timeout -k 10 180 python3 "$(dirname "$0")/telemetry_emit.py" || log "telemetry emit failed or timed out (non-fatal)"
|
||||
|
||||
[[ "$FAILED" -ne 0 ]] && { log "COMPLETED WITH FAILURES"; exit 1; }
|
||||
log "all repos in step with GitHub"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user