Compare commits
40 Commits
db055a1922
...
telemetry-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e3b69fa759 | ||
|
|
4acf50d9ef | ||
|
|
b7a7e94df0 | ||
| c83f808a60 | |||
| eb27e63db3 | |||
| 1c3b5b0638 | |||
| 90643fe48e | |||
| 00ec963f82 | |||
| b5e4eaf57a | |||
| baaa542bae | |||
| 0634a6cb1b | |||
| 1a171eabd5 | |||
| 28c31236b8 | |||
| cd5967031b | |||
| f246417095 | |||
| 50c1464043 | |||
| 7a63f90da3 | |||
| b8f97f0731 | |||
| 95c33c8004 | |||
| d5181f1c6d | |||
| e6530d3171 | |||
| 419443573a | |||
| 4a34b35441 | |||
| dfe5543eda | |||
| 40cb455d0d | |||
| 8c404eb410 | |||
| 5b16114b98 | |||
| 18ea9a4686 | |||
| 32e8ac8474 | |||
| 8e9fa3116c | |||
| 74ad4950b2 | |||
| e7bbf9af51 | |||
| 45686283be | |||
| e4a15869c0 | |||
| dcf9286f16 | |||
| 1b09b9b0d3 | |||
| 390c1e7479 | |||
| 2b30b0ac99 | |||
| cd7dd9b7ae | |||
| 9fc27eabd6 |
14
AGENTS.md
14
AGENTS.md
@@ -2,11 +2,17 @@
|
||||
|
||||
Read this before touching anything. Then read `DNA_STRAND_MASTER_PLAN.md`, which is the source of truth.
|
||||
|
||||
## Current state
|
||||
## Current state (2026-09-23)
|
||||
|
||||
**GENESIS.** No code. No `make dev` yet — building it is codon **G0.8**.
|
||||
**LIVE on Veron 1** — `app.windygit.com` (Gitea 1.24.6, Windy SSO only),
|
||||
`api.windygit.com` (our plane: humans via hub JWKS, agents via Eternitas EPT),
|
||||
`models.windygit.com`. Strands G0–G5, G7, G11 done; see the plan for the rest.
|
||||
|
||||
The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Cloudflare Tunnel), then **G2** (Gitea, stock and branded), then **G3** (identity), then **G4** (storage). G0–G4 are sequential. G5–G12 are concurrent once G4 lands.
|
||||
It is also **the permanent CI for the private platform repos** (GitHub Actions
|
||||
cannot run on them): `scripts/sync_from_github.sh` + `scripts/pr_status_bridge.py`,
|
||||
onboarding in `docs/CUTOVER.md`, operations in `docs/RUNBOOK-VERON.md`.
|
||||
|
||||
Standing dev checkout: **OC5 `~/windy-git`**. Deploy copy: Veron `/srv/windygit/src`.
|
||||
|
||||
## The rules that will get you reverted if you break them
|
||||
|
||||
@@ -28,7 +34,7 @@ The next work is Strand **G0** (cell substrate), then **G1** (Veron 1 host + Clo
|
||||
- Errors are 4-field repair pointers: `{code, speak, machine_cause, remediation_tool}`. No exceptions, including validation errors.
|
||||
- Every tool response carries `state_proof` + `next_actions`.
|
||||
- Telemetry `actor_type` comes from the enum `{human, agent, system}`. **`'service'` is not legal** — it 422s and silently drops the whole batch. A sibling service is losing telemetry to exactly this today.
|
||||
- Runner labels are explicit and pinned. **`ubuntu-latest` is banned** — all four `windy-registry` workflows use it and every run fails.
|
||||
- Runner labels are explicit and pinned: `[self-hosted, linux, x64]` or `veron-1`. **`ubuntu-latest` is banned** — no runner here has it, so the job queues forever.
|
||||
|
||||
## Membrane
|
||||
|
||||
|
||||
@@ -81,7 +81,7 @@ Numbered because code cites them. Changing one requires an ADR that names it.
|
||||
|
||||
1. **I-1 · Gitea is a component, never a merged tree.** Our code lives in our services and calls Gitea's REST API. Any patch to Gitea source lives in `patches/` as a numbered, rebasable diff with a one-line justification, and `make check` fails if `patches/` grows past **3** files without an ADR.
|
||||
2. **I-2 · The membrane is ENUMERATED.**
|
||||
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo).
|
||||
**Calls out:** `windy-cloud` kernel `GET /api/v1/storage/objects` + `HEAD` (read user objects to version them) · `windy-cloud` `POST /api/v1/storage/quota/check` · `eternitas` `GET /api/v1/trust/{passport}` (band + allowed_actions) · `eternitas` `GET /api/v1/registry/{passport}/integrity` · `account-server` OIDC discovery + JWKS · `windy-cloud-sites` `POST /api/v1/sites/{id}/versions` (publish docs from a repo). · windy-admin ledger `POST https://admin.windyword.ai/v1/events` (field telemetry, 2026-09-23: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed` — shapes declared with the ledger owner first; no content, no passports, no emails)
|
||||
**Calls in:** `POST /internal/repo-from-folder` (Cloud portal: git-enable a folder) · `POST /internal/mirror-status` (ops).
|
||||
**Events out:** `repo.created`, `repo.pushed`, `release.published`, `model.published`, `ci.completed`.
|
||||
**Events in:** `passport.revoked` (fail-closed), `storage.quota.exceeded`, `identity.created`.
|
||||
|
||||
@@ -22,7 +22,7 @@ boot guard in `api/app/main.py` that refuses to start there in production.
|
||||
| 8600 | `windy-git-api` — our plane |
|
||||
| **3080** | Gitea — host 3000 and 3300 are taken by resident projects on Veron 1 |
|
||||
| 5432 | Postgres |
|
||||
| 2000 | cloudflared metrics (probe target) |
|
||||
| 2001 | cloudflared metrics — NOT 2000: `cornercall-tunnel` (another project) takes 2000, and a metrics bind failure kills the whole tunnel |
|
||||
|
||||
## Ingress — Cloudflare Tunnel `windy-git`
|
||||
|
||||
|
||||
@@ -27,6 +27,8 @@ from fastapi import Header, Request
|
||||
from api.app.config import Settings
|
||||
from api.app.ept import EptInvalid, looks_like_ept, verify_ept
|
||||
from api.app.errors import RepairPointer, passport_unresolvable
|
||||
from api.app.hub_jwt import HubTokenInvalid, verify_hub_token
|
||||
from api.app.telemetry import synthetic_headers
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -124,7 +126,7 @@ async def resolve_passport(settings: Settings, passport: str) -> tuple[str, tupl
|
||||
)
|
||||
|
||||
url = f"{settings.eternitas_base_url}/api/v1/trust/{passport}"
|
||||
headers = {"X-API-Key": settings.eternitas_platform_api_key}
|
||||
headers = {"X-API-Key": settings.eternitas_platform_api_key, **synthetic_headers()}
|
||||
last_status = 0
|
||||
for attempt in range(3):
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(8.0, connect=3.0)) as client:
|
||||
@@ -239,22 +241,29 @@ async def get_caller(
|
||||
allowed_actions=actions,
|
||||
)
|
||||
|
||||
# --- human (account-server RS256) -------------------------------------
|
||||
if settings.is_production and settings.require_verified_jwt:
|
||||
# I-8, applied to ourselves. G3.2's JWKS verifier is not written yet, and
|
||||
# an unverified JWT is an authentication bypass rather than a shortcut.
|
||||
# Refusing is the only honest answer until the verifier exists.
|
||||
raise RepairPointer(
|
||||
status_code=503,
|
||||
code="human_signin_not_ready",
|
||||
speak="Signing in isn't switched on yet. Nothing you have is affected.",
|
||||
machine_cause=(
|
||||
"JWKS verification (G3.2) is not implemented; refusing to accept "
|
||||
"an unverified human token in production"
|
||||
),
|
||||
remediation_tool=None,
|
||||
)
|
||||
# --- human (hub RS256 access token, G3.2) ------------------------------
|
||||
# Production ALWAYS verifies, whatever require_verified_jwt says: the flag
|
||||
# only exists to let local dev run against unsigned fixture tokens.
|
||||
if settings.require_verified_jwt or settings.is_production:
|
||||
try:
|
||||
human = verify_hub_token(
|
||||
token,
|
||||
settings.account_server_base_url,
|
||||
issuers=tuple(settings.hub_issuers),
|
||||
audiences=tuple(settings.hub_audiences),
|
||||
require_aud=settings.hub_require_aud,
|
||||
)
|
||||
except HubTokenInvalid as exc:
|
||||
raise RepairPointer(
|
||||
status_code=401,
|
||||
code="token_invalid",
|
||||
speak="We couldn't confirm that sign-in. Try signing in again.",
|
||||
machine_cause=f"hub token verification failed: {exc}",
|
||||
remediation_tool=None,
|
||||
) from exc
|
||||
return Caller(actor_type=ActorType.human, identity_id=human.identity_id)
|
||||
|
||||
# Local dev only (require_verified_jwt=False outside production).
|
||||
identity_id = _unverified_claim(token, "windy_identity_id") or _unverified_claim(token, "sub")
|
||||
if not identity_id:
|
||||
raise RepairPointer(
|
||||
@@ -274,10 +283,8 @@ def _unverified_claim(token: str, claim: str) -> str | None:
|
||||
on the result re-establishes trust independently: an agent's authority comes
|
||||
from a live Eternitas trust lookup, never from the token's own assertions.
|
||||
|
||||
⚠️ Full RS256/ES256 JWKS verification for the human path lands in G3.2's
|
||||
verifier and MUST be in place before `api.windygit.com` accepts a human
|
||||
token from outside. Until then the human path is reachable only from inside
|
||||
the tunnel, and `settings.require_verified_jwt` refuses it in production.
|
||||
Humans are verified by hub_jwt.verify_hub_token (G3.2); this reader backs
|
||||
only the local-dev path, which production never takes.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
|
||||
@@ -53,16 +53,33 @@ class Settings(BaseSettings):
|
||||
|
||||
# ---- account-server OIDC (human identity) -----------------------------
|
||||
account_server_base_url: str = "https://account.windyword.ai"
|
||||
# G3.2 — what a hub ACCESS token must say about itself (see hub_jwt.py).
|
||||
# Token contract v1 (lane 8c, 2026-09-23): access tokens may carry either
|
||||
# issuer. id_tokens are kept out by `type` + `windy_identity_id` + aud, not
|
||||
# by issuer.
|
||||
hub_issuers: list[str] = ["windy-identity", "https://account.windyword.ai"]
|
||||
# Contract v1: aud is an ARRAY; first-party tokens list every product, and
|
||||
# Windy Git's entry is `windy_git` (underscore). ⚠️ NEVER add "windy-git"
|
||||
# (hyphen): that is Gitea's OIDC client_id, so an id_token minted for the
|
||||
# forge would carry it and pass as a bearer here.
|
||||
hub_audiences: list[str] = ["windy_git"]
|
||||
# Flip to True once the hub emits aud on every access token.
|
||||
hub_require_aud: bool = False
|
||||
|
||||
# ---- field telemetry (admin.windyword.ai ledger) ----------------------
|
||||
# Unset token = nothing sent, nothing buffered. The token lives in the
|
||||
# root-only /etc/windygit/telemetry.env on Veron, never in the repo.
|
||||
windygit_telemetry_token: str = ""
|
||||
telemetry_ingest_url: str = "https://admin.windyword.ai/v1/events"
|
||||
|
||||
# Internal callers (the Cloud portal calling /internal/*). A first-class
|
||||
# caller class, not a bypass: unset means service calls are REFUSED.
|
||||
service_token: str = ""
|
||||
|
||||
# ⚠️ FAIL-CLOSED GATE. Full RS256/ES256 JWKS verification lands in G3.2.
|
||||
# Until it does, the human token path must not be reachable in production —
|
||||
# accepting an unverified JWT is not a shortcut, it is an authentication
|
||||
# bypass. Agents are unaffected: their authority comes from a live Eternitas
|
||||
# trust lookup, not from anything the token asserts about itself.
|
||||
# ⚠️ FAIL-CLOSED GATE. Human tokens are verified against the hub's JWKS
|
||||
# (G3.2, hub_jwt.py). False only enables the unverified local-dev path, and
|
||||
# production verifies regardless — an unverified JWT is a bypass, not a
|
||||
# shortcut.
|
||||
require_verified_jwt: bool = True
|
||||
|
||||
# ---- storage law (I-3, G4.4) ------------------------------------------
|
||||
|
||||
133
api/app/hub_jwt.py
Normal file
133
api/app/hub_jwt.py
Normal file
@@ -0,0 +1,133 @@
|
||||
"""Human token verification (G3.2) — hub access tokens from account.windyword.ai.
|
||||
|
||||
Until this existed the human path refused every token in production (503
|
||||
`human_signin_not_ready`), because reading an unverified JWT's claims is an
|
||||
authentication bypass, not a shortcut. This module is what lets it say yes.
|
||||
|
||||
The token it accepts is the hub's ACCESS token, as observed live 2026-09-23:
|
||||
|
||||
header {alg: RS256, typ: JWT, kid: <published at /.well-known/jwks.json>}
|
||||
claims iss = "windy-identity" (contract v1 also allows the discovery URL)
|
||||
type = "human", exp - iat = 900 s
|
||||
sub = per-row user id ← NOT the cross-product identity
|
||||
windy_identity_id = the Windy Account UUID (what Gitea's OIDC links on)
|
||||
no `aud` yet
|
||||
|
||||
What it refuses, by construction:
|
||||
|
||||
* **Anything but RS256.** One algorithm, never a list. Closes `alg: none` and
|
||||
HS256-with-the-public-key confusion.
|
||||
* **An unknown `kid`**, a wrong issuer, an expired token — library-checked.
|
||||
* **An id_token used as a bearer.** id_tokens prove a login happened to a
|
||||
relying party (for the forge: aud `windy-git`), not that this caller may act
|
||||
here. They carry no `type` and no `windy_identity_id`, and their aud is a
|
||||
client id, not the product name `windy_git` — any one of the three refuses.
|
||||
* **A non-human `type`.** An agent's authority comes from its EPT and a live
|
||||
Eternitas lookup, never from a hub token dressed as a person.
|
||||
* **A token with no `windy_identity_id`.** `sub` is a different namespace (the
|
||||
per-row user id); falling back to it would silently mint identities that
|
||||
match nothing Gitea knows.
|
||||
|
||||
`aud` (token contract v1, lane 8c): an array; first-party tokens list every
|
||||
product and Windy Git's is `windy_git`. Optional until the hub emits it; when
|
||||
present it MUST include `windy_git`.
|
||||
`hub_require_aud=True` makes it mandatory — flip it once the hub emits it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import jwt
|
||||
from jwt import PyJWKClient
|
||||
|
||||
ALGORITHMS = ["RS256"] # exactly one. Never widen this list.
|
||||
|
||||
_jwks_client: PyJWKClient | None = None
|
||||
_jwks_url: str | None = None
|
||||
|
||||
|
||||
class HubTokenInvalid(Exception):
|
||||
"""Not a valid, currently-signed hub access token for a human."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VerifiedHuman:
|
||||
identity_id: str
|
||||
email: str | None
|
||||
expires_at: int | None
|
||||
|
||||
|
||||
def _client(base_url: str) -> PyJWKClient:
|
||||
"""Cached JWKS client; refetches on an unknown kid so rotation self-heals."""
|
||||
global _jwks_client, _jwks_url
|
||||
url = f"{base_url.rstrip('/')}/.well-known/jwks.json"
|
||||
if _jwks_client is None or _jwks_url != url:
|
||||
_jwks_client = PyJWKClient(url, cache_keys=True, lifespan=300)
|
||||
_jwks_url = url
|
||||
return _jwks_client
|
||||
|
||||
|
||||
def verify_hub_token(
|
||||
token: str,
|
||||
base_url: str,
|
||||
*,
|
||||
issuers: tuple[str, ...],
|
||||
audiences: tuple[str, ...],
|
||||
require_aud: bool,
|
||||
signing_key=None,
|
||||
) -> VerifiedHuman:
|
||||
"""Verify a hub access token. Raises HubTokenInvalid on ANY doubt.
|
||||
|
||||
`signing_key` exists for tests only (a locally generated key, no network).
|
||||
"""
|
||||
try:
|
||||
key = (
|
||||
signing_key
|
||||
if signing_key is not None
|
||||
else _client(base_url).get_signing_key_from_jwt(token).key
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - unknown kid, unreachable JWKS, malformed
|
||||
raise HubTokenInvalid(f"no usable signing key: {type(exc).__name__}: {exc}") from exc
|
||||
|
||||
try:
|
||||
claims = jwt.decode(
|
||||
token,
|
||||
key,
|
||||
algorithms=ALGORITHMS,
|
||||
issuer=list(issuers),
|
||||
options={
|
||||
"require": ["iss", "exp", "iat"],
|
||||
"verify_signature": True,
|
||||
"verify_exp": True,
|
||||
"verify_iss": True,
|
||||
# Checked by hand below: PyJWT rejects any token CARRYING aud
|
||||
# when no audience is passed, which would break the moment the
|
||||
# hub starts emitting it — the exact trap the SSO matrix names.
|
||||
"verify_aud": False,
|
||||
},
|
||||
)
|
||||
except jwt.PyJWTError as exc:
|
||||
raise HubTokenInvalid(f"{type(exc).__name__}: {exc}") from exc
|
||||
|
||||
aud = claims.get("aud")
|
||||
if aud is None:
|
||||
if require_aud:
|
||||
raise HubTokenInvalid("token carries no aud and hub_require_aud is on")
|
||||
else:
|
||||
presented = {aud} if isinstance(aud, str) else set(aud) if isinstance(aud, list) else set()
|
||||
if not presented & set(audiences):
|
||||
raise HubTokenInvalid(f"aud {sorted(presented)} does not name Windy Git")
|
||||
|
||||
# REQUIRED, not defaulted: id_tokens carry no `type`, and this is one of the
|
||||
# two claims (with windy_identity_id) that keep them from acting as bearers.
|
||||
if claims.get("type") != "human":
|
||||
raise HubTokenInvalid(f"token type {claims.get('type')!r} is not a human access token")
|
||||
|
||||
identity = claims.get("windy_identity_id") or claims.get("windyIdentityId")
|
||||
if not isinstance(identity, str) or not identity.strip():
|
||||
raise HubTokenInvalid("token carries no windy_identity_id")
|
||||
|
||||
return VerifiedHuman(
|
||||
identity_id=identity, email=claims.get("email"), expires_at=claims.get("exp")
|
||||
)
|
||||
@@ -6,11 +6,13 @@ component and is reached only over its REST API (D-2 / I-1).
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import socket
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
@@ -25,6 +27,7 @@ from api.app.providers.registry import (
|
||||
R2Provider,
|
||||
)
|
||||
from api.app.routes import health, repos, webhooks
|
||||
from api.app.telemetry import SYNTHETIC, Telemetry, caller_class, is_synthetic
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
@@ -44,9 +47,7 @@ def _refuse_kit_zero(settings) -> None:
|
||||
if not settings.is_production:
|
||||
return
|
||||
try:
|
||||
local_ips = {
|
||||
info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)
|
||||
}
|
||||
local_ips = {info[4][0] for info in socket.getaddrinfo(socket.gethostname(), None)}
|
||||
except socket.gaierror:
|
||||
return
|
||||
if settings.kit0_host in local_ips:
|
||||
@@ -98,8 +99,23 @@ async def lifespan(app: FastAPI):
|
||||
# systemd Restart=always, plus the runbook's `systemctl status`.
|
||||
]
|
||||
|
||||
telemetry = Telemetry(
|
||||
settings.telemetry_ingest_url,
|
||||
settings.windygit_telemetry_token,
|
||||
environment=settings.environment,
|
||||
commit_sha=info.commit_sha,
|
||||
version=info.version,
|
||||
)
|
||||
app.state.telemetry = telemetry
|
||||
telemetry.boot()
|
||||
await telemetry.flush()
|
||||
task = asyncio.create_task(telemetry.run()) if telemetry.enabled else None
|
||||
|
||||
yield
|
||||
|
||||
if task is not None:
|
||||
task.cancel()
|
||||
await telemetry.flush()
|
||||
if engine is not None:
|
||||
await engine.dispose()
|
||||
|
||||
@@ -119,8 +135,44 @@ app.include_router(repos.router)
|
||||
app.include_router(webhooks.router)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def _count_requests(request: Request, call_next):
|
||||
"""Heartbeat counts (requests, 4xx/5xx, refusals, p95). Never raises."""
|
||||
start = time.perf_counter()
|
||||
marker = SYNTHETIC.set(is_synthetic(request.headers))
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
SYNTHETIC.reset(marker)
|
||||
tel = getattr(request.app.state, "telemetry", None)
|
||||
if tel is not None:
|
||||
tel.record_request(
|
||||
response.status_code,
|
||||
(time.perf_counter() - start) * 1000,
|
||||
refused=getattr(request.state, "refused", False),
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(RepairPointer)
|
||||
async def _repair_pointer_handler(_, exc: RepairPointer) -> JSONResponse:
|
||||
async def _repair_pointer_handler(request: Request, exc: RepairPointer) -> JSONResponse:
|
||||
tel = getattr(request.app.state, "telemetry", None)
|
||||
detail = exc.detail if isinstance(exc.detail, dict) else {}
|
||||
code = detail.get("code")
|
||||
if tel is not None and code in tel.auth_codes:
|
||||
# A refusal is a failure row (field-visibility rule 1). The caller is
|
||||
# unauthenticated by definition, so: system actor, no actor_id, and
|
||||
# the route TEMPLATE, never the concrete path.
|
||||
request.state.refused = True
|
||||
route = request.scope.get("route")
|
||||
tel.auth_failed(
|
||||
code=code,
|
||||
http_status=exc.status_code,
|
||||
caller=caller_class(request.headers),
|
||||
route=getattr(route, "path", None),
|
||||
upstream_status=getattr(exc, "upstream_status", None),
|
||||
synthetic=is_synthetic(request.headers),
|
||||
)
|
||||
return JSONResponse(status_code=exc.status_code, content=exc.detail)
|
||||
|
||||
|
||||
|
||||
@@ -105,7 +105,7 @@ class DatabaseProvider(Provider):
|
||||
|
||||
|
||||
# TunnelProvider was removed deliberately. See the note in main.py: cloudflared
|
||||
# binds 127.0.0.1:2000 on the HOST, and this process runs in a container whose
|
||||
# binds 127.0.0.1:2001 on the HOST, and this process runs in a container whose
|
||||
# only route to the host is the bridge gateway (172.17.0.1), where nothing is
|
||||
# listening. Binding the metrics endpoint wider would fix the probe and make a
|
||||
# metrics bind failure able to take down ingress -- a worse trade than losing
|
||||
|
||||
@@ -20,6 +20,7 @@ import httpx
|
||||
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer, provider_unconfigured
|
||||
from api.app.telemetry import synthetic_headers
|
||||
|
||||
_TIMEOUT = httpx.Timeout(20.0, connect=5.0)
|
||||
|
||||
@@ -36,6 +37,7 @@ class GiteaClient:
|
||||
return {
|
||||
"Authorization": f"token {self._s.gitea_admin_token}",
|
||||
"Content-Type": "application/json",
|
||||
**synthetic_headers(), # end-to-end synthetic convention (Telemetry UPDATE 4)
|
||||
}
|
||||
|
||||
async def _request(self, method: str, path: str, **kw: Any) -> httpx.Response:
|
||||
|
||||
255
api/app/telemetry.py
Normal file
255
api/app/telemetry.py
Normal file
@@ -0,0 +1,255 @@
|
||||
"""Field telemetry to the admin ledger (admin.windyword.ai) — step 2, 2026-09-23.
|
||||
|
||||
Shapes are DECLARED with Telemetry Boss (the ledger owner); the server
|
||||
quarantines any row that doesn't match, so never add a key or a code here
|
||||
without re-declaring it first:
|
||||
|
||||
service.boot once per process start {commit_sha, version, environment}
|
||||
service.health hourly, in-process interval_s, uptime_s, requests,
|
||||
errors_5xx, errors_4xx,
|
||||
refusals_4xx, p95_ms
|
||||
forge.auth.failed every refused request {code, http_status, caller,
|
||||
route?, upstream_status?}
|
||||
|
||||
Refusals come first: a refused caller is the most expensive silent failure
|
||||
("the button did nothing"). An UNAUTHENTICATED caller has no trustworthy id, so
|
||||
per the all-lanes actor rule the row is actor_type "system", no actor_id, and
|
||||
the caller class goes in metadata.caller.
|
||||
|
||||
Privacy: codes, statuses, route TEMPLATES, counts, durations. Never a passport
|
||||
number, an email, a token fragment or a concrete path with names in it.
|
||||
|
||||
No token → nothing is sent and nothing is buffered. A failed flush keeps the
|
||||
rows (bounded) and retries on the next tick; it never raises into a request.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextvars
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import urllib.request
|
||||
from datetime import UTC, datetime
|
||||
|
||||
log = logging.getLogger("windy-git.telemetry")
|
||||
|
||||
PLATFORM, SERVICE = "windy-git", "api"
|
||||
FLUSH_EVERY_S = 60
|
||||
HEALTH_EVERY_S = 3600
|
||||
MAX_BUFFER = 5000
|
||||
MAX_LATENCY_SAMPLES = 20000
|
||||
|
||||
# The declared forge.auth.failed code enum (Telemetry Boss, 2026-09-23). A code
|
||||
# outside this set is NOT a refusal row — it counts in errors_* instead.
|
||||
AUTH_CODES = frozenset(
|
||||
{
|
||||
"not_signed_in",
|
||||
"token_invalid",
|
||||
"token_unrecognised",
|
||||
"ept_invalid",
|
||||
"passport_revoked",
|
||||
"passport_unresolvable",
|
||||
"agent_read_only",
|
||||
"agent_rate_limited",
|
||||
"quota_exceeded",
|
||||
"trust_unavailable",
|
||||
"throttle_unavailable",
|
||||
"service_token_invalid",
|
||||
"service_auth_unconfigured",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _iso(epoch: float) -> str:
|
||||
return datetime.fromtimestamp(epoch, UTC).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
# Ecosystem convention (Telemetry UPDATE 4): synthetic traffic travels END TO
|
||||
# END. Originators (canaries, probes, journeys) send `X-Windy-Synthetic: 1`;
|
||||
# every service marks all of that request's rows synthetic:true AND forwards the
|
||||
# header on every downstream call. Absent = real. Never strip it, never set it
|
||||
# on real traffic. The label separates rows — it never suppresses them.
|
||||
SYNTHETIC: contextvars.ContextVar[bool] = contextvars.ContextVar("windy_synthetic", default=False)
|
||||
|
||||
|
||||
def is_synthetic(headers) -> bool:
|
||||
return bool((headers.get("x-windy-synthetic") or "").strip())
|
||||
|
||||
|
||||
def synthetic_headers() -> dict:
|
||||
"""Merge into every downstream request made while serving this one."""
|
||||
return {"X-Windy-Synthetic": "1"} if SYNTHETIC.get() else {}
|
||||
|
||||
|
||||
def caller_class(headers) -> str:
|
||||
"""Declared values: anonymous_human | anonymous_agent | unknown."""
|
||||
from api.app.ept import looks_like_ept
|
||||
|
||||
if headers.get("x-service-token"):
|
||||
return "unknown"
|
||||
auth = headers.get("authorization") or ""
|
||||
if not auth.lower().startswith("bearer "):
|
||||
return "unknown"
|
||||
return "anonymous_agent" if looks_like_ept(auth.split(" ", 1)[1].strip()) else "anonymous_human"
|
||||
|
||||
|
||||
class Telemetry:
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
token: str,
|
||||
*,
|
||||
environment: str = "",
|
||||
commit_sha: str | None = None,
|
||||
version: str = "",
|
||||
) -> None:
|
||||
self.url, self.token = url, token
|
||||
self.environment, self.commit_sha, self.version = environment, commit_sha, version
|
||||
self.started = time.time()
|
||||
self.buffer: list[dict] = []
|
||||
self.auth_codes = AUTH_CODES
|
||||
self._reset_window()
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return bool(self.token)
|
||||
|
||||
def _reset_window(self) -> None:
|
||||
self.window_start = time.time()
|
||||
self.requests = self.errors_5xx = self.errors_4xx = self.refusals_4xx = 0
|
||||
self.latencies_ms: list[float] = []
|
||||
# UPDATE 7: rows the ledger quarantined (it still answers 202) and rows
|
||||
# this process lost (buffer overflow). Non-zero = a bug in this emitter.
|
||||
self.quarantined = self.dropped = 0
|
||||
|
||||
# ---- recording (never raises into a request) --------------------------
|
||||
def record_request(self, status: int, duration_ms: float, *, refused: bool = False) -> None:
|
||||
self.requests += 1
|
||||
if status >= 500:
|
||||
self.errors_5xx += 1
|
||||
elif refused:
|
||||
self.refusals_4xx += 1
|
||||
elif status >= 400:
|
||||
self.errors_4xx += 1
|
||||
if len(self.latencies_ms) < MAX_LATENCY_SAMPLES:
|
||||
self.latencies_ms.append(duration_ms)
|
||||
|
||||
def _event(self, event_type: str, metadata: dict, *, ts: float | None = None) -> None:
|
||||
if not self.enabled:
|
||||
return
|
||||
self.buffer.append(
|
||||
{
|
||||
"ts": _iso(ts or time.time()),
|
||||
"platform": PLATFORM,
|
||||
"service": SERVICE,
|
||||
"event_type": event_type,
|
||||
"actor_type": "system",
|
||||
"metadata": metadata,
|
||||
}
|
||||
)
|
||||
if len(self.buffer) > MAX_BUFFER:
|
||||
self.dropped += len(self.buffer) - MAX_BUFFER
|
||||
del self.buffer[: len(self.buffer) - MAX_BUFFER]
|
||||
|
||||
def boot(self) -> None:
|
||||
meta = {"version": self.version, "environment": self.environment}
|
||||
if self.commit_sha: # unknown is absent, never invented (I-12)
|
||||
meta["commit_sha"] = self.commit_sha
|
||||
self._event("service.boot", meta, ts=self.started)
|
||||
|
||||
def auth_failed(
|
||||
self,
|
||||
*,
|
||||
code: str,
|
||||
http_status: int,
|
||||
caller: str,
|
||||
route: str | None = None,
|
||||
upstream_status: int | None = None,
|
||||
synthetic: bool = False,
|
||||
) -> None:
|
||||
if code not in AUTH_CODES:
|
||||
return
|
||||
meta: dict = {
|
||||
"code": code,
|
||||
"http_status": int(http_status),
|
||||
"caller": caller,
|
||||
"synthetic": bool(synthetic),
|
||||
}
|
||||
if route:
|
||||
meta["route"] = route
|
||||
if upstream_status is not None:
|
||||
meta["upstream_status"] = int(upstream_status)
|
||||
self._event("forge.auth.failed", meta)
|
||||
|
||||
def health_row(self) -> dict:
|
||||
now = time.time()
|
||||
meta = {
|
||||
"interval_s": int(now - self.window_start),
|
||||
"uptime_s": int(now - self.started),
|
||||
"requests": self.requests,
|
||||
"errors_5xx": self.errors_5xx,
|
||||
"errors_4xx": self.errors_4xx,
|
||||
"refusals_4xx": self.refusals_4xx,
|
||||
"telemetry_quarantined": self.quarantined,
|
||||
"telemetry_dropped": self.dropped,
|
||||
}
|
||||
if self.latencies_ms: # no traffic = no p95, not a fake 0
|
||||
s = sorted(self.latencies_ms)
|
||||
meta["p95_ms"] = int(s[min(len(s) - 1, int(0.95 * (len(s) - 1) + 0.5))])
|
||||
return meta
|
||||
|
||||
def health(self) -> None:
|
||||
self._event("service.health", self.health_row())
|
||||
self._reset_window()
|
||||
|
||||
# ---- sending ------------------------------------------------------------
|
||||
def _post(self, batch: list[dict]) -> tuple[int, dict]:
|
||||
req = urllib.request.Request(
|
||||
self.url,
|
||||
data=json.dumps({"events": batch}).encode(),
|
||||
method="POST",
|
||||
headers={
|
||||
"Authorization": f"Bearer {self.token}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "windy-git-api-telemetry/1",
|
||||
},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
try:
|
||||
body = json.loads(r.read() or b"{}")
|
||||
except ValueError:
|
||||
body = {}
|
||||
return r.status, body if isinstance(body, dict) else {}
|
||||
|
||||
async def flush(self) -> None:
|
||||
if not self.enabled or not self.buffer:
|
||||
return
|
||||
batch = self.buffer[:500]
|
||||
try:
|
||||
status, body = await asyncio.to_thread(self._post, batch)
|
||||
except Exception as exc: # noqa: BLE001 - telemetry must never take the API down
|
||||
log.warning("telemetry flush failed (%d rows kept): %s", len(self.buffer), exc)
|
||||
return
|
||||
if 200 <= status < 300:
|
||||
del self.buffer[: len(batch)]
|
||||
self.note_quarantine(body)
|
||||
|
||||
def note_quarantine(self, body: dict) -> None:
|
||||
# 202 does NOT mean every row landed: refused rows are dead-lettered.
|
||||
q = body.get("quarantined")
|
||||
if isinstance(q, int) and q > 0:
|
||||
self.quarantined += q
|
||||
log.warning("telemetry: %d row(s) QUARANTINED by the ledger: %s", q,
|
||||
"; ".join(map(str, body.get("rejections") or [])) or "no reason given")
|
||||
|
||||
async def run(self) -> None:
|
||||
"""The one in-process timer: flush every minute, heartbeat every hour."""
|
||||
last_health = time.monotonic()
|
||||
while True:
|
||||
await asyncio.sleep(FLUSH_EVERY_S)
|
||||
if time.monotonic() - last_health >= HEALTH_EVERY_S:
|
||||
self.health()
|
||||
last_health = time.monotonic()
|
||||
await self.flush()
|
||||
169
api/tests/test_hub_jwt.py
Normal file
169
api/tests/test_hub_jwt.py
Normal file
@@ -0,0 +1,169 @@
|
||||
"""G3.2 / I-8 — human tokens are verified, never read (SSO #14, 2026-09-23).
|
||||
|
||||
Behavioral: every case signs a real RS256 token with a locally generated key
|
||||
and drives `get_caller`, so a green run means the gate refuses what it must —
|
||||
not that some string appears in auth.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import time
|
||||
|
||||
import jwt
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
|
||||
from api.app import hub_jwt
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
KEY = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
OTHER = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
IDENTITY = "5e1b9569-7f01-489d-bf14-6fe5a367fa3f"
|
||||
|
||||
|
||||
def _claims(**over):
|
||||
now = int(time.time())
|
||||
c = {
|
||||
"iss": "windy-identity",
|
||||
"type": "human",
|
||||
"sub": "row-id-not-identity",
|
||||
"windy_identity_id": IDENTITY,
|
||||
"email": "grant@example.com",
|
||||
"iat": now,
|
||||
"exp": now + 900,
|
||||
}
|
||||
c.update(over)
|
||||
return {k: v for k, v in c.items() if v is not None}
|
||||
|
||||
|
||||
def _sign(claims, key=KEY, alg="RS256"):
|
||||
return jwt.encode(claims, key, algorithm=alg, headers={"kid": "test"})
|
||||
|
||||
|
||||
class _Req:
|
||||
def __init__(self, settings):
|
||||
self.app = type("A", (), {"state": type("S", (), {"settings": settings})()})()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _local_jwks(monkeypatch):
|
||||
"""The hub's JWKS, served from KEY's public half — no network."""
|
||||
|
||||
class _Key:
|
||||
key = KEY.public_key()
|
||||
|
||||
class _Client:
|
||||
def get_signing_key_from_jwt(self, token):
|
||||
return _Key()
|
||||
|
||||
monkeypatch.setattr(hub_jwt, "_client", lambda base_url: _Client())
|
||||
|
||||
|
||||
async def _caller(token, **settings):
|
||||
from api.app.auth import get_caller
|
||||
|
||||
s = Settings(environment="production", **settings)
|
||||
return await get_caller(_Req(s), authorization=f"Bearer {token}", x_service_token=None)
|
||||
|
||||
|
||||
async def _refused(token, **settings):
|
||||
with pytest.raises(RepairPointer) as exc:
|
||||
await _caller(token, **settings)
|
||||
assert exc.value.status_code == 401 and exc.value.code == "token_invalid"
|
||||
return exc.value
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_genuine_hub_token_is_a_human_named_by_windy_identity_id():
|
||||
c = await _caller(_sign(_claims()))
|
||||
assert c.actor_type == "human"
|
||||
assert c.identity_id == IDENTITY # NOT `sub`, which is the per-row user id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_forged_signature_is_refused():
|
||||
await _refused(_sign(_claims(), key=OTHER))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expired_token_is_refused():
|
||||
await _refused(_sign(_claims(iat=int(time.time()) - 2000, exp=int(time.time()) - 60)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wrong_issuer_and_id_tokens_are_refused():
|
||||
await _refused(_sign(_claims(iss="https://evil.example")))
|
||||
# Contract v1: the discovery-URL issuer is legal for ACCESS tokens...
|
||||
c = await _caller(_sign(_claims(iss="https://account.windyword.ai")))
|
||||
assert c.identity_id == IDENTITY
|
||||
# ...but an id_token minted for the forge (aud = Gitea's client id
|
||||
# "windy-git", no type, sub = identity) must never act as a bearer here.
|
||||
id_token = _claims(
|
||||
iss="https://account.windyword.ai",
|
||||
aud="windy-git",
|
||||
type=None,
|
||||
windy_identity_id=None,
|
||||
sub=IDENTITY,
|
||||
)
|
||||
await _refused(_sign(id_token))
|
||||
await _refused(_sign(dict(id_token, windy_identity_id=IDENTITY, type="human")))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hs256_confusion_is_refused():
|
||||
# The classic forgery: HMAC the token with the PUBLIC key as the secret.
|
||||
pub = KEY.public_key().public_bytes(
|
||||
serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo
|
||||
)
|
||||
header = base64.urlsafe_b64encode(json.dumps({"alg": "HS256", "typ": "JWT"}).encode()).rstrip(
|
||||
b"="
|
||||
)
|
||||
body = base64.urlsafe_b64encode(json.dumps(_claims()).encode()).rstrip(b"=")
|
||||
sig = base64.urlsafe_b64encode(
|
||||
hmac.new(pub, header + b"." + body, hashlib.sha256).digest()
|
||||
).rstrip(b"=")
|
||||
await _refused((header + b"." + body + b"." + sig).decode())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_human_or_missing_type_is_refused():
|
||||
await _refused(_sign(_claims(type="agent")))
|
||||
await _refused(_sign(_claims(type=None)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_windy_identity_is_refused_not_read_from_sub():
|
||||
await _refused(_sign(_claims(windy_identity_id=None)))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aud_is_tolerated_when_it_names_windy_git_and_refused_otherwise():
|
||||
"""PyJWT rejects ANY aud-bearing token when no audience is configured — the
|
||||
trap that would break the day the hub starts emitting aud."""
|
||||
c = await _caller(_sign(_claims(aud=["windy_chat", "windy_git", "windy_mail"])))
|
||||
assert c.identity_id == IDENTITY
|
||||
await _refused(_sign(_claims(aud=["windy_chat"])))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_require_aud_refuses_tokens_without_it():
|
||||
await _refused(_sign(_claims()), hub_require_aud=True)
|
||||
c = await _caller(_sign(_claims(aud=["windy_git"])), hub_require_aud=True)
|
||||
assert c.identity_id == IDENTITY
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_production_verifies_even_if_the_flag_is_off():
|
||||
"""require_verified_jwt=False is a local-dev convenience; production must
|
||||
never take the unverified path."""
|
||||
await _refused(_sign(_claims(), key=OTHER), require_verified_jwt=False)
|
||||
|
||||
|
||||
def test_algorithm_list_is_exactly_rs256():
|
||||
assert hub_jwt.ALGORITHMS == ["RS256"]
|
||||
@@ -308,12 +308,13 @@ def test_g36_trust_client_never_soft_allows():
|
||||
|
||||
def test_g36_unverified_human_jwt_is_refused_in_production():
|
||||
"""I-8 applied to ourselves: an unverified JWT is an authentication bypass,
|
||||
not a shortcut. Until G3.2's JWKS verifier exists, production refuses."""
|
||||
not a shortcut. G3.2's verifier now exists; behavioral proof that forged,
|
||||
expired, mis-issued and mis-audienced tokens are refused lives in
|
||||
test_hub_jwt.py. Here: the gate defaults closed."""
|
||||
from api.app.config import Settings
|
||||
|
||||
assert Settings().require_verified_jwt is True
|
||||
src = (ROOT / "api" / "app" / "auth.py").read_text()
|
||||
assert "human_signin_not_ready" in src
|
||||
assert "windy-git" not in Settings().hub_audiences # Gitea's client_id: id_token confusion
|
||||
|
||||
|
||||
def test_no_auth_bypass_env_var_anywhere():
|
||||
@@ -733,3 +734,21 @@ def test_g23_brand_css_filename_is_versioned():
|
||||
assert m, "brand CSS must carry a version in its FILENAME"
|
||||
assert (ROOT / "deploy" / "branding" / "public" / "assets" / "css"
|
||||
/ f"theme-windy.v{m.group(1)}.css").exists()
|
||||
|
||||
|
||||
def test_backup_never_bundles_credential_repos_to_r2():
|
||||
"""kit-army-config (the lockbox) and the *-soul / anima repos carry
|
||||
credentials; the R2 bundles are plaintext. Behavioural: run the script's
|
||||
own exclusion function against the names."""
|
||||
import subprocess
|
||||
|
||||
script = (ROOT / "scripts" / "backup.sh").read_text()
|
||||
fn = script[script.index('EXCLUDE="'):script.index("cleanup()")]
|
||||
# A file named like a pattern in cwd must not break the match (glob expansion).
|
||||
probe = "cd \"$(mktemp -d)\" && touch x-soul && " + fn + (
|
||||
'for n in kit-army-config anima windy-0-soul kit-0c5-soul herm-0-soul '
|
||||
'soulsafe windy-chat eternitas; do excluded "$n" && echo "X $n" || echo "- $n"; done'
|
||||
)
|
||||
out = subprocess.run(["bash", "-c", probe], capture_output=True, text=True, check=True).stdout
|
||||
skipped = {ln[2:] for ln in out.splitlines() if ln.startswith("X ")}
|
||||
assert skipped == {"kit-army-config", "anima", "windy-0-soul", "kit-0c5-soul", "herm-0-soul"}
|
||||
|
||||
274
api/tests/test_pr_status_bridge.py
Normal file
274
api/tests/test_pr_status_bridge.py
Normal file
@@ -0,0 +1,274 @@
|
||||
"""Behavioral tests for scripts/pr_status_bridge.py.
|
||||
|
||||
The bridge is the ONLY CI signal the private platform repos get on GitHub, so
|
||||
these drive its real functions against fake Gitea/GitHub APIs rather than
|
||||
grepping its source: a status painted green that nobody tested is worse than
|
||||
no status at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"pr_status_bridge", ROOT / "scripts" / "pr_status_bridge.py"
|
||||
)
|
||||
bridge = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(bridge)
|
||||
|
||||
SHA = "a" * 40
|
||||
|
||||
|
||||
def _run(i, wf, job, status, sha=SHA, n=1):
|
||||
return {
|
||||
"id": i,
|
||||
"workflow_id": wf,
|
||||
"name": job,
|
||||
"status": status,
|
||||
"head_sha": sha,
|
||||
"run_number": n,
|
||||
}
|
||||
|
||||
|
||||
class Fake:
|
||||
def __init__(self, runs=(), statuses=(), gh_prs=(), wg_prs=(), workflows=None):
|
||||
self.runs, self.statuses = list(runs), list(statuses)
|
||||
self.workflows = workflows or {} # {path: yaml text} at every commit
|
||||
self.gh_prs, self.wg_prs = list(gh_prs), list(wg_prs)
|
||||
self.posted, self.opened, self.closed = [], [], []
|
||||
|
||||
def gitea(self, method, path, body=None):
|
||||
if "/contents/" in path:
|
||||
want = path.split("/contents/", 1)[1].split("?", 1)[0]
|
||||
if want in self.workflows:
|
||||
return 200, {"content": base64.b64encode(self.workflows[want].encode()).decode()}
|
||||
files = [
|
||||
{"type": "file", "name": k.rsplit("/", 1)[1], "path": k}
|
||||
for k in self.workflows
|
||||
if k.rsplit("/", 1)[0] == want
|
||||
]
|
||||
return (200, files) if files else (404, None)
|
||||
if "/actions/tasks" in path:
|
||||
page = int(path.rsplit("page=", 1)[1])
|
||||
return 200, {"workflow_runs": self.runs[(page - 1) * 50 : page * 50]}
|
||||
if method == "GET" and path.endswith("/pulls?state=open&limit=50"):
|
||||
return 200, self.wg_prs
|
||||
if method == "POST" and path.endswith("/pulls"):
|
||||
self.opened.append(body)
|
||||
return 201, {}
|
||||
if method == "PATCH":
|
||||
self.closed.append(path)
|
||||
return 201, {}
|
||||
raise AssertionError(path)
|
||||
|
||||
def github(self, method, path, body=None):
|
||||
if "/statuses" in path and method == "GET":
|
||||
return 200, self.statuses
|
||||
if "/statuses/" in path and method == "POST":
|
||||
self.posted.append(body)
|
||||
return 201, {}
|
||||
if "/pulls?" in path:
|
||||
return 200, self.gh_prs
|
||||
raise AssertionError(path)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake(monkeypatch):
|
||||
def make(**kw):
|
||||
f = Fake(**kw)
|
||||
monkeypatch.setattr(bridge, "gitea", f.gitea)
|
||||
monkeypatch.setattr(bridge, "github", f.github)
|
||||
return f
|
||||
|
||||
return make
|
||||
|
||||
|
||||
def test_posts_latest_verdict_per_job(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "failure"), _run(2, "ci.yml", "test", "success", n=2)])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/test", "success")]
|
||||
assert f.posted[0]["target_url"].endswith("/actions/runs/2")
|
||||
|
||||
|
||||
def test_unchanged_state_is_not_reposted(fake):
|
||||
f = fake(
|
||||
runs=[_run(1, "ci.yml", "test", "success")],
|
||||
statuses=[{"context": "windy-git/ci/test", "state": "success"}],
|
||||
)
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_skipped_job_is_never_painted_green(fake):
|
||||
f = fake(runs=[_run(1, "substrate-drift.yml", "check", "skipped")])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_other_commits_runs_are_ignored(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "failure", sha="b" * 40)])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_runs_past_the_first_page_are_seen(fake):
|
||||
noise = [_run(100 + i, "drift.yml", "x", "skipped", sha="c" * 40) for i in range(50)]
|
||||
f = fake(runs=noise + [_run(1, "ci.yml", "test", "success")])
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert [p["state"] for p in f.posted] == ["success"]
|
||||
|
||||
|
||||
def _gh_pr(n, repo="sneakyfree/r"):
|
||||
return {
|
||||
"number": n,
|
||||
"title": "t",
|
||||
"html_url": "u",
|
||||
"head": {"ref": f"b{n}", "sha": SHA, "repo": {"full_name": repo} if repo else None},
|
||||
"base": {"ref": "main"},
|
||||
}
|
||||
|
||||
|
||||
def test_fork_prs_are_never_mirrored(fake, monkeypatch):
|
||||
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
|
||||
f = fake(gh_prs=[_gh_pr(1, repo="stranger/r"), _gh_pr(2, repo=None)])
|
||||
assert bridge.sync_prs("r") == []
|
||||
assert f.opened == []
|
||||
|
||||
|
||||
def test_pr_mirror_opened_once_and_closed_when_github_closes(fake, monkeypatch):
|
||||
monkeypatch.setattr(bridge, "GH_OWNER", "sneakyfree")
|
||||
f = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 3, "title": "[GH#5] gone"}])
|
||||
assert bridge.sync_prs("r") == [SHA]
|
||||
assert [o["head"] for o in f.opened] == ["b7"]
|
||||
assert f.closed == ["/repos/windyadmin/r/pulls/3"]
|
||||
|
||||
f2 = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 4, "title": "[GH#7] t"}])
|
||||
bridge.sync_prs("r")
|
||||
assert f2.opened == [] and f2.closed == []
|
||||
|
||||
|
||||
def test_image_build_jobs_are_not_posted(fake):
|
||||
"""No Docker daemon in job containers (I-5): a build job's red is structural."""
|
||||
f = fake(
|
||||
runs=[_run(1, "ci.yml", "Docker Build", "failure"), _run(2, "ci.yml", "docker", "failure")]
|
||||
)
|
||||
bridge.post_statuses("r", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_non_blocking_jobs_are_not_posted_for_that_repo_only(fake, monkeypatch):
|
||||
"""Grant ruled windy-pro's desktop/installer jobs non-blocking: they must not
|
||||
reach GitHub for windy-pro, and the rule must not leak to other repos."""
|
||||
monkeypatch.setattr(bridge, "NON_BLOCKING", {"windy-pro": {"ci/build-desktop"}})
|
||||
runs = [_run(1, "ci.yml", "build-desktop", "failure"), _run(2, "ci.yml", "test", "success")]
|
||||
f = fake(runs=runs)
|
||||
bridge.post_statuses("windy-pro", SHA)
|
||||
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
|
||||
f2 = fake(runs=runs)
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert sorted(p["context"] for p in f2.posted) == [
|
||||
"windy-git/ci/build-desktop",
|
||||
"windy-git/ci/test",
|
||||
]
|
||||
|
||||
|
||||
def test_default_non_blocking_is_grants_ruling():
|
||||
assert bridge.NON_BLOCKING.get("windy-pro") == {
|
||||
"ci/build-desktop",
|
||||
"ci/test-installer",
|
||||
"ci/reality-check",
|
||||
}
|
||||
|
||||
|
||||
def test_transport_blips_are_retried_but_http_errors_are_not(monkeypatch):
|
||||
import urllib.error
|
||||
|
||||
calls = {"n": 0}
|
||||
|
||||
class _R:
|
||||
status = 200
|
||||
|
||||
def read(self):
|
||||
return b"{}"
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def flaky(req, timeout):
|
||||
calls["n"] += 1
|
||||
if calls["n"] < 3:
|
||||
raise urllib.error.URLError("_ssl.c:983: The handshake operation timed out")
|
||||
return _R()
|
||||
|
||||
monkeypatch.setattr(bridge.urllib.request, "urlopen", flaky)
|
||||
monkeypatch.setattr(bridge.time, "sleep", lambda s: None)
|
||||
assert bridge._call("http://x", "t", "GET", "/p") == (200, {})
|
||||
assert calls["n"] == 3
|
||||
|
||||
def forbidden(req, timeout):
|
||||
calls["n"] += 1
|
||||
raise urllib.error.HTTPError("http://x/p", 403, "no", {}, None)
|
||||
|
||||
calls["n"] = 0
|
||||
monkeypatch.setattr(bridge.urllib.request, "urlopen", forbidden)
|
||||
assert bridge._call("http://x", "t", "GET", "/p") == (403, None)
|
||||
assert calls["n"] == 1
|
||||
|
||||
|
||||
GOOD = "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: []\n"
|
||||
BROKEN = "on: push\njobs:\n test:\n runs-on: x\n steps: [\n"
|
||||
|
||||
|
||||
def test_invalid_workflow_gets_an_error_status_even_with_no_runs(fake):
|
||||
# Gitea fires NO run for an invalid file: without this the PR shows nothing.
|
||||
f = fake(workflows={".github/workflows/ci.yml": BROKEN})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [(p["context"], p["state"]) for p in f.posted] == [("windy-git/ci/workflow", "error")]
|
||||
assert "invalid YAML at line 5" in f.posted[0]["description"]
|
||||
assert f.posted[0]["target_url"].endswith(f"/src/commit/{SHA}/.github/workflows/ci.yml")
|
||||
|
||||
|
||||
def test_valid_workflows_post_nothing_extra(fake):
|
||||
f = fake(runs=[_run(1, "ci.yml", "test", "success")], workflows={".github/workflows/ci.yml": GOOD})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert [p["context"] for p in f.posted] == ["windy-git/ci/test"]
|
||||
|
||||
|
||||
def test_workflow_error_is_not_reposted(fake):
|
||||
f = fake(
|
||||
workflows={".github/workflows/ci.yml": BROKEN},
|
||||
statuses=[{"context": "windy-git/ci/workflow", "state": "error"}],
|
||||
)
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
def test_gitea_dir_wins_over_github_dir(fake):
|
||||
# Gitea runs .gitea/workflows when it has files and ignores .github/workflows.
|
||||
f = fake(workflows={".gitea/workflows/ci.yml": GOOD, ".github/workflows/old.yml": BROKEN})
|
||||
bridge.post_statuses("windy-chat", SHA)
|
||||
assert f.posted == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text, problem",
|
||||
[
|
||||
(GOOD, None),
|
||||
("on: push\njobs:\n a:\n uses: ./x.yml\n", None),
|
||||
(BROKEN, "invalid YAML at line 5"),
|
||||
("jobs:\n a:\n runs-on: x\n", "no `on:` trigger"),
|
||||
("on: push\n", "no `jobs:`"),
|
||||
("on: push\njobs:\n a:\n steps: []\n", "job `a` has no `runs-on:`"),
|
||||
("- a\n", "not a YAML mapping"),
|
||||
],
|
||||
)
|
||||
def test_workflow_problem(text, problem):
|
||||
assert bridge.workflow_problem(text) == problem
|
||||
202
api/tests/test_telemetry.py
Normal file
202
api/tests/test_telemetry.py
Normal file
@@ -0,0 +1,202 @@
|
||||
"""Field telemetry from the API (step 2): behavioural, no network.
|
||||
|
||||
A refusal must become exactly one forge.auth.failed row in the DECLARED shape
|
||||
(the ledger quarantines anything else); non-refusal errors must not; the
|
||||
heartbeat must count what happened and never invent a p95 for no traffic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import Depends, FastAPI
|
||||
|
||||
from api.app import telemetry as tmod
|
||||
from api.app.errors import RepairPointer
|
||||
|
||||
DECLARED_AUTH_KEYS = {"code", "http_status", "caller", "route", "upstream_status", "synthetic"}
|
||||
|
||||
|
||||
def _app(tel: tmod.Telemetry) -> FastAPI:
|
||||
"""The real middleware + handler, re-registered on a bare app (no DB)."""
|
||||
from api.app import main
|
||||
|
||||
app = FastAPI()
|
||||
app.state.telemetry = tel
|
||||
app.middleware("http")(main._count_requests)
|
||||
app.exception_handler(RepairPointer)(main._repair_pointer_handler)
|
||||
|
||||
def refuse(code: str, status: int):
|
||||
def dep():
|
||||
raise RepairPointer(
|
||||
status_code=status,
|
||||
code=code,
|
||||
speak="no",
|
||||
machine_cause="test",
|
||||
remediation_tool=None,
|
||||
)
|
||||
|
||||
return dep
|
||||
|
||||
@app.get("/api/v1/repos/{repo}/grants", dependencies=[Depends(refuse("passport_revoked", 403))])
|
||||
async def grants(repo: str):
|
||||
return {}
|
||||
|
||||
@app.get("/api/v1/nope", dependencies=[Depends(refuse("repo_not_found", 404))])
|
||||
async def nope():
|
||||
return {}
|
||||
|
||||
@app.get("/ok")
|
||||
async def ok():
|
||||
return {"ok": True}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
async def _get(app, path, headers=None):
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://t") as c:
|
||||
return await c.get(path, headers=headers or {})
|
||||
|
||||
|
||||
def _tel():
|
||||
return tmod.Telemetry(
|
||||
"http://ledger.invalid/v1/events",
|
||||
"tok",
|
||||
environment="test",
|
||||
commit_sha="abc1234",
|
||||
version="0.1.0",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refusal_emits_one_declared_row_with_route_template_not_path():
|
||||
tel = _tel()
|
||||
r = await _get(
|
||||
_app(tel),
|
||||
"/api/v1/repos/grandmas-secret-project/grants",
|
||||
{"Authorization": "Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkVQVCJ9.e30.x"},
|
||||
)
|
||||
assert r.status_code == 403
|
||||
rows = [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row["actor_type"] == "system" and "actor_id" not in row
|
||||
assert set(row["metadata"]) <= DECLARED_AUTH_KEYS
|
||||
assert row["metadata"]["code"] == "passport_revoked"
|
||||
assert row["metadata"]["http_status"] == 403
|
||||
assert row["metadata"]["caller"] == "anonymous_agent"
|
||||
assert row["metadata"]["route"] == "/api/v1/repos/{repo}/grants"
|
||||
assert "grandmas-secret-project" not in str(row)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_auth_errors_are_counted_but_not_refusal_rows():
|
||||
tel = _tel()
|
||||
await _get(_app(tel), "/api/v1/nope")
|
||||
assert not [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"]
|
||||
assert tel.errors_4xx == 1 and tel.refusals_4xx == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_heartbeat_counts_requests_refusals_and_p95():
|
||||
tel = _tel()
|
||||
app = _app(tel)
|
||||
for _ in range(3):
|
||||
await _get(app, "/ok")
|
||||
await _get(app, "/api/v1/repos/x/grants")
|
||||
meta = tel.health_row()
|
||||
assert meta["requests"] == 4 and meta["refusals_4xx"] == 1 and meta["errors_5xx"] == 0
|
||||
assert isinstance(meta["p95_ms"], int)
|
||||
assert {"interval_s", "uptime_s"} <= set(meta)
|
||||
|
||||
|
||||
def test_no_traffic_means_no_p95_not_a_fake_zero():
|
||||
assert "p95_ms" not in _tel().health_row()
|
||||
|
||||
|
||||
def test_no_token_sends_and_buffers_nothing():
|
||||
tel = tmod.Telemetry("http://ledger.invalid", "")
|
||||
tel.boot()
|
||||
tel.auth_failed(code="token_invalid", http_status=401, caller="unknown")
|
||||
assert tel.buffer == []
|
||||
|
||||
|
||||
def test_unknown_code_is_never_sent_as_a_refusal():
|
||||
tel = _tel()
|
||||
tel.auth_failed(code="made_up_code", http_status=401, caller="unknown")
|
||||
assert tel.buffer == []
|
||||
|
||||
|
||||
def test_boot_omits_an_unknown_commit_rather_than_inventing_one():
|
||||
tel = tmod.Telemetry("http://x", "tok", commit_sha=None, version="0.1.0")
|
||||
tel.boot()
|
||||
assert "commit_sha" not in tel.buffer[0]["metadata"]
|
||||
|
||||
|
||||
def test_caller_classes_are_the_declared_three():
|
||||
assert tmod.caller_class({}) == "unknown"
|
||||
assert tmod.caller_class({"x-service-token": "s"}) == "unknown"
|
||||
assert (
|
||||
tmod.caller_class({"authorization": "Bearer eyJhbGciOiJSUzI1NiJ9.e30.x"})
|
||||
== "anonymous_human"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_synthetic_header_marks_the_row_and_absent_means_real():
|
||||
async def refusal(headers):
|
||||
tel = _tel()
|
||||
await _get(_app(tel), "/api/v1/repos/x/grants", headers)
|
||||
return [e for e in tel.buffer if e["event_type"] == "forge.auth.failed"][0]["metadata"]["synthetic"]
|
||||
|
||||
assert await refusal({"X-Windy-Synthetic": "1"}) is True
|
||||
assert await refusal({}) is False
|
||||
|
||||
|
||||
def test_synthetic_is_forwarded_downstream_only_for_synthetic_requests():
|
||||
token = tmod.SYNTHETIC.set(True)
|
||||
try:
|
||||
assert tmod.synthetic_headers() == {"X-Windy-Synthetic": "1"}
|
||||
finally:
|
||||
tmod.SYNTHETIC.reset(token)
|
||||
assert tmod.synthetic_headers() == {}
|
||||
|
||||
|
||||
# ---- UPDATE 7: the ledger answers 202 even when it quarantines rows ----------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quarantined_rows_are_warned_and_counted_on_the_next_heartbeat(monkeypatch, caplog):
|
||||
tel = _tel()
|
||||
tel.boot()
|
||||
monkeypatch.setattr(
|
||||
tel, "_post", lambda b: (202, {"accepted": 0, "quarantined": 1, "rejections": ["undeclared key"]})
|
||||
)
|
||||
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
|
||||
await tel.flush()
|
||||
assert tel.buffer == [] # sent; the ledger dead-lettered it, retrying won't help
|
||||
assert "QUARANTINED" in caplog.text and "undeclared key" in caplog.text
|
||||
tel.health()
|
||||
assert tel.buffer[-1]["metadata"]["telemetry_quarantined"] == 1
|
||||
assert tel.health_row()["telemetry_quarantined"] == 0 # reset per heartbeat window
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_clean_send_reports_zero_and_logs_nothing(monkeypatch, caplog):
|
||||
tel = _tel()
|
||||
tel.boot()
|
||||
monkeypatch.setattr(tel, "_post", lambda b: (202, {"accepted": 1, "quarantined": 0, "rejections": []}))
|
||||
with caplog.at_level("WARNING", logger="windy-git.telemetry"):
|
||||
await tel.flush()
|
||||
assert caplog.text == ""
|
||||
row = tel.health_row()
|
||||
assert row["telemetry_quarantined"] == 0 and row["telemetry_dropped"] == 0
|
||||
|
||||
|
||||
def test_buffer_overflow_is_counted_as_dropped(monkeypatch):
|
||||
monkeypatch.setattr(tmod, "MAX_BUFFER", 3)
|
||||
tel = _tel()
|
||||
for _ in range(5):
|
||||
tel.boot()
|
||||
assert len(tel.buffer) == 3
|
||||
assert tel.health_row()["telemetry_dropped"] == 2
|
||||
106
api/tests/test_webhooks_behavior.py
Normal file
106
api/tests/test_webhooks_behavior.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""G3.5 — the Eternitas webhook receiver, driven over HTTP (audit 2026-08-13).
|
||||
|
||||
The G3.5 invariants in test_invariants.py grep webhooks.py for strings; a
|
||||
refactor that kept the strings and broke the behaviour would pass them all.
|
||||
These send real requests through the real route (no DB: every case here stops
|
||||
before the revocation handler) and assert what the receiver DOES.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from api.app.config import Settings
|
||||
from api.app.errors import RepairPointer
|
||||
from api.app.routes import webhooks
|
||||
|
||||
SECRET = "s" * 64
|
||||
URL = "/api/v1/webhooks/eternitas"
|
||||
|
||||
|
||||
def _app(secret: str = SECRET) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.include_router(webhooks.router)
|
||||
app.state.settings = Settings(eternitas_webhook_secret=secret)
|
||||
|
||||
@app.exception_handler(RepairPointer)
|
||||
async def _h(_, exc: RepairPointer) -> JSONResponse:
|
||||
return JSONResponse(status_code=exc.status_code, content=exc.detail)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
async def _post(body: bytes, headers: dict, secret: str = SECRET) -> httpx.Response:
|
||||
transport = httpx.ASGITransport(app=_app(secret))
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
|
||||
return await c.post(
|
||||
URL, content=body, headers={"content-type": "application/json", **headers}
|
||||
)
|
||||
|
||||
|
||||
def _sig(raw: bytes, secret: str = SECRET) -> str:
|
||||
return hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
# Deliberately odd spacing/key order: a receiver that re-serialises before
|
||||
# hashing produces a different digest and must fail.
|
||||
RAW = b'{"event":"windygit.selftest", "data": {"b": 2, "a": 1}}'
|
||||
EVENT = {"x-eternitas-event": "windygit.selftest"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prefixed_and_bare_digests_are_both_accepted():
|
||||
for header in (f"sha256={_sig(RAW)}", _sig(RAW)):
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["acted"] is False # unknown event: received, nothing done
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_digest_of_reserialised_json_is_refused():
|
||||
reserialised = json.dumps(json.loads(RAW)).encode()
|
||||
assert reserialised != RAW
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(reserialised)}"})
|
||||
assert r.status_code == 401 and r.json()["code"] == "webhook_signature_invalid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_forged_or_wrong_key_signature_is_refused():
|
||||
for header in ("sha256=" + "0" * 64, f"sha256={_sig(RAW, 'other-secret')}", "garbage"):
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": header})
|
||||
assert r.status_code == 401, header
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_signed_event_without_signature_is_refused():
|
||||
r = await _post(RAW, EVENT)
|
||||
assert r.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unset_secret_refuses_rather_than_accepts():
|
||||
r = await _post(RAW, {**EVENT, "x-eternitas-signature": f"sha256={_sig(RAW)}"}, secret="")
|
||||
assert r.status_code == 503 and r.json()["code"] == "webhook_secret_unset"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_revocation_with_bad_signature_never_reaches_the_handler():
|
||||
body = b'{"event":"passport.revoked","passport":"ET26-TEST-GOOD"}'
|
||||
r = await _post(
|
||||
body,
|
||||
{"x-eternitas-event": "passport.revoked", "x-eternitas-signature": "sha256=" + "f" * 64},
|
||||
)
|
||||
assert r.status_code == 401 # refused before any DB work
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reachability_ping_acknowledges_but_never_acts():
|
||||
r = await _post(b'{"anything": "at all"}', {"x-eternitas-event": "platform.test_ping"})
|
||||
assert r.status_code == 200 and r.json()["acted"] is False
|
||||
@@ -11,8 +11,9 @@ log:
|
||||
|
||||
runner:
|
||||
file: /data/.runner
|
||||
capacity: 4 # concurrent jobs; Veron has 24 cores, dind is capped at 12
|
||||
timeout: 30m
|
||||
capacity: 1 # per runner; parallelism = number of runner services (6). See docker-compose.yml
|
||||
timeout: 90m # hard ceiling per job. eternitas's serial pytest is ~50 min; keep
|
||||
# timeout-minutes in each workflow — a hang still reads as a hang
|
||||
shutdown_timeout: 3m
|
||||
insecure: false
|
||||
fetch_timeout: 5s
|
||||
|
||||
@@ -49,8 +49,29 @@ services:
|
||||
mem_limit: 64g
|
||||
restart: unless-stopped
|
||||
|
||||
runner:
|
||||
image: docker.io/gitea/act_runner:0.2.11
|
||||
# ── FOUR runners × capacity 1, not one runner × capacity 4 (2026-09-23) ──
|
||||
#
|
||||
# act caches every action repo at /root/.cache/act/<hash> INSIDE the runner
|
||||
# process and re-fetches it at the start of each job. With capacity 4, four
|
||||
# concurrent jobs share that one directory: one job's refresh rewrites it while
|
||||
# another is tarring it into its job container, and the job dies with
|
||||
# `lstat /root/.cache/act/<hash>/…: no such file or directory` on
|
||||
# `actions/setup-node` / `setup-uv` — a failure that reads like a broken
|
||||
# workflow. windy-chat (~20 jobs per push) hit it on 3 jobs in its first run.
|
||||
# `rm -rf /root/.cache/act` only reset the clock. Separate processes get
|
||||
# separate caches, so the race cannot occur. Same total parallelism, same
|
||||
# single capped dind — the blast radius is unchanged.
|
||||
runner: &runner
|
||||
# 0.2.11 -> 0.6.1 on 2026-08-14. The bundled act in 0.2.11 only knows
|
||||
# `runs.using: node12|node16|node20`, so ANY repo pinning a current action
|
||||
# major dies before its first step with "The runs.using key in action.yml
|
||||
# must be one of: [...], got node24" — Windy-Clone on actions/checkout@v5
|
||||
# is how this surfaced. Verified: `node24` is absent from the 0.2.11 binary
|
||||
# and present in 0.6.1. Every key in this directory's config.yaml still
|
||||
# exists in 0.6.1's schema (0.6.1 only ADDS keys), so the config carries
|
||||
# over unchanged. Rollback is re-pinning 0.2.11; the registration in the
|
||||
# runner-data volume survives either way.
|
||||
image: docker.io/gitea/act_runner:0.6.1
|
||||
depends_on: [dind]
|
||||
environment:
|
||||
# The runner reaches its OWN daemon. Never the host's.
|
||||
@@ -90,6 +111,47 @@ services:
|
||||
mem_limit: 4g
|
||||
restart: unless-stopped
|
||||
|
||||
# Each extra runner registers itself on first start (own name, own volume —
|
||||
# the registration lives in /data/.runner, so volumes must never be shared).
|
||||
runner-2:
|
||||
<<: *runner
|
||||
environment: &env2
|
||||
DOCKER_HOST: tcp://dind:2375
|
||||
GITEA_INSTANCE_URL: https://app.windygit.com
|
||||
GITEA_RUNNER_REGISTRATION_TOKEN: ${RUNNER_TOKEN:?set RUNNER_TOKEN}
|
||||
GITEA_RUNNER_NAME: veron-1-2
|
||||
CONFIG_FILE: /config.yaml
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-2:/data]
|
||||
runner-3:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-3
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-3:/data]
|
||||
runner-4:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-4
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-4:/data]
|
||||
# 5 and 6 added the same day: with ~11 private repos onboarded (windy-chat
|
||||
# alone queues ~24 jobs per push) four runners left 50+ jobs waiting. The
|
||||
# CPU ceiling is dind's (12 of 24 cores, G1.5), not the runner count, so more
|
||||
# runners add concurrency for I/O-bound jobs (npm ci, uv sync) without
|
||||
# taking more of Grant's workstation.
|
||||
runner-5:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-5
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-5:/data]
|
||||
runner-6:
|
||||
<<: *runner
|
||||
environment:
|
||||
<<: *env2
|
||||
GITEA_RUNNER_NAME: veron-1-6
|
||||
volumes: [./config.yaml:/config.yaml:ro, runner-data-6:/data]
|
||||
|
||||
networks:
|
||||
jobs:
|
||||
# Untrusted job containers live here. No route to the forge.
|
||||
@@ -98,4 +160,9 @@ networks:
|
||||
volumes:
|
||||
dind-storage:
|
||||
runner-data:
|
||||
runner-data-2:
|
||||
runner-data-3:
|
||||
runner-data-4:
|
||||
runner-data-5:
|
||||
runner-data-6:
|
||||
|
||||
|
||||
51
deploy/runner/egress.sh
Executable file
51
deploy/runner/egress.sh
Executable file
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI egress filter (2026-09-23) — jobs reach the internet, never Grant's network.
|
||||
#
|
||||
# Measured before this existed: an ordinary (unprivileged) job container inside
|
||||
# the CI dind could open SSH, Ollama, and every dev server on Veron
|
||||
# (192.168.1.73:22/3000/3300/8080/11434) and anything else on the LAN, WireGuard
|
||||
# or Tailscale. No container escape needed — a malicious npm/pip dependency in
|
||||
# any first-party repo's CI could walk straight onto the fleet.
|
||||
#
|
||||
# All CI traffic leaves through the `windy-git-runner_jobs` bridge (dind NATs
|
||||
# its job containers onto it). This script, run at boot and after any runner
|
||||
# compose change, allows on that bridge:
|
||||
# * traffic between the runners and dind (same bridge)
|
||||
# * replies (ESTABLISHED/RELATED)
|
||||
# * DNS (53) — Docker's embedded resolver forwards to the LAN router
|
||||
# * everything public
|
||||
# and drops: RFC1918, CGNAT/Tailscale (100.64/10), link-local, and ANY packet
|
||||
# addressed to the host itself (INPUT), whatever interface IP it targets.
|
||||
# Idempotent: owned chains are flushed and rebuilt; hooks are added once.
|
||||
set -euo pipefail
|
||||
|
||||
NET=windy-git-runner_jobs
|
||||
id=$(docker network inspect "$NET" --format '{{.Id}}')
|
||||
BR="br-${id:0:12}"
|
||||
ip link show "$BR" >/dev/null
|
||||
|
||||
iptables -N WG-CI-EGRESS 2>/dev/null || iptables -F WG-CI-EGRESS
|
||||
iptables -A WG-CI-EGRESS -o "$BR" -j RETURN
|
||||
iptables -A WG-CI-EGRESS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A WG-CI-EGRESS -p udp --dport 53 -j RETURN
|
||||
iptables -A WG-CI-EGRESS -p tcp --dport 53 -j RETURN
|
||||
for cidr in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16; do
|
||||
iptables -A WG-CI-EGRESS -d "$cidr" -j DROP
|
||||
done
|
||||
iptables -A WG-CI-EGRESS -j RETURN
|
||||
|
||||
iptables -N WG-CI-INPUT 2>/dev/null || iptables -F WG-CI-INPUT
|
||||
iptables -A WG-CI-INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A WG-CI-INPUT -j DROP
|
||||
|
||||
# Hooks: remove any stale ones (the bridge name changes if the network is
|
||||
# recreated), then add exactly one of each at the top.
|
||||
for chain in DOCKER-USER INPUT; do
|
||||
target=$([ "$chain" = INPUT ] && echo WG-CI-INPUT || echo WG-CI-EGRESS)
|
||||
while read -r rule; do
|
||||
iptables -D $chain ${rule#-A $chain }
|
||||
done < <(iptables -S "$chain" | grep -- "-j $target" || true)
|
||||
iptables -I "$chain" 1 -i "$BR" -j "$target"
|
||||
done
|
||||
|
||||
echo "ci egress filter active on $BR ($NET)"
|
||||
28
deploy/runner/prune.sh
Executable file
28
deploy/runner/prune.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Keep CI storage bounded (2026-09-23).
|
||||
#
|
||||
# Kit 0's 09-01 wipe began with CI `_work` dirs (74 GB) + Docker filling the
|
||||
# disk. Windy Git's runners have no host `_work` dir — every job runs in a
|
||||
# container inside the CI-only dind — so the thing that grows here is dind's
|
||||
# image/volume store (38 GB when this was written, never pruned). This prunes
|
||||
# ONLY that daemon, over its own socket. It never touches the host's Docker.
|
||||
#
|
||||
# In-use images/volumes are never removed, so a running job is safe.
|
||||
set -euo pipefail
|
||||
CAP_GB="${CI_STORAGE_CAP_GB:-60}"
|
||||
D=(docker exec windy-git-runner-dind-1 docker -H tcp://127.0.0.1:2375) # dind listens on TCP only
|
||||
|
||||
"${D[@]}" container prune -f --filter until=6h >/dev/null
|
||||
"${D[@]}" volume prune -af >/dev/null # job workspaces of finished jobs
|
||||
"${D[@]}" image prune -af --filter until=168h >/dev/null
|
||||
"${D[@]}" builder prune -af --filter until=168h >/dev/null 2>&1 || true
|
||||
|
||||
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
|
||||
if (( used_gb > CAP_GB )); then
|
||||
# Over the cap even after the age-based pass: drop every unused image. The
|
||||
# next jobs re-pull (the act image is ~2 GB) — slower, never wrong.
|
||||
"${D[@]}" image prune -af >/dev/null
|
||||
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
|
||||
fi
|
||||
echo "ci storage ${used_gb}G (cap ${CAP_GB}G)"
|
||||
(( used_gb <= CAP_GB )) || { echo "STILL OVER CAP"; exit 1; }
|
||||
13
deploy/runner/windygit-ci-egress.service
Normal file
13
deploy/runner/windygit-ci-egress.service
Normal file
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Windy Git - CI egress filter (jobs reach the internet, never the LAN/host)
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
# The jobs network exists once the runner compose project is up; retry until it does.
|
||||
ExecStart=/bin/bash -c 'for i in $(seq 1 60); do /srv/windygit/src/deploy/runner/egress.sh && exit 0; sleep 5; done; exit 1'
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
6
deploy/runner/windygit-ci-prune.service
Normal file
6
deploy/runner/windygit-ci-prune.service
Normal file
@@ -0,0 +1,6 @@
|
||||
[Unit]
|
||||
Description=Windy Git - prune CI-only dind storage (bounded, never the host daemon)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/srv/windygit/src/deploy/runner/prune.sh
|
||||
9
deploy/runner/windygit-ci-prune.timer
Normal file
9
deploy/runner/windygit-ci-prune.timer
Normal file
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Windy Git - prune CI storage every 6 hours
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 00/6:37:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
13
deploy/systemd/windygit-backup.service
Normal file
13
deploy/systemd/windygit-backup.service
Normal file
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Windy Git nightly backup (git bundles + windgit schema -> R2)
|
||||
After=network-online.target docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/windygit/src
|
||||
# The .env holds the R2 credentials. The script refuses to run without them
|
||||
# rather than reporting a backup that did not happen.
|
||||
EnvironmentFile=/srv/windygit/src/.env
|
||||
ExecStart=/bin/bash /srv/windygit/src/scripts/backup.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
3
deploy/systemd/windygit-backup.service.d/windy-job.conf
Normal file
3
deploy/systemd/windygit-backup.service.d/windy-job.conf
Normal file
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-backup 26h --expect "ok — [0-9]+ repos" --owner 13 -- /bin/bash /srv/windygit/src/scripts/backup.sh
|
||||
12
deploy/systemd/windygit-backup.timer
Normal file
12
deploy/systemd/windygit-backup.timer
Normal file
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Nightly Windy Git backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 04:17:00
|
||||
# Grant's workstation is not always on at 04:17. Without this a missed window
|
||||
# is simply skipped and the backup silently never runs.
|
||||
Persistent=true
|
||||
RandomizedDelaySec=600
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-ci-prune 7h --expect "ci storage [0-9]+G" --owner 13 -- /srv/windygit/src/deploy/runner/prune.sh
|
||||
12
deploy/systemd/windygit-sync.service
Normal file
12
deploy/systemd/windygit-sync.service
Normal file
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Sync GitHub -> Windy Git (Phase 1: GitHub is the source of truth)
|
||||
After=network-online.target docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/windygit/src
|
||||
EnvironmentFile=/srv/windygit/src/.env
|
||||
# GITHUB_TOKEN is set on the host only (root-only unit file / .env) — NEVER commit it.
|
||||
Environment=GITHUB_OWNER=sneakyfree
|
||||
ExecStart=/bin/bash /srv/windygit/src/scripts/sync_from_github.sh
|
||||
Nice=10
|
||||
3
deploy/systemd/windygit-sync.service.d/windy-job.conf
Normal file
3
deploy/systemd/windygit-sync.service.d/windy-job.conf
Normal file
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/local/bin/windy-job windygit-sync 20m --expect "all repos in step with GitHub" --owner 13 -- /bin/bash /srv/windygit/src/scripts/sync_from_github.sh
|
||||
10
deploy/systemd/windygit-sync.timer
Normal file
10
deploy/systemd/windygit-sync.timer
Normal file
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Keep Windy Git in step with GitHub every 5 minutes
|
||||
|
||||
[Timer]
|
||||
OnBootSec=3min
|
||||
OnUnitActiveSec=5min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
16
deploy/systemd/windygit-tunnel.service
Normal file
16
deploy/systemd/windygit-tunnel.service
Normal file
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Windy Git - Cloudflare Tunnel (the only ingress; no inbound port is opened)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
# G1.4 - bounded, so a misbehaving ingress can never starve Grant's workstation.
|
||||
MemoryMax=512M
|
||||
CPUQuota=100%
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -16,7 +16,11 @@ services:
|
||||
# I-12: baked at build time. A runtime COMMIT_SHA override is ignored.
|
||||
COMMIT_SHA: ${COMMIT_SHA_BUILD:-}
|
||||
BUILT_AT: ${BUILT_AT:-}
|
||||
env_file: [.env]
|
||||
env_file:
|
||||
- .env
|
||||
# WINDYGIT_TELEMETRY_TOKEN (root-only on Veron). Optional: no file = no telemetry.
|
||||
- path: /etc/windygit/telemetry.env
|
||||
required: false
|
||||
environment:
|
||||
DATABASE_URL: postgresql+asyncpg://windygit:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@db:5432/windygit
|
||||
GITEA_BASE_URL: http://gitea:3000
|
||||
@@ -57,12 +61,32 @@ services:
|
||||
# G2.2 — OIDC only. No local password login, no self-registration.
|
||||
GITEA__service__DISABLE_REGISTRATION: "true"
|
||||
GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "true"
|
||||
# SSO #8 (2026-09-23): the password and passkey forms are OFF. windyadmin
|
||||
# is site admin with a local password; leaving the form up made that
|
||||
# password a second, phishable way into the whole forge. Break-glass is
|
||||
# the CLI on Veron (`docker exec -u git windy-git-gitea-1 gitea admin ...`).
|
||||
GITEA__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
|
||||
GITEA__service__ENABLE_PASSKEY_AUTHENTICATION: "false"
|
||||
# G3.1 — a Windy account IS the account. Signing in with Windy provisions
|
||||
# the Gitea user on first arrival; nobody is asked to invent a second
|
||||
# identity for the same person, and no local password ever exists.
|
||||
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "true"
|
||||
# 🔴 OFF (2026-09-23). With it on, ANY stranger with a Windy Word account
|
||||
# (public signup, not even email-verified) got a forge account on first
|
||||
# sign-in — and the CI runners were instance-wide, so their workflows
|
||||
# would run on Veron beside the R2 god token. Proven with a throwaway
|
||||
# account, then closed. Opening the forge to non-Grant users is a §7
|
||||
# Grant decision; until then new accounts are created deliberately.
|
||||
GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "false"
|
||||
GITEA__oauth2_client__USERNAME: email
|
||||
GITEA__oauth2_client__ACCOUNT_LINKING: auto
|
||||
# 🔴 `login`, NOT `auto` (SSO #8). `auto` linked any hub login whose EMAIL
|
||||
# matched an existing account — and windyadmin (SITE ADMIN) carries Grant's
|
||||
# email, so the forge's admin rights rested on the hub never letting anyone
|
||||
# else hold that address. `login` makes an email match prove possession of
|
||||
# the existing account first. Grant is unaffected: his account is already
|
||||
# linked by the hub's stable `sub`, which is matched before email.
|
||||
# ⚠️ env-to-ini SETS but never UNSETS — this value must also be edited in
|
||||
# /srv/windygit/git/gitea/conf/app.ini if it is ever removed from here.
|
||||
GITEA__oauth2_client__ACCOUNT_LINKING: login
|
||||
# The email is asserted by account-server, which is the authority on it.
|
||||
# Asking the user to re-verify an address their identity provider already
|
||||
# verified is friction that buys nothing.
|
||||
|
||||
@@ -93,3 +93,24 @@ mattered most, *is an agent really that agent*, shipped inverted and untested.
|
||||
The remedy is not more process; it is **behavioral tests and canary probes for
|
||||
the security-critical paths**, so verification persists instead of living in a
|
||||
transcript.
|
||||
|
||||
## Disposition update — 2026-09-23 (lane 13)
|
||||
|
||||
**Privileged dind beside the tokens — materially reduced, not closed.**
|
||||
- The account-wide R2 token is **gone from Veron**. `.env` now carries a token scoped
|
||||
to Workers R2 Bucket Item Read/Write on `windy-git-lfs` + `windy-git-backups` only,
|
||||
minted by API (verified: works on both buckets, refused on any other). A CI escape
|
||||
now reaches Windy Git's own two buckets, not every bucket and zone in the account.
|
||||
- Runners take jobs **only from windyadmin-owned repos** (`action_runner.owner_id`), and
|
||||
forge self-registration is off, so no stranger's workflow can run here.
|
||||
- A host egress filter (`deploy/runner/egress.sh`) stops job containers reaching Veron,
|
||||
the LAN, WireGuard or Tailscale.
|
||||
- Still open: dind runs `--privileged` (next: Sysbox); the host still holds a GitHub
|
||||
token and the Gitea admin token.
|
||||
|
||||
**Revocation / webhook secret** — `ETERNITAS_WEBHOOK_SECRET` recovered from the
|
||||
Eternitas platform row and set; signed deliveries verify.
|
||||
|
||||
**Tests are string asserts** — the security paths now have behavioural suites
|
||||
(`test_hub_jwt.py`, `test_webhooks_behavior.py`, `test_pr_status_bridge.py`);
|
||||
a mutation check showed the old grep invariant passing a broken HMAC prefix strip.
|
||||
|
||||
@@ -87,6 +87,50 @@ Per repo, deliberately, when that repo is quiet:
|
||||
4. later, when it flips to Windy-Git-first: remove it from `REPOS` *first*,
|
||||
repoint its sessions, add a push-mirror back to GitHub
|
||||
|
||||
## Private repos: Windy Git IS their CI (permanent, 2026-09-23)
|
||||
|
||||
The platform repos stay **private** on GitHub (Grant, 2026-09-23), and private
|
||||
repos cannot run GitHub Actions on this account at all. Windy Git is therefore
|
||||
their CI permanently, not a stopgap:
|
||||
|
||||
GitHub push ──sync (15 min)──▶ Windy Git ──runner──▶ Veron 1
|
||||
▲ │
|
||||
└──── commit status windy-git/<workflow>/<job> ◀───┘ scripts/pr_status_bridge.py
|
||||
|
||||
- `pr_status_bridge.py` runs at the end of every sync. It opens a `[GH#N]`
|
||||
mirror PR in Windy Git for every open **same-repo** GitHub PR (so
|
||||
`pull_request` workflows fire), closes it when the GitHub PR closes, and posts
|
||||
each job's result back to GitHub on PR heads and the default-branch head.
|
||||
**Never merge a `[GH#N]` PR here** — merge on GitHub.
|
||||
- Fork PRs are never run: their branch is never synced, and untrusted code
|
||||
beside the privileged dind is the open audit finding.
|
||||
- Covered repos: `BRIDGE_REPOS` in the script. Public repos are left out on
|
||||
purpose; they run real GitHub Actions and two verdicts per commit is noise.
|
||||
- `skipped` jobs post nothing — no green for a job nobody ran.
|
||||
- **Image-build jobs** (name matches `docker`) post nothing: job containers
|
||||
have no Docker daemon by design (I-5), so they are red on every commit. A
|
||||
rootless builder (BuildKit rootless / buildx in the capped dind) is the open
|
||||
decision that would bring them back.
|
||||
|
||||
**Onboarding another private repo** — the promotion steps below, then:
|
||||
|
||||
# on Veron 1, as root
|
||||
set -a; . /srv/windygit/src/.env; set +a
|
||||
python3 scripts/import_from_github.py <repo> # writable; aborts if the repo exists
|
||||
# disable EVERY deploying workflow before anything is pushed:
|
||||
curl -X PUT -H "Authorization: token $GITEA_ADMIN_TOKEN" \
|
||||
http://localhost:3080/api/v1/repos/windyadmin/<repo>/actions/workflows/deploy.yml/disable
|
||||
# add <repo> to REPOS in sync_from_github.sh AND BRIDGE_REPOS in pr_status_bridge.py
|
||||
|
||||
An import fires no push event, so `main` has no verdict until its next commit.
|
||||
To get one now: force Windy Git's `main` back one commit, then
|
||||
`systemctl start windygit-sync` — the sync pushes it forward and CI fires.
|
||||
|
||||
⚠️ **`/actions/tasks` lists only jobs a runner has PICKED UP.** Queued runs are
|
||||
invisible there, so a repo can read "0 runs" while work is waiting. The truth is
|
||||
`action_run` in the `gitea` database (status 1 success, 2 failure, 5 waiting,
|
||||
6 running).
|
||||
|
||||
## ⚠️ Deploy workflows are DISABLED on Windy Git, deliberately
|
||||
|
||||
Six workflows fire on `push:` and deploy to production:
|
||||
|
||||
@@ -15,6 +15,7 @@ Mirrored into `windy-cloud` and `eternitas` on change.
|
||||
| eternitas | `GET /api/v1/trust/{passport}` | band + allowed_actions |
|
||||
| eternitas | `GET /api/v1/registry/{passport}/integrity` | ⚠️ note the path — `windy-registry` calls `/api/v1/passports/{p}/status`, which 404s, which is why the integrity index has never been populated |
|
||||
| account-server | OIDC discovery + JWKS | human identity (G3.1) |
|
||||
| windy-admin ledger | `POST /v1/events` (admin.windyword.ai) | field telemetry: `ci.run`, `ci.job_cancelled`, `service.boot`, `service.health`, `forge.auth.failed`. Shapes are declared with the ledger owner BEFORE shipping (the server quarantines undeclared keys). Codes, counts, route templates only |
|
||||
| windy-cloud-sites | `POST /api/v1/sites/{id}/versions` | publish docs from a repo |
|
||||
|
||||
## Calls IN
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
# RUNBOOK — Windy Git on Veron 1 (rung R0)
|
||||
|
||||
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron`. Passwordless sudo.
|
||||
Host `Veron-1-5090`, WireGuard `10.10.0.6`, alias `wg-veron` (or `ts-veron`). Passwordless sudo.
|
||||
|
||||
**Checkouts (one-repo doctrine):** the ONE standing dev checkout is **OC5
|
||||
`~/windy-git`** (platform repos live on OC5). `/srv/windygit/src` on Veron is the
|
||||
*deploy* copy — it holds no local work. Nothing else should exist.
|
||||
|
||||
⛔ **Kit 0 is never a host for this service** (D-4). `api/app/main.py` refuses to
|
||||
boot in production if it finds itself on `72.60.118.54`.
|
||||
@@ -15,6 +19,9 @@ boot in production if it finds itself on `72.60.118.54`.
|
||||
| `/etc/cloudflared/config.yml` | tunnel ingress |
|
||||
| `/etc/cloudflared/windy-git.json` | tunnel credentials, mode 600 |
|
||||
| `/srv/windygit/src/.env` | secrets, mode 600, **never committed** |
|
||||
| `/srv/windygit/git/gitea/conf/app.ini` | Gitea's persisted config — env-to-ini SETS but never UNSETS; edit here when removing a `GITEA__*` var |
|
||||
| `/srv/windygit/sync/*.git` | bare staging copies the GitHub→Windy Git sync pushes from |
|
||||
| `/srv/windygit/src/deploy/runner/.env` | `RUNNER_TOKEN` — a **windyadmin user-level** registration token (not instance-level; see CI) |
|
||||
|
||||
## Ports — all loopback, on purpose
|
||||
|
||||
@@ -22,7 +29,7 @@ boot in production if it finds itself on `72.60.118.54`.
|
||||
|---|---|
|
||||
| `127.0.0.1:3080` | Gitea (host 3000 is a resident node dev server; 3300 is nginx — **do not fight them for a port**) |
|
||||
| `127.0.0.1:8600` | windy-git API |
|
||||
| `127.0.0.1:2000` | cloudflared metrics |
|
||||
| `127.0.0.1:2001` | cloudflared metrics (`metrics:` in `/etc/cloudflared/config.yml`) — **not 2000**, see Troubleshooting |
|
||||
|
||||
**No inbound port is opened.** cloudflared dials out, so the dynamic residential
|
||||
IP is irrelevant and there is no firewall hole to maintain.
|
||||
@@ -41,12 +48,15 @@ sudo systemctl status windygit-tunnel
|
||||
|
||||
```bash
|
||||
ssh wg-veron
|
||||
cd /srv/windygit/src && git pull
|
||||
cd /srv/windygit/src && git fetch origin && git merge --ff-only origin/main # READ the output
|
||||
export COMMIT_SHA_BUILD=$(git rev-parse HEAD) BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
sudo -E docker compose up -d --build
|
||||
sudo -E docker compose up -d --build --no-deps api # API only: no forge restart
|
||||
curl -s https://api.windygit.com/version # MUST equal git rev-parse HEAD
|
||||
```
|
||||
|
||||
A Gitea config change (compose `GITEA__*`) needs `sudo docker compose up -d --no-deps gitea`
|
||||
— a ~6 s forge outage; running CI jobs survive it. Check `app.ini` afterwards.
|
||||
|
||||
⚠️ **Never `git pull -q` in a deploy script.** `-q` hides *errors*, not just
|
||||
noise. On 2026-08-14 a divergent branch made `pull -q` fail silently and the
|
||||
"deploy" ran for 20 minutes against stale code while reporting success. Use
|
||||
@@ -72,11 +82,54 @@ curl -sI https://app.windygit.com/ | head -1 # Gitea, 200
|
||||
sudo ss -tlnp | grep -E "3080|8600" # both must be 127.0.0.1
|
||||
```
|
||||
|
||||
## Timers (host systemd units — the sync timer is NOT in the repo)
|
||||
|
||||
| Unit | Cadence | Does |
|
||||
|---|---|---|
|
||||
| `windygit-sync.timer` | every 5 min (`OnUnitActiveSec`) | GitHub → Windy Git for `REPOS` in `scripts/sync_from_github.sh`, then `scripts/pr_status_bridge.py` (mirror PRs + GitHub commit statuses). A manual `systemctl start` RESETS the 5-min clock. |
|
||||
| `windygit-backup.timer` | nightly | `git bundle` + pg_dump → R2, 30-day retention |
|
||||
| `windygit-ci-prune.timer` | every 6 h | `deploy/runner/prune.sh` — CI dind storage, 60 GB cap |
|
||||
| `windygit-tunnel.service` | always | the only ingress |
|
||||
|
||||
## CI (Gitea Actions) — see `docs/CUTOVER.md` for onboarding a repo
|
||||
|
||||
- **Six runners × capacity 1** (`deploy/runner/docker-compose.yml`), one shared
|
||||
dind capped at 12 cores / 64 GB. Capacity >1 in one runner shares
|
||||
`/root/.cache/act` between jobs and races (`lstat …: no such file`).
|
||||
- **Runners are scoped to the `windyadmin` user** (`action_runner.owner_id=1`),
|
||||
so only first-party repos run. A repo owned by anyone else — a plane-created
|
||||
agent or `u-system` repo — gets NO runner. Re-registrations inherit this
|
||||
because `RUNNER_TOKEN` is user-level.
|
||||
- Job ceiling 90 min (`config.yaml` `runner.timeout`); a `config.yaml` change
|
||||
needs each runner restarted **while idle** — `compose up -d` won't recreate it.
|
||||
- `/actions/tasks` lists only PICKED-UP jobs. Queue truth is `action_run_job`
|
||||
in the `gitea` DB: `sudo docker exec -i windy-git-db-1 psql -U windygit -d gitea`
|
||||
(status 1 ok · 2 fail · 3 cancelled · 4 skipped · 5 waiting · 6 running).
|
||||
- Job logs are in R2, not on disk. `GET /api/v1/repos/{o}/{r}/actions/jobs/{JOB_ID}/logs`
|
||||
takes the `action_run_job` id, not the task id.
|
||||
|
||||
## Sign-in posture
|
||||
|
||||
- Windy SSO only: password + passkey forms OFF, `ACCOUNT_LINKING=login`,
|
||||
**auto-registration OFF** — opening the forge to non-Grant users is a §7
|
||||
Grant decision.
|
||||
- **Break-glass:** `sudo docker exec -u git windy-git-gitea-1 gitea admin user generate-access-token --username windyadmin --token-name <name> --scopes <scopes> --raw`
|
||||
(delete it after: `delete from access_token where name='<name>'` in the gitea DB —
|
||||
Gitea refuses token management over token auth).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**A hostname returns 530 or won't resolve** — the tunnel is down. `sudo systemctl
|
||||
restart windygit-tunnel`, then `journalctl -u windygit-tunnel -n 50`.
|
||||
|
||||
**`windygit-tunnel` crash-loops with `bind: address already in use` on the metrics
|
||||
port** — cloudflared exits if it cannot bind `metrics:`, taking ingress with it.
|
||||
Until 2026-09-23 this unit restarted ~91,000 times because another project's
|
||||
`cornercall-tunnel` held 127.0.0.1:2000; ingress only survived because a stray
|
||||
generic `cloudflared.service` ran the same config (now disabled). Windy Git's
|
||||
metrics port is **2001**. `sudo ss -ltnp | grep :2001` names any squatter.
|
||||
Keep exactly ONE unit running `/etc/cloudflared/config.yml`: `windygit-tunnel`.
|
||||
|
||||
**TLS handshake fails with `curl` exit 35 and no HTTP status at all** — someone
|
||||
added a **two-level** hostname. Free Universal SSL covers `windygit.com` and
|
||||
`*.windygit.com` only. The request dies before the tunnel is consulted, so it
|
||||
|
||||
@@ -13,56 +13,150 @@ Grant signs in with his existing Windy Word credentials — no second account.
|
||||
Agents authenticate with real Eternitas EPT signature verification and are
|
||||
rate-limited by integrity band.
|
||||
|
||||
## DONE since this was written — the three-repo CI fix
|
||||
## SOLVED — the CI failures were never about Postgres
|
||||
|
||||
All three PRs are **merged and synced**: windy-mind #100, WindyCloud #89,
|
||||
windy-registry #31 (eternitas #149 earlier).
|
||||
The `localhost` → `postgres` fix was correct and is worth keeping, but it was
|
||||
**not** what was failing these jobs. They died at step 2, before Postgres was
|
||||
ever contacted.
|
||||
|
||||
**Result: 1 of 3 verified fixed, 2 still failing for an undetermined reason.**
|
||||
**Root cause: `astral-sh/setup-uv@v4` asks the forge for uv's latest release.**
|
||||
|
||||
- ✅ **windy-registry** — `postgres integration` went **failure → success**. The
|
||||
fix is proven correct.
|
||||
- ❌ **windy-mind**, **WindyCloud** — `migrations` still fails. The DATABASE_URL
|
||||
is definitely right now; the cause is something else and was **not
|
||||
determined** — the jobs API returns "job not found" for the ids the runs
|
||||
report, so logs could not be retrieved that way.
|
||||
setup-uv v4 added "resolve latest version instead of downloading latest release"
|
||||
(astral-sh/setup-uv#178). Resolution goes through `@actions/github`, whose
|
||||
octokit reads **`GITHUB_API_URL`** — which act_runner points at *our forge*. So
|
||||
the action requested:
|
||||
|
||||
**Next session: read those job logs from the Gitea web UI** (`app.windygit.com`
|
||||
→ repo → Actions → the failing run), not the jobs API. Suspicion worth checking
|
||||
first: both use `astral-sh/setup-uv`, and eternitas' equivalent job failed with
|
||||
`error: Failed to spawn: pytest` even after the action resolved — so the uv
|
||||
toolchain may not be landing on PATH in these containers. That would be a
|
||||
different, shared root cause.
|
||||
```
|
||||
GET https://app.windygit.com/api/v1/repos/astral-sh/uv/releases/latest → 404
|
||||
```
|
||||
|
||||
**A trap worth keeping:** the three repos did NOT share one pattern. A naive
|
||||
`localhost` → `postgres` swap would have left **WindyCloud on port 15432** (it
|
||||
maps `15432:5432`) and windy-registry on a `job.services.postgres.ports[…]`
|
||||
expression. Service-name networking always uses the container's **internal**
|
||||
port — 5432 — never the mapped host port.
|
||||
Gitea has no `astral-sh/uv`, so it answered its standard 404 body, *"The target
|
||||
couldn't be found."* setup-uv threw that string, act printed it as `::error::`,
|
||||
and every later step was skipped by `success()`.
|
||||
|
||||
## The original task description (superseded above)
|
||||
**The fix (merged to GitHub, 11 repos):** pin an explicit `version:` on every
|
||||
`setup-uv@v4`/`@v5` step. `resolveVersion()` short-circuits on an explicit
|
||||
version *before* any API call, and the download URL is hardcoded to github.com —
|
||||
so the forge round-trip disappears. Pinned to `0.12.5`, which is what `latest`
|
||||
already resolved to.
|
||||
|
||||
**Three repos need a one-line CI fix.** Their workflows reach a Postgres service
|
||||
at `@localhost:5432`, which works on GitHub-hosted runners (services are
|
||||
port-mapped to the VM) and fails on ours (the job runs *inside* a container, so
|
||||
`localhost` is the job itself). The service is reachable as **`postgres`**.
|
||||
windy-mind #101, WindyCloud #90, eternitas #150, then the sweep: Windy-Clone #77,
|
||||
windy-agent #355, windy-call #34, windy-cell #31, windy-hand #6, windy-mail #105,
|
||||
windy-search #77, windy-text #29. All merged and synced.
|
||||
|
||||
| repo | workflow |
|
||||
|---|---|
|
||||
| `windy-mind` | `migrations.yml` |
|
||||
| `windy-registry` | `ci.yml` |
|
||||
| `WindyCloud` | `ci.yml` |
|
||||
### Two things that made this hard to see, both worth keeping
|
||||
|
||||
`eternitas` was already fixed this way — see **eternitas PR #149** for the exact
|
||||
shape, including the comment explaining why. Fix must go to **GitHub**, not Windy
|
||||
Git: the sync runs GitHub → Windy Git and force-pushes over local edits.
|
||||
- **act attributes the error to the wrong step.** `::error::The target couldn't
|
||||
be found.` is printed immediately after `actions/checkout`'s `::remove-matcher`,
|
||||
so it reads exactly like a checkout failure. It is not. What settled it was the
|
||||
**Gitea access log** — `sudo docker logs windy-git-gitea-1 | grep " 404 "` — which
|
||||
named the real URL at the same millisecond as the job error. When a job fails
|
||||
with an opaque forge-shaped message, go to the forge's access log, not the job log.
|
||||
|
||||
Proven by direct comparison, same runner and same `postgres:16-alpine` image:
|
||||
windy-git's own gate uses `@postgres:5432` and passes its migration round-trip;
|
||||
eternitas' used `@localhost:5432` and failed.
|
||||
- **The natural experiment was sitting right there.** windy-registry and
|
||||
windy-drops use `setup-uv@v3` and always passed; every v4/v5 caller failed. A
|
||||
version skew across otherwise-identical repos is a diagnosis, not a coincidence.
|
||||
|
||||
### The jobs API "job not found" that blocked the last session
|
||||
|
||||
Not a bug. `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs` requires the
|
||||
job id to belong to **the repo in the path** — a valid id under the wrong owner/repo
|
||||
404s. The API works fine; the URLs were mismatched. Logs are readable this way and
|
||||
you do **not** need the web UI.
|
||||
|
||||
Note logs are **not on disk** — `[storage] STORAGE_TYPE = minio` sends action logs
|
||||
to R2, so `actions_log/` on the host is empty. Read them through the API.
|
||||
|
||||
## SOLVED — second root cause: the runner was four majors behind
|
||||
|
||||
Windy-Clone failed for a completely different reason: `The runs.using key in
|
||||
action.yml must be one of: [composite docker node12 node16 node20 go], got
|
||||
**node24**`. act_runner **0.2.11**'s bundled act predates node24, so any repo
|
||||
pinning a current action major (`actions/checkout@v5`, `actions/setup-python@v6`)
|
||||
died before its first step.
|
||||
|
||||
**Bumped to `gitea/act_runner:0.6.1`** (`cd7dd9b`). Verified beforehand that
|
||||
`node24` is absent from the 0.2.11 binary and present in 0.6.1, and that every
|
||||
key in `deploy/runner/config.yaml` still exists in 0.6.1's schema — 0.6.1 only
|
||||
*adds* keys, so the config carried over untouched. The runner re-declared with
|
||||
the same id and labels `[veron-1 linux-x64 self-hosted linux x64]`; the
|
||||
registration in the `runner-data` volume survived. Windy-Clone went 4/4 red →
|
||||
4/4 green. Rollback is re-pinning 0.2.11; registration is backed up at
|
||||
`/srv/windygit/runner-registration.bak`.
|
||||
|
||||
## What is still red, and why each one is real
|
||||
|
||||
The CI plane is healthy. windy-mind is fully green (`742 passed, 1 skipped`).
|
||||
What remains are genuine repo defects that were **invisible before**, because
|
||||
every job died at step 2:
|
||||
|
||||
| repo | job | cause |
|
||||
|---|---|---|
|
||||
| WindyCloud | `lint` | `ruff format --check` — 7 files would be reformatted |
|
||||
| eternitas | `py-sdk` | `uv run pytest` → `Failed to spawn: pytest`; pytest isn't a declared dep of that project |
|
||||
| eternitas | `test` | needs its own look |
|
||||
| windy-agent | `test (3.12/3.13/3.14)` | all three died **together** at 22:50:19 after ~43 min, mid-suite at 64%, with no verdict in the log. Not the 30m `runner.timeout` (that would have fired at 22:36) and not the runner bump (that was 23:00). Something bulk-killed them; unexplained. |
|
||||
| WindyCloud | `docker`, windy-search `Docker build` | **architectural** — see below |
|
||||
|
||||
`WindyCloud`'s `docker` job wants to build an image and gets `failed to connect
|
||||
to the docker API at unix:///var/run/docker.sock`. Job containers deliberately
|
||||
have **no** docker socket (I-5, and `deploy/runner/docker-compose.yml` says in
|
||||
so many words not to mount it). Mounting the host socket would hand every
|
||||
workflow root on Veron 1. This needs a decision — buildx-in-dind, a rootless
|
||||
builder, or "this job does not run on Windy Git" — not a quiet socket mount.
|
||||
|
||||
The WindyCloud `lint` and eternitas `py-sdk` rows are small fixes in their own
|
||||
repos, left alone on purpose: they are product defects, not forge defects.
|
||||
|
||||
## Second, smaller finding — act's action cache rots
|
||||
|
||||
act caches action repos at `/root/.cache/act/<hash>` inside the runner container
|
||||
and refreshes them with a go-git mirror fetch of `refs/*:refs/*`, unforced. That
|
||||
includes `refs/pull/*`, which GitHub **recomputes** whenever a base branch moves.
|
||||
Reproduced directly:
|
||||
|
||||
```
|
||||
! [rejected] refs/pull/1015/merge -> refs/pull/1015/merge (non-fast-forward)
|
||||
```
|
||||
|
||||
which surfaces as `Non-terminating error while running 'git clone': some refs
|
||||
were not updated`, after which the action does not report `Checked out <ref>`.
|
||||
|
||||
It **has** now failed a job on its own. After the runner bump, `windy-mind tests`
|
||||
died with:
|
||||
|
||||
```
|
||||
❌ Failure - Main Install uv
|
||||
lstat /root/.cache/act/d3e6…/.git-blame-ignore-revs: no such file or directory
|
||||
```
|
||||
|
||||
act tars the cached action directory into the job container, and a file vanished
|
||||
mid-walk. The cache dir had been created at 23:01 and mutated again at 23:03,
|
||||
with `.gitignore` showing as deleted — act removes it before `docker cp`. The
|
||||
likely mechanism is **concurrent jobs sharing one cache dir**: `capacity: 4`, and
|
||||
windy-mind fires four jobs at once that all use setup-uv. Wiping the cache and
|
||||
re-running the job alone made it pass (`742 passed, 1 skipped`). *Mechanism not
|
||||
isolated* — the wipe alone may have been sufficient.
|
||||
|
||||
One dead end worth not repeating: the cached worktree sits at `38f3f104` while
|
||||
`git rev-parse v4` says `e4db8464`. That is **not** a wrong checkout — `v4` is an
|
||||
*annotated tag*, and `v4^{commit}` is `38f3f104`. Don't chase it.
|
||||
|
||||
Wipe with `sudo docker exec windy-git-runner-runner-1 rm -rf /root/.cache/act`
|
||||
(safe — container layer, not a volume). It will rot again. A real fix is either
|
||||
lowering `capacity` or upstream act; neither was attempted.
|
||||
|
||||
## Traps that will waste your time
|
||||
|
||||
- **Gitea status codes are not what they look like.** `1 = success, 2 = failure`,
|
||||
3 cancelled, 4 skipped, 5 waiting, 6 running, 7 blocked. Reading 1/2 as
|
||||
waiting/running inverts every conclusion you draw from `action_run_job`.
|
||||
- **Gitea sets `Secure` cookies** (ROOT_URL is https), so a `curl` login against
|
||||
`http://127.0.0.1:3080` silently keeps no session — it 303s to `/` and you
|
||||
still get "Sign In". Log in through `https://app.windygit.com`.
|
||||
- **There is no rerun API in 1.24.6.** `POST /api/v1/.../runs/{n}/rerun` 404s.
|
||||
Use the web route `POST /{owner}/{repo}/actions/runs/{n}/rerun` with the session
|
||||
cookie plus an `X-Csrf-Token` header taken from the `_csrf` cookie.
|
||||
- **`git pull -q` hides errors.** A divergent branch once made a "deploy" run 20
|
||||
minutes against stale code while reporting success. Use `git fetch && git
|
||||
merge --ff-only` and read the output.
|
||||
@@ -79,20 +173,30 @@ eternitas' used `@localhost:5432` and failed.
|
||||
- **Kit 0 is fragile.** 54 containers on 4 vCPU. Two production incidents in two
|
||||
days, both from *non-production* workloads. Check `uptime` before deploying
|
||||
anything there, and build before recreating so the swap is seconds.
|
||||
- **Service containers**: use the service NAME and its INTERNAL port (5432),
|
||||
never the mapped host port. The three repos did NOT share one pattern — a naive
|
||||
`localhost` → `postgres` swap would have left WindyCloud on port 15432 (it maps
|
||||
`15432:5432`) and windy-registry on a `job.services.postgres.ports[…]` expression.
|
||||
|
||||
## Open items, roughly by value
|
||||
|
||||
1. The three-repo `localhost` fix above.
|
||||
2. **Get non-prod work off Kit 0.** 12 dev/demo containers on the box running
|
||||
1. **Decide what the image-building jobs should do on Windy Git** — WindyCloud
|
||||
`docker` and windy-search `Docker build` (above). The only remaining *forge*
|
||||
question; it needs a decision, not code.
|
||||
2. **windy-agent's three `test` jobs were bulk-killed at 22:50:19** after ~43
|
||||
minutes, mid-suite, with no verdict. Unexplained and not the runner bump.
|
||||
3. The product-level test failures in the table above.
|
||||
4. **act's action cache race** — lower `capacity` below 4, or accept re-runs.
|
||||
5. **Get non-prod work off Kit 0.** 12 dev/demo containers on the box running
|
||||
identity, the CA, mail, Matrix and the broker. Cost two incidents already;
|
||||
the postgres-adapter fix would not have prevented either.
|
||||
3. **Login is ~4–6s** — `postgres-adapter.ts:114` forks a `node -e` process per
|
||||
6. **Login is ~4–6s** — `postgres-adapter.ts:114` forks a `node -e` process per
|
||||
query. Measured: node startup alone is 1.7s on Kit 0 vs 0.01s on Veron. The
|
||||
fix is **one function** (persistent worker + `pg.Pool`), not the "468 call
|
||||
sites" the SOTU scoped. See `docs/incidents/2026-08-12-login-latency-analysis.md`.
|
||||
4. **Privileged dind sits beside broad-scoped tokens** on the CI host — Grant's
|
||||
7. **Privileged dind sits beside broad-scoped tokens** on the CI host — Grant's
|
||||
call, needs a decision not a code change.
|
||||
5. Push-velocity throttling is declared but unenforceable from our plane (git
|
||||
8. Push-velocity throttling is declared but unenforceable from our plane (git
|
||||
push never touches the API); needs a Gitea pre-receive hook.
|
||||
|
||||
## Read these first
|
||||
@@ -114,32 +218,45 @@ app.windygit.com). Read these before doing anything:
|
||||
2. ~/windy-git/docs/TURNOVER-2026-08-14.md
|
||||
3. ~/windy-git/DNA_STRAND_MASTER_PLAN.md (D-1..D-9, I-1..I-13)
|
||||
|
||||
State: live and in use. Grant signs in with his existing Windy account (SSO
|
||||
fixed across windy-pro #346/#347). Agents authenticate with real EPT signature
|
||||
verification. 143 repos, 85 tests green, health ok.
|
||||
State: live and in use. Grant signs in with his existing Windy account. Agents
|
||||
authenticate with real EPT signature verification. 143 repos, 85 tests green.
|
||||
|
||||
TASK: finish the CI fix. Four repos had workflows reaching Postgres through a
|
||||
host port; all four are patched and merged (eternitas #149, windy-mind #100,
|
||||
WindyCloud #89, windy-registry #31). windy-registry's `postgres integration`
|
||||
went failure -> success, proving the approach. But windy-mind and WindyCloud
|
||||
`migrations` still FAIL and I could not determine why.
|
||||
TWO CI root causes are SOLVED and verified:
|
||||
(a) setup-uv v4+ resolved uv's "latest" through GITHUB_API_URL, which
|
||||
act_runner points at our own forge, so it 404'd ("The target couldn't be
|
||||
found.") and every job died at step 2. Fixed by pinning an explicit uv
|
||||
version across 11 repos.
|
||||
(b) act_runner 0.2.11 predates `runs.using: node24`, so any repo on
|
||||
actions/checkout@v5 died before its first step. Bumped to 0.6.1.
|
||||
windy-mind is fully green (742 passed). Windy-Clone went 4/4 red to 4/4 green.
|
||||
|
||||
Start by reading those job logs from the GITEA WEB UI (app.windygit.com -> repo
|
||||
-> Actions -> failing run). Do NOT use the jobs API — it returns "job not found"
|
||||
for the ids the runs report, which is what blocked the last session.
|
||||
|
||||
First hypothesis to test: windy-mind, WindyCloud and eternitas all use
|
||||
`astral-sh/setup-uv`, and eternitas' job failed with `error: Failed to spawn:
|
||||
pytest` even after the action resolved correctly. The uv toolchain may not be
|
||||
landing on PATH inside these job containers — one shared root cause rather than
|
||||
three.
|
||||
TASK: one decision, then cleanup.
|
||||
1. DECIDE what the image-building CI jobs should do here — WindyCloud `docker`
|
||||
and windy-search `Docker build`. They need a Docker daemon; job containers
|
||||
deliberately have no socket (I-5 — mounting the host socket hands every
|
||||
workflow root on Veron 1). Options: buildx inside the existing dind, a
|
||||
rootless builder, or exclude the job. Do NOT mount the host socket.
|
||||
2. windy-agent's three `test` jobs were bulk-killed together at 22:50:19 after
|
||||
~43 min, mid-suite, with no verdict in the log. Not the 30m runner.timeout,
|
||||
not the runner bump. Unexplained — worth a look.
|
||||
3. Small product defects: WindyCloud `lint` (ruff format, 7 files), eternitas
|
||||
`py-sdk` (pytest not a declared dep), eternitas `test`.
|
||||
|
||||
Ground rules already paid for the hard way:
|
||||
- Gitea job status: 1=SUCCESS, 2=FAILURE, 5=waiting, 6=running. Not what you'd guess.
|
||||
- When a job fails with an opaque forge-shaped error, read the FORGE access log
|
||||
(`docker logs windy-git-gitea-1 | grep " 404 "`) — act misattributes the error
|
||||
to the previous step.
|
||||
- Job logs: `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs`. The job id
|
||||
must belong to the repo in the path or you get a misleading "job not found".
|
||||
Logs are in R2, not on disk.
|
||||
- Log into the forge over https://app.windygit.com — Gitea's cookies are Secure,
|
||||
so a curl login to http://127.0.0.1:3080 silently keeps no session.
|
||||
- verify the WHOLE flow, not the half that curls easily
|
||||
- never `git pull -q` in a deploy path; it hides errors
|
||||
- fixes go to GitHub, not Windy Git (sync is GitHub -> Windy Git, force-push)
|
||||
- service containers: use the service NAME and its INTERNAL port (5432),
|
||||
never the mapped host port
|
||||
- if a job fails with `lstat .../<file>: no such file or directory` on an
|
||||
action, act's cache rotted: `docker exec windy-git-runner-runner-1 rm -rf
|
||||
/root/.cache/act`, then re-run. Safe; it is a container layer, not a volume.
|
||||
- check Kit 0's `uptime` before deploying there; two incidents in two days
|
||||
from non-production workloads
|
||||
```
|
||||
|
||||
@@ -31,7 +31,7 @@ dependencies = [
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8.3", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
|
||||
dev = ["pytest>=8.3", "pyyaml>=6.0", "pytest-asyncio>=0.24", "ruff>=0.7", "mypy>=1.13"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
|
||||
@@ -23,6 +23,23 @@ BUCKET="${R2_BUCKET_BACKUPS:-windy-git-backups}"
|
||||
KEEP_DAYS="${BACKUP_KEEP_DAYS:-30}"
|
||||
FAILED=0
|
||||
|
||||
# NEVER bundle these to R2 (orchestrator decision 2026-09-23). They carry
|
||||
# credentials in plaintext — kit-army-config IS the lockbox, and the soul repos
|
||||
# hold agent memory with keys in it — and these bundles are unencrypted, so
|
||||
# anyone holding the R2 key could read every secret in the fleet. They are
|
||||
# backed up ENCRYPTED elsewhere (Windy Drops lane, restic, restore-tested) and
|
||||
# stay mirrored on Veron's own disk in Gitea. Extended globs, matched on name.
|
||||
EXCLUDE="${BACKUP_EXCLUDE:-kit-army-config anima *-soul}"
|
||||
excluded() {
|
||||
local n=$1 pat pats
|
||||
read -ra pats <<< "$EXCLUDE" # read never glob-expands; `for p in $EXCLUDE` would
|
||||
for pat in "${pats[@]}"; do
|
||||
# shellcheck disable=SC2053 # unquoted RHS: glob match is the point
|
||||
[[ "$n" == $pat ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
cleanup() { rm -rf "$WORK"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
@@ -44,6 +61,10 @@ count=0
|
||||
for repo in "$GIT_ROOT"/*/*.git; do
|
||||
owner="$(basename "$(dirname "$repo")")"
|
||||
name="$(basename "$repo" .git)"
|
||||
if excluded "$name"; then
|
||||
log "skip ${owner}/${name} (credential-bearing: never bundled to R2 in plaintext)"
|
||||
continue
|
||||
fi
|
||||
out="$WORK/${owner}__${name}.bundle"
|
||||
|
||||
# --all captures every ref, not just the default branch. A bundle of one
|
||||
|
||||
@@ -73,6 +73,9 @@ class Check:
|
||||
def _probe(c: Check) -> Result:
|
||||
data = json.dumps(c.body).encode() if c.body else None
|
||||
headers = {"User-Agent": "windy-git-canary/1.0", **c.headers}
|
||||
# Our own probes are synthetic traffic (ecosystem convention, Telemetry
|
||||
# UPDATE 4): every service they touch labels the resulting rows.
|
||||
headers["X-Windy-Synthetic"] = "1"
|
||||
if data:
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(c.url, data=data, method=c.method, headers=headers)
|
||||
|
||||
10
scripts/cancel_unrunnable.sh
Executable file
10
scripts/cancel_unrunnable.sh
Executable file
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Cancel jobs no runner can ever take (see cancel_unrunnable.sql). Run on Veron as root.
|
||||
set -euo pipefail
|
||||
SPOOL="${JANITOR_SPOOL:-/var/lib/windy-git/janitor-cancelled.jsonl}"
|
||||
mkdir -p "$(dirname "$SPOOL")"
|
||||
out=$(docker exec -i windy-git-db-1 sh -c 'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1' \
|
||||
< "$(dirname "$0")/cancel_unrunnable.sql")
|
||||
printf '%s\n' "$out" | grep '^{' >> "$SPOOL" || true
|
||||
n=$(printf '%s\n' "$out" | grep -c '^{' || true)
|
||||
echo "[janitor] cancelled ${n} unrunnable job(s)"
|
||||
53
scripts/cancel_unrunnable.sql
Normal file
53
scripts/cancel_unrunnable.sql
Normal file
@@ -0,0 +1,53 @@
|
||||
-- Cancel CI jobs that can never run (called by scripts/cancel_unrunnable.sh).
|
||||
--
|
||||
-- A job whose runs-on names a label no Windy Git runner offers (ubuntu-latest,
|
||||
-- macos-latest, windows-latest …) waits forever: Gitea evaluates a job's `if:`
|
||||
-- only when a runner picks it, so even `if: false` / tag-only jobs sit in the
|
||||
-- queue, invisible to /actions/tasks, and keep their run "waiting" for good.
|
||||
-- After 30 minutes they are cancelled here; the run's status is then recomputed
|
||||
-- (failure > still-active > cancelled > success), the same precedence Gitea uses.
|
||||
-- Keep RUNNER_LABELS in step with deploy/runner/config.yaml.
|
||||
BEGIN;
|
||||
WITH dead AS (
|
||||
UPDATE action_run_job j
|
||||
SET status = 3, stopped = extract(epoch from now())::bigint, updated = extract(epoch from now())::bigint
|
||||
WHERE j.status IN (5, 7)
|
||||
AND to_timestamp(j.created) < now() - interval '30 minutes'
|
||||
AND EXISTS (SELECT 1 FROM jsonb_array_elements_text(j.runs_on::jsonb) l
|
||||
WHERE l NOT IN ('veron-1', 'linux-x64', 'self-hosted', 'linux', 'x64'))
|
||||
RETURNING j.id, j.run_id, j.name, j.runs_on, j.created
|
||||
), runs AS (
|
||||
UPDATE action_run r
|
||||
SET status = CASE
|
||||
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status = 2) THEN 2
|
||||
WHEN EXISTS (SELECT 1 FROM action_run_job x WHERE x.run_id = r.id AND x.status IN (5, 6, 7)
|
||||
AND x.id NOT IN (SELECT id FROM dead)) THEN r.status
|
||||
ELSE 3 END,
|
||||
stopped = CASE WHEN r.stopped = 0 THEN extract(epoch from now())::bigint ELSE r.stopped END
|
||||
WHERE r.id IN (SELECT DISTINCT run_id FROM dead)
|
||||
RETURNING r.id
|
||||
)
|
||||
-- One JSON line per cancelled job: the telemetry emitter ships these as
|
||||
-- ci.job_cancelled (declared with Telemetry Boss, 2026-09-23).
|
||||
SELECT json_build_object(
|
||||
'repo', p.lower_name,
|
||||
'workflow', regexp_replace(r.workflow_id, '\.ya?ml$', ''),
|
||||
'job', d.name,
|
||||
'reason', 'unrunnable_label',
|
||||
'runs_on', (SELECT string_agg(l, ',') FROM jsonb_array_elements_text(d.runs_on::jsonb) l),
|
||||
'waited_s', (extract(epoch from now())::bigint - d.created))::text
|
||||
FROM dead d JOIN action_run r ON r.id = d.run_id JOIN repository p ON p.id = r.repo_id
|
||||
WHERE (SELECT count(*) FROM runs) >= 0;
|
||||
|
||||
-- Jobs BLOCKED on `needs:` inside a run that has already finished (a needed job
|
||||
-- failed): Gitea leaves them status 7 forever. They were never going to run;
|
||||
-- mark them skipped (4), which is what GitHub shows for the same situation.
|
||||
UPDATE action_run_job j
|
||||
SET status = 4, updated = extract(epoch from now())::bigint
|
||||
FROM action_run r
|
||||
WHERE r.id = j.run_id
|
||||
AND j.status = 7
|
||||
AND r.status IN (1, 2, 3)
|
||||
AND to_timestamp(j.created) < now() - interval '30 minutes'
|
||||
RETURNING j.run_id;
|
||||
COMMIT;
|
||||
@@ -43,7 +43,12 @@ import urllib.request
|
||||
#
|
||||
# Bulk import belongs on the host anyway: no hairpin through the edge, no
|
||||
# Cloudflare ~100s proxy ceiling (G4A.5) on a large clone. Run this on Veron 1.
|
||||
GITEA = os.environ.get("GITEA_BASE_URL", "http://localhost:3080")
|
||||
#
|
||||
# 🔴 Deliberately NOT `GITEA_BASE_URL`: the deploy `.env` sets that to
|
||||
# `http://gitea:3000` for the API container, and sourcing `.env` on the host
|
||||
# made this script die on DNS *after* a caller had already deleted the mirror it
|
||||
# was meant to replace (2026-09-23).
|
||||
GITEA = os.environ.get("IMPORT_GITEA_URL", "http://localhost:3080")
|
||||
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
|
||||
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
|
||||
GITHUB_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
|
||||
@@ -224,13 +229,12 @@ def main() -> int:
|
||||
if not targets:
|
||||
ap.error("name a repo, or pass --safe-batch / --list-candidates")
|
||||
|
||||
if "windy-pro" in targets:
|
||||
sys.exit(
|
||||
"REFUSING windy-pro. Six checkouts exist, the build counter has forked "
|
||||
"three ways (main 12 / overnight 34 / wave-44 56), and two sessions "
|
||||
"recorded different HEADs hours apart. Resolve which is current and "
|
||||
"write it down BEFORE importing (G11.5)."
|
||||
)
|
||||
# G11.5 RESOLVED 2026-09-23 (lane 8c, ~/windy-orchestra/WINDYPRO_CHECKOUTS.md):
|
||||
# a read-only audit of all 14 windy-pro checkouts on 5 machines found GitHub
|
||||
# main is canonical (Kit 0 prod and Windy 0 sit exactly on it; the others are
|
||||
# stale, not divergent). Phase 1 keeps GitHub the source of truth anyway, so a
|
||||
# writable Windy Git copy is CI only. Its six deploy/release workflows must be
|
||||
# disabled on import — see docs/CUTOVER.md.
|
||||
|
||||
if args.mirror:
|
||||
print("mirror mode: repos will be read-only and will NOT run CI.\n")
|
||||
|
||||
307
scripts/pr_status_bridge.py
Executable file
307
scripts/pr_status_bridge.py
Executable file
@@ -0,0 +1,307 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Give private GitHub repos a CI signal from Windy Git (P1, 2026-09-23).
|
||||
|
||||
GitHub Actions cannot run on private repos on this account — not even on
|
||||
self-hosted runners ([[reference-github-actions-billing-lock]]). The code is
|
||||
already synced into Windy Git every 15 min and CI runs there, so the only thing
|
||||
missing is the *signal on GitHub*, where people and agents actually read PRs.
|
||||
|
||||
Two jobs, run after every sync:
|
||||
|
||||
1. **Mirror open PRs.** The sync carries branches, not PRs, and the workflows
|
||||
trigger on `pull_request` — so a PR branch alone fires nothing. For each open
|
||||
same-repo GitHub PR we keep one open Windy Git PR with the same head/base.
|
||||
Gitea then fires `pull_request` on open and `synchronize` whenever the sync
|
||||
moves the branch. Windy Git PRs whose GitHub PR closed are closed here too.
|
||||
Fork PRs are ignored: their head branch is never synced, and untrusted fork
|
||||
code on this runner is exactly the blast radius the audit warned about.
|
||||
|
||||
2. **Post results back** as GitHub commit statuses (context
|
||||
`windy-git/<workflow>/<job>`) on each PR head and on the default-branch head.
|
||||
Only posts when a context's state changed, so a 15-min loop doesn't pile
|
||||
hundreds of identical statuses onto one commit.
|
||||
|
||||
Runs ON Veron 1 (localhost Gitea; no Cloudflare hairpin). Needs
|
||||
GITEA_ADMIN_TOKEN and a GITHUB_TOKEN with `repo` scope. Nothing here executes
|
||||
repo code, and no secret is handed to any repo.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import yaml
|
||||
|
||||
GITEA = os.environ.get("BRIDGE_GITEA_URL", "http://localhost:3080").rstrip("/")
|
||||
PUBLIC = "https://app.windygit.com"
|
||||
GITEA_TOKEN = os.environ.get("GITEA_ADMIN_TOKEN", "")
|
||||
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
|
||||
GH_OWNER = os.environ.get("GITHUB_OWNER", "sneakyfree")
|
||||
WG_OWNER = os.environ.get("WINDYGIT_OWNER", "windyadmin")
|
||||
# Private repos only. Public repos run real GitHub Actions on veron1's GitHub
|
||||
# runner; bridging those too would put two competing verdicts on every commit.
|
||||
REPOS = os.environ.get(
|
||||
"BRIDGE_REPOS",
|
||||
"windy-chat windy-mail windy-calendar Windy-Clone WindyCloud windy-search windy-connect"
|
||||
" windy-drops windy-code-web windy-code windy-traveler windy-registry eternitas"
|
||||
" windy-translate windytranslate-site windytraveler-site windy-hand"
|
||||
" windy-cloud-sites windy-cloud-domains windy-cloud-vps windytalk windy-pro windy-mind",
|
||||
).split()
|
||||
|
||||
# Gitea run status -> GitHub status state. `skipped` is deliberately absent: a
|
||||
# job skipped by its own `if:` (e.g. substrate-drift's no-secrets path) has no
|
||||
# verdict, and painting it green would be a claim nobody tested.
|
||||
STATE = {
|
||||
"success": "success",
|
||||
"failure": "failure",
|
||||
"cancelled": "error",
|
||||
"running": "pending",
|
||||
"waiting": "pending",
|
||||
"blocked": "pending",
|
||||
}
|
||||
MIRROR_TAG = "[GH#"
|
||||
|
||||
# Image-build jobs cannot pass here BY DESIGN: job containers get no Docker
|
||||
# daemon (I-5 — the host socket would hand every workflow root on Veron 1).
|
||||
# Posting them would put a permanent red X on every commit, and a signal that is
|
||||
# always red trains everyone to ignore red. Not posted until a rootless builder
|
||||
# exists; that is a decision, recorded in docs/CUTOVER.md, not a failure.
|
||||
NO_DAEMON_JOB = re.compile(r"docker", re.IGNORECASE)
|
||||
|
||||
# Jobs Grant ruled NON-BLOCKING (GRANT_DECISIONS_2026-09-23): still run on
|
||||
# Windy Git and visible there, but not posted to GitHub, so they cannot turn a
|
||||
# commit's combined status red. Format: "repo:workflow/job,workflow/job;repo2:..."
|
||||
# windy-pro's desktop/installer jobs belong to Grant's desktop side (fixed from
|
||||
# his Mac mini), not to any lane's merge gate.
|
||||
NON_BLOCKING: dict[str, set[str]] = {}
|
||||
for _entry in os.environ.get(
|
||||
"BRIDGE_NON_BLOCKING", "windy-pro:ci/build-desktop,ci/test-installer,ci/reality-check"
|
||||
).split(";"):
|
||||
if ":" in _entry:
|
||||
_repo, _jobs = _entry.split(":", 1)
|
||||
NON_BLOCKING[_repo.strip()] = {j.strip() for j in _jobs.split(",") if j.strip()}
|
||||
|
||||
|
||||
# Gitea reads the FIRST of these dirs that has workflow files at a commit (1.24).
|
||||
WORKFLOW_DIRS = (".gitea/workflows", ".github/workflows")
|
||||
|
||||
|
||||
def workflow_problem(text: str) -> str | None:
|
||||
"""Why Gitea would drop this workflow file, or None if it looks runnable.
|
||||
|
||||
Gitea skips an invalid workflow with one log line and fires no run at all,
|
||||
so on GitHub the PR just shows nothing, and people wait for CI that is never
|
||||
coming. These are the shapes we have actually hit, not a full schema.
|
||||
"""
|
||||
try:
|
||||
doc = yaml.safe_load(text)
|
||||
except yaml.YAMLError as e:
|
||||
mark = getattr(e, "problem_mark", None)
|
||||
return f"invalid YAML at line {mark.line + 1}" if mark else "invalid YAML"
|
||||
if not isinstance(doc, dict):
|
||||
return "not a YAML mapping"
|
||||
if "on" not in doc and True not in doc: # YAML 1.1 reads a bare `on` as True
|
||||
return "no `on:` trigger"
|
||||
jobs = doc.get("jobs")
|
||||
if not isinstance(jobs, dict) or not jobs:
|
||||
return "no `jobs:`"
|
||||
for name, job in jobs.items():
|
||||
if not isinstance(job, dict):
|
||||
return f"job `{name}` is not a mapping"
|
||||
if "runs-on" not in job and "uses" not in job:
|
||||
return f"job `{name}` has no `runs-on:`"
|
||||
return None
|
||||
|
||||
|
||||
def invalid_workflows(repo: str, sha: str) -> dict[str, tuple[str, str]]:
|
||||
"""{context: (path, problem)} for each workflow file at `sha` that won't run."""
|
||||
for d in WORKFLOW_DIRS:
|
||||
st, entries = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{d}?ref={sha}")
|
||||
if st == 404:
|
||||
continue
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git {d}@{sha[:7]} -> {st}")
|
||||
files = [e for e in entries or [] if e.get("type") == "file"
|
||||
and e["name"].endswith((".yml", ".yaml"))]
|
||||
if not files:
|
||||
continue
|
||||
bad = {}
|
||||
for e in files:
|
||||
st, f = gitea("GET", f"/repos/{WG_OWNER}/{repo}/contents/{e['path']}?ref={sha}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git {e['path']}@{sha[:7]} -> {st}")
|
||||
problem = workflow_problem(base64.b64decode(f["content"]).decode("utf-8", "replace"))
|
||||
if problem:
|
||||
stem = re.sub(r"\.ya?ml$", "", e["name"])
|
||||
bad[f"windy-git/{stem}/workflow"] = (e["path"], problem)
|
||||
return bad
|
||||
return {}
|
||||
|
||||
|
||||
def _call(base: str, token_header: str, method: str, path: str, body=None):
|
||||
req = urllib.request.Request(
|
||||
base + path,
|
||||
data=json.dumps(body).encode() if body is not None else None,
|
||||
method=method,
|
||||
headers={
|
||||
"Authorization": token_header,
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
# urllib's default UA is 403'd as a bot by GitHub's edge and CF.
|
||||
"User-Agent": "windy-git-pr-bridge/1",
|
||||
},
|
||||
)
|
||||
# Transport errors (TLS handshake timeout, reset) are retried: one GitHub
|
||||
# blip used to fail the whole sync, flip its heartbeat to ok:false and page
|
||||
# someone for nothing. HTTP errors are answers, not blips — never retried.
|
||||
for attempt in range(3):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
raw = r.read()
|
||||
return r.status, (json.loads(raw) if raw else None)
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, None
|
||||
except (urllib.error.URLError, TimeoutError, ConnectionError):
|
||||
if attempt == 2:
|
||||
raise
|
||||
time.sleep(2 * (attempt + 1))
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def gitea(method, path, body=None):
|
||||
return _call(GITEA + "/api/v1", f"token {GITEA_TOKEN}", method, path, body)
|
||||
|
||||
|
||||
def github(method, path, body=None):
|
||||
return _call("https://api.github.com", f"Bearer {GITHUB_TOKEN}", method, path, body)
|
||||
|
||||
|
||||
def sync_prs(repo: str) -> list[str]:
|
||||
"""Mirror open same-repo GitHub PRs into Windy Git. Returns their head shas."""
|
||||
st, gh_prs = github("GET", f"/repos/{GH_OWNER}/{repo}/pulls?state=open&per_page=100")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: GitHub PR list -> {st}")
|
||||
st, wg_prs = gitea("GET", f"/repos/{WG_OWNER}/{repo}/pulls?state=open&limit=50")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git PR list -> {st}")
|
||||
ours = {p["title"].split("]")[0] + "]": p for p in wg_prs if p["title"].startswith(MIRROR_TAG)}
|
||||
|
||||
heads, wanted = [], set()
|
||||
for pr in gh_prs:
|
||||
if pr["head"]["repo"] is None or pr["head"]["repo"]["full_name"] != f"{GH_OWNER}/{repo}":
|
||||
continue # fork PR — never synced, never run here
|
||||
tag = f"{MIRROR_TAG}{pr['number']}]"
|
||||
wanted.add(tag)
|
||||
heads.append(pr["head"]["sha"])
|
||||
if tag in ours:
|
||||
continue
|
||||
st, _ = gitea(
|
||||
"POST",
|
||||
f"/repos/{WG_OWNER}/{repo}/pulls",
|
||||
{
|
||||
"head": pr["head"]["ref"],
|
||||
"base": pr["base"]["ref"],
|
||||
"title": f"{tag} {pr['title']}"[:250],
|
||||
"body": f"Mirror of {pr['html_url']} so CI runs here. Do not merge in Windy Git — "
|
||||
"GitHub is the source of truth; merge there.",
|
||||
},
|
||||
)
|
||||
print(f" {repo}: opened mirror PR for GH#{pr['number']} -> {st}")
|
||||
|
||||
for tag, p in ours.items():
|
||||
if tag not in wanted:
|
||||
gitea("PATCH", f"/repos/{WG_OWNER}/{repo}/pulls/{p['number']}", {"state": "closed"})
|
||||
print(f" {repo}: closed mirror PR {tag} (closed on GitHub)")
|
||||
return heads
|
||||
|
||||
|
||||
def post_statuses(repo: str, sha: str) -> None:
|
||||
# Gitea caps a page at 50 (MAX_RESPONSE_ITEMS) whatever `limit` says, and a
|
||||
# daily scheduled workflow can push a quiet main's runs off page 1.
|
||||
runs = []
|
||||
for page in range(1, 6):
|
||||
st, body = gitea("GET", f"/repos/{WG_OWNER}/{repo}/actions/tasks?limit=50&page={page}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"{repo}: Windy Git runs -> {st}")
|
||||
runs += body.get("workflow_runs", [])
|
||||
if len(body.get("workflow_runs", [])) < 50:
|
||||
break
|
||||
latest: dict[str, dict] = {}
|
||||
for r in runs:
|
||||
if r["head_sha"] != sha or NO_DAEMON_JOB.search(r["name"]):
|
||||
continue
|
||||
if f"{r['workflow_id'].removesuffix('.yml')}/{r['name']}" in NON_BLOCKING.get(repo, ()):
|
||||
continue
|
||||
ctx = f"windy-git/{r['workflow_id'].removesuffix('.yml')}/{r['name']}"
|
||||
if ctx not in latest or r["id"] > latest[ctx]["id"]:
|
||||
latest[ctx] = r
|
||||
bad = invalid_workflows(repo, sha)
|
||||
if not (latest or bad):
|
||||
return
|
||||
|
||||
st, existing = github("GET", f"/repos/{GH_OWNER}/{repo}/commits/{sha}/statuses?per_page=100")
|
||||
current: dict[str, str] = {}
|
||||
for s in existing or []: # newest first
|
||||
current.setdefault(s["context"], s["state"])
|
||||
|
||||
for ctx, (path, problem) in sorted(bad.items()):
|
||||
if current.get(ctx) == "error":
|
||||
continue
|
||||
st, _ = github(
|
||||
"POST",
|
||||
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
|
||||
{
|
||||
"state": "error",
|
||||
"context": ctx,
|
||||
"description": f"Windy Git ignored this workflow, no CI ran: {problem}"[:140],
|
||||
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/src/commit/{sha}/{path}",
|
||||
},
|
||||
)
|
||||
print(f" {repo}@{sha[:7]} {ctx} = error ({problem}) -> {st}")
|
||||
|
||||
for ctx, r in sorted(latest.items()):
|
||||
state = STATE.get(r["status"])
|
||||
if state is None or current.get(ctx) == state:
|
||||
continue
|
||||
st, _ = github(
|
||||
"POST",
|
||||
f"/repos/{GH_OWNER}/{repo}/statuses/{sha}",
|
||||
{
|
||||
"state": state,
|
||||
"context": ctx,
|
||||
"description": f"Windy Git CI on Veron 1: {r['status']}"[:140],
|
||||
"target_url": f"{PUBLIC}/{WG_OWNER}/{repo}/actions/runs/{r['run_number']}",
|
||||
},
|
||||
)
|
||||
print(f" {repo}@{sha[:7]} {ctx} = {state} -> {st}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not (GITEA_TOKEN and GITHUB_TOKEN):
|
||||
sys.exit("GITEA_ADMIN_TOKEN and GITHUB_TOKEN are required")
|
||||
failed = 0
|
||||
for repo in REPOS:
|
||||
try:
|
||||
shas = sync_prs(repo)
|
||||
st, br = github("GET", f"/repos/{GH_OWNER}/{repo}")
|
||||
if st == 200:
|
||||
st, b = github("GET", f"/repos/{GH_OWNER}/{repo}/branches/{br['default_branch']}")
|
||||
if st == 200:
|
||||
shas.append(b["commit"]["sha"])
|
||||
for sha in dict.fromkeys(shas):
|
||||
post_statuses(repo, sha)
|
||||
except Exception as e: # one repo's failure must not hide the others'
|
||||
print(f" FAILED {repo}: {e}")
|
||||
failed = 1
|
||||
return failed
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
24
scripts/promote_to_ci.sh
Executable file
24
scripts/promote_to_ci.sh
Executable file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# promote_to_ci.sh <repo> [workflow-to-disable ...] — pull mirror -> writable CI repo.
|
||||
# Run ON Veron as root. See docs/CUTOVER.md "Onboarding another private repo".
|
||||
#
|
||||
# ⚠️ It DELETES the mirror before importing (Gitea's migrate refuses an existing
|
||||
# name). If the import then fails, the Windy Git copy is gone until you re-run —
|
||||
# GitHub and the nightly R2 bundles still hold everything, but check first that
|
||||
# scripts/import_from_github.py will accept the repo. (2026-09-23: windy-pro was
|
||||
# deleted this way while the importer still refused it by name.)
|
||||
set -euo pipefail
|
||||
set -a; . /srv/windygit/src/.env; set +a
|
||||
export IMPORT_GITEA_URL=http://localhost:3080
|
||||
A=http://localhost:3080/api/v1; H="Authorization: token $GITEA_ADMIN_TOKEN"; r=$1; shift
|
||||
info=$(curl -s -H "$H" $A/repos/windyadmin/$r)
|
||||
m=$(echo "$info" | python3 -c 'import json,sys;print(json.load(sys.stdin).get("mirror"))')
|
||||
if [ "$m" = True ]; then
|
||||
curl -sf -o /dev/null -X DELETE -H "$H" $A/repos/windyadmin/$r
|
||||
(cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1)
|
||||
elif [ "$m" = False ]; then echo "$r already writable"; else echo "$r absent -> importing"; (cd /srv/windygit/src && python3 scripts/import_from_github.py "$r" | tail -1); fi
|
||||
db=$(curl -s -H "$H" $A/repos/windyadmin/$r | python3 -c 'import json,sys;print(json.load(sys.stdin).get("default_branch","main"))')
|
||||
for i in $(seq 1 120); do curl -sf -o /dev/null -H "$H" $A/repos/windyadmin/$r/branches/$db && break; sleep 5; done
|
||||
for w in "$@"; do printf " disable %s: " "$w"; curl -s -o /dev/null -w '%{http_code}\n' -X PUT -H "$H" $A/repos/windyadmin/$r/actions/workflows/$w/disable; done
|
||||
curl -s -H "$H" $A/repos/windyadmin/$r/actions/workflows | python3 -c 'import json,sys,os;print(" "+os.environ.get("R",""),[(w["path"].split("/")[-1],w["state"]) for w in json.load(sys.stdin).get("workflows",[])])'
|
||||
echo " default=$db"
|
||||
@@ -38,7 +38,17 @@ FAILED=0
|
||||
|
||||
# Repos Windy Git tracks FROM GitHub. Remove a repo from this list at the moment
|
||||
# it flips to Windy-Git-first, or the sync will fight its authors and win.
|
||||
REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent}"
|
||||
REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent windy-git windy-chat windy-mail windy-connect windy-drops windy-code-web windy-code windy-traveler windy-translate windytranslate-site windytraveler-site windy-hand windy-cloud-domains windy-cloud-vps windytalk windy-pro}"
|
||||
|
||||
# Repos whose TAGS must not reach Windy Git. A tag push fires `on: push: tags`
|
||||
# workflows; windy-pro's build-electron is a matrix over ubuntu/macos/windows-
|
||||
# latest, labels no runner here has, so every leg would queue forever (and
|
||||
# queued jobs are invisible in /actions/tasks). Releases are built elsewhere.
|
||||
NO_TAGS="${SYNC_NO_TAGS:-windy-pro}"
|
||||
|
||||
# `archive/*` branches never reach Windy Git (negative refspec, git >= 2.29).
|
||||
# They are off-machine safety copies of unpushed work (one-repo doctrine), not
|
||||
# work in progress: GitHub holds them, and CI time on them is waste.
|
||||
|
||||
mkdir -p "$WORK"
|
||||
log() { printf '[sync %s] %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }
|
||||
@@ -63,12 +73,26 @@ for r in $REPOS; do
|
||||
|
||||
if git --git-dir="$bare" push --quiet --force \
|
||||
"https://${WG_OWNER}:${GITEA_ADMIN_TOKEN}@${WG}/${WG_OWNER}/${r}.git" \
|
||||
'+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*' 2>/dev/null; then
|
||||
'+refs/heads/*:refs/heads/*' '^refs/heads/archive/*' $([[ " $NO_TAGS " == *" $r "* ]] || echo '+refs/tags/*:refs/tags/*') 2>/dev/null; then
|
||||
log "$r ok (${before:0:7})"
|
||||
else
|
||||
log "FAILED push $r -> windy git"; FAILED=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Jobs that name labels no runner has (ubuntu/macos/windows-latest) would wait
|
||||
# forever and invisibly; cancel them after 30 min. Never fails the sync.
|
||||
bash "$(dirname "$0")/cancel_unrunnable.sh" || log "janitor failed (non-fatal)"
|
||||
|
||||
# Private repos can't run GitHub Actions; mirror their open PRs here so CI
|
||||
# fires, and post the verdicts back to GitHub as commit statuses.
|
||||
if ! python3 "$(dirname "$0")/pr_status_bridge.py"; then
|
||||
log "FAILED pr status bridge"; FAILED=1
|
||||
fi
|
||||
|
||||
# CI telemetry -> admin.windyword.ai (shapes declared with Windy Telemetry 40).
|
||||
# Sends nothing until WINDYGIT_TELEMETRY_TOKEN is set; never fails the sync.
|
||||
python3 "$(dirname "$0")/telemetry_emit.py" || log "telemetry emit failed (non-fatal)"
|
||||
|
||||
[[ "$FAILED" -ne 0 ]] && { log "COMPLETED WITH FAILURES"; exit 1; }
|
||||
log "all repos in step with GitHub"
|
||||
|
||||
276
scripts/telemetry_emit.py
Normal file
276
scripts/telemetry_emit.py
Normal file
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit Windy Git CI telemetry to admin.windyword.ai (Windy Telemetry 40's ledger).
|
||||
|
||||
Runs on Veron after every sync (root; reads the gitea DB via `docker exec`).
|
||||
Shapes are declared with Telemetry 40 (2026-09-23) — do not add keys or enum
|
||||
values without re-declaring: a declared family quarantines any row that
|
||||
doesn't match.
|
||||
|
||||
ci.run one row per FINISHED job, exactly once — cursor on
|
||||
(finish time, job id) in STATE; jobs finish out of id order
|
||||
service.health one row per invocation: CI plane counts for the interval
|
||||
|
||||
Privacy: ids, names of repos/jobs, codes, counts, durations. No commit
|
||||
messages, no logs, no author names.
|
||||
|
||||
--dry-run print the batch instead of posting (and don't advance STATE)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import UTC, datetime
|
||||
|
||||
INGEST = os.environ.get("TELEMETRY_INGEST_URL", "https://admin.windyword.ai/v1/events")
|
||||
TOKEN = os.environ.get("WINDYGIT_TELEMETRY_TOKEN", "")
|
||||
STATE = os.environ.get("TELEMETRY_STATE", "/var/lib/windy-git/telemetry-state.json")
|
||||
PLATFORM, SERVICE = "windy-git", "ci"
|
||||
OUTCOME = {1: "success", 2: "failure", 3: "cancelled", 4: "skipped"}
|
||||
EVENTS = {"push", "pull_request", "pull_request_sync", "schedule", "workflow_dispatch"}
|
||||
RUNNERS_EXPECTED = 6
|
||||
|
||||
|
||||
def sql(query: str) -> list[dict]:
|
||||
"""Rows as dicts, via psql's json_agg — no driver needed on the host."""
|
||||
wrapped = f"select coalesce(json_agg(t), '[]'::json) from ({query}) t;"
|
||||
out = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"exec",
|
||||
"-i",
|
||||
"windy-git-db-1",
|
||||
"sh",
|
||||
"-c",
|
||||
'psql -U "$POSTGRES_USER" -d gitea -At -v ON_ERROR_STOP=1',
|
||||
],
|
||||
input=wrapped,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout.strip()
|
||||
return json.loads(out or "[]")
|
||||
|
||||
|
||||
def load_state() -> dict:
|
||||
try:
|
||||
with open(STATE) as f:
|
||||
return json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
|
||||
def iso(epoch: float) -> str:
|
||||
return datetime.fromtimestamp(epoch, UTC).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
dry = "--dry-run" in sys.argv
|
||||
state = load_state()
|
||||
now = time.time()
|
||||
since = float(state.get("last_ts", now - 300))
|
||||
# Cursor = (finish time, job id), NOT job id alone: jobs finish out of id
|
||||
# order, so an id high-water mark silently drops every long job that started
|
||||
# before the mark and finished after it (Telemetry Boss caught this: 43
|
||||
# finished vs 8 ci.run rows). Finish time = stopped, or updated for jobs
|
||||
# Gitea/the janitor skipped without a stop time.
|
||||
if "last_fin" in state:
|
||||
last_fin, last_id = int(state["last_fin"]), int(state["last_id"])
|
||||
else: # first run or pre-cursor state: start now, never replay history
|
||||
last_fin, last_id = int(state.get("last_ts", now)), 0
|
||||
cutoff = int(now) - 5 # leave the current second alone; late writers land next run
|
||||
FIN = "coalesce(nullif(j.stopped, 0), j.updated)"
|
||||
|
||||
jobs = sql(f"""
|
||||
select j.id, j.name as job, j.status, j.started, j.stopped, {FIN} as fin,
|
||||
p.lower_name as repo, p.default_branch, r.workflow_id, r.event,
|
||||
r.ref, r.index as run, left(r.commit_sha, 7) as sha
|
||||
from action_run_job j
|
||||
join action_run r on r.id = j.run_id
|
||||
join repository p on p.id = r.repo_id
|
||||
where j.status in (1, 2, 3, 4)
|
||||
and ({FIN}, j.id) > ({last_fin}, {last_id})
|
||||
and {FIN} <= {cutoff}
|
||||
order by {FIN}, j.id
|
||||
limit 2000""")
|
||||
|
||||
try: # posted_to_github: the bridge's own rules, from the same checkout
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import pr_status_bridge as bridge
|
||||
except Exception: # noqa: BLE001
|
||||
bridge = None
|
||||
|
||||
events = []
|
||||
for j in jobs:
|
||||
ref = j["ref"] or ""
|
||||
if ref.startswith("refs/pull/"):
|
||||
kind = "pr"
|
||||
elif ref == f"refs/heads/{j['default_branch']}":
|
||||
kind = "default"
|
||||
else:
|
||||
kind = "other"
|
||||
# Gitea stores whole seconds; duration_ms is seconds*1000 (so 10000 = 10 s).
|
||||
dur = (j["stopped"] - j["started"]) * 1000 if j["started"] and j["stopped"] else None
|
||||
ev = {
|
||||
"ts": iso(j["stopped"]),
|
||||
"platform": PLATFORM,
|
||||
"service": SERVICE,
|
||||
"event_type": "ci.run",
|
||||
"actor_type": "system",
|
||||
"metadata": {
|
||||
"repo": j["repo"],
|
||||
"workflow": (j["workflow_id"] or "").removesuffix(".yml").removesuffix(".yaml"),
|
||||
"job": j["job"],
|
||||
"outcome": OUTCOME[j["status"]],
|
||||
"event": j["event"] if j["event"] in EVENTS else "other",
|
||||
"branch_kind": kind,
|
||||
"run": j["run"],
|
||||
"sha": j["sha"],
|
||||
},
|
||||
}
|
||||
if dur is not None and dur >= 0:
|
||||
ev["duration_ms"] = int(dur)
|
||||
if bridge is not None:
|
||||
wf = ev["metadata"]["workflow"]
|
||||
ev["metadata"]["posted_to_github"] = bool(
|
||||
j["repo"] in {r.lower() for r in bridge.REPOS}
|
||||
and kind in ("default", "pr")
|
||||
and j["status"] != 4
|
||||
and not bridge.NO_DAEMON_JOB.search(j["job"])
|
||||
and f"{wf}/{j['job']}" not in bridge.NON_BLOCKING.get(j["repo"], set())
|
||||
)
|
||||
events.append(ev)
|
||||
|
||||
# --- heartbeat: counts since the previous invocation --------------------
|
||||
# Interval counts come from EXACTLY the rows emitted above, so
|
||||
# sum(jobs_finished) over any window == count(ci.run) in it, by construction.
|
||||
h = sql(f"""
|
||||
select
|
||||
(select count(*) from action_run_job where status in (5, 7)) as jobs_waiting,
|
||||
(select count(*) from action_run_job where status = 6) as jobs_running,
|
||||
(select count(*) from action_runner where deleted is null and last_online >= {int(now) - 120}) as runners_online,
|
||||
(select coalesce(extract(epoch from now())::bigint - min(created), 0)
|
||||
from action_run_job where status in (5, 7)) as oldest_waiting_s""")[0]
|
||||
h["jobs_finished"] = len(jobs)
|
||||
h["jobs_failed"] = sum(1 for j in jobs if j["status"] == 2)
|
||||
h["jobs_cancelled"] = sum(1 for j in jobs if j["status"] == 3)
|
||||
meta = {k: int(v) for k, v in h.items()}
|
||||
meta["interval_s"] = int(now - since) # ecosystem-standard key
|
||||
# UPDATE 7. Quarantines seen on earlier sends (the ledger answers 202 anyway)
|
||||
# are carried in the state file until a heartbeat reports them. Dropped is 0
|
||||
# by construction: a failed send keeps the cursor and the spool, so every row
|
||||
# is re-sent next run (a partial failure can duplicate, never lose).
|
||||
meta["telemetry_quarantined"] = int(state.get("quarantined_unreported", 0))
|
||||
meta["telemetry_dropped"] = 0
|
||||
for k in ("repos_synced", "repos_sync_failed", "statuses_posted", "bridge_errors"):
|
||||
v = os.environ.get(f"TELEMETRY_{k.upper()}")
|
||||
if v is not None and v.isdigit(): # absent = couldn't count; never invent 0
|
||||
meta[k] = int(v)
|
||||
events.append(
|
||||
{
|
||||
"ts": iso(now),
|
||||
"platform": PLATFORM,
|
||||
"service": SERVICE,
|
||||
"event_type": "service.health",
|
||||
"actor_type": "system",
|
||||
"metadata": meta,
|
||||
}
|
||||
)
|
||||
|
||||
# ci.job_cancelled: spooled by the janitor (cancel_unrunnable.sh), one JSON per job.
|
||||
spool = os.environ.get("JANITOR_SPOOL", "/var/lib/windy-git/janitor-cancelled.jsonl")
|
||||
spooled = 0
|
||||
try:
|
||||
with open(spool) as f:
|
||||
for line in f:
|
||||
try:
|
||||
m = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
events.append(
|
||||
{
|
||||
"ts": iso(now),
|
||||
"platform": PLATFORM,
|
||||
"service": SERVICE,
|
||||
"event_type": "ci.job_cancelled",
|
||||
"actor_type": "system",
|
||||
"metadata": {
|
||||
k: m[k]
|
||||
for k in ("repo", "workflow", "job", "reason", "runs_on", "waited_s")
|
||||
},
|
||||
}
|
||||
)
|
||||
spooled += 1
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
if dry:
|
||||
out = os.environ.get("TELEMETRY_DRY_OUT")
|
||||
if out:
|
||||
with open(out, "w") as f:
|
||||
json.dump({"events": events}, f)
|
||||
else:
|
||||
print(json.dumps({"events": events}, indent=1)[:4000])
|
||||
print(f"[telemetry] DRY RUN: {len(events)} events ({len(jobs)} ci.run)")
|
||||
return 0
|
||||
if not TOKEN:
|
||||
print("[telemetry] WINDYGIT_TELEMETRY_TOKEN unset — not sending (not a failure)")
|
||||
return 0
|
||||
|
||||
quarantined = 0
|
||||
for i in range(0, len(events), 500):
|
||||
req = urllib.request.Request(
|
||||
INGEST,
|
||||
data=json.dumps({"events": events[i : i + 500]}).encode(),
|
||||
method="POST",
|
||||
headers={
|
||||
"Authorization": f"Bearer {TOKEN}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "windy-git-telemetry/1",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
body = r.read()
|
||||
if r.status >= 300:
|
||||
raise urllib.error.HTTPError(
|
||||
INGEST, r.status, body[:300].decode(errors="replace"), None, None
|
||||
)
|
||||
try:
|
||||
resp = json.loads(body or b"{}")
|
||||
except ValueError:
|
||||
resp = {}
|
||||
q = resp.get("quarantined") if isinstance(resp, dict) else None
|
||||
if isinstance(q, int) and q > 0:
|
||||
quarantined += q
|
||||
reasons = "; ".join(map(str, resp.get("rejections") or [])) or "no reason given"
|
||||
print(f"[telemetry] WARNING {q} row(s) QUARANTINED by the ledger: {reasons}")
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f"[telemetry] FAILED ingest HTTP {e.code}: {e.read()[:200]!r}")
|
||||
return 1 # state NOT advanced: the same rows retry next run
|
||||
except urllib.error.URLError as e:
|
||||
print(f"[telemetry] FAILED ingest: {e.reason}")
|
||||
return 1
|
||||
|
||||
if spooled:
|
||||
open(spool, "w").close() # only after every batch was accepted
|
||||
os.makedirs(os.path.dirname(STATE), exist_ok=True)
|
||||
new_fin, new_id = (jobs[-1]["fin"], jobs[-1]["id"]) if jobs else (last_fin, last_id)
|
||||
with open(STATE + ".tmp", "w") as f:
|
||||
json.dump(
|
||||
{"last_fin": new_fin, "last_id": new_id, "last_ts": now,
|
||||
"quarantined_unreported": quarantined},
|
||||
f,
|
||||
)
|
||||
os.replace(STATE + ".tmp", STATE)
|
||||
print(f"[telemetry] sent {len(events)} events ({len(jobs)} ci.run)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user