ci: bound CI storage; don't bridge image-build jobs

- deploy/runner/prune.sh + windygit-ci-prune.timer (6h): age-based prune
  of the CI-only dind (containers, finished-job volumes, images/builder
  cache >7d) plus a hard 60 GB cap. Only that daemon, over its own TCP
  socket; never the host's Docker. It was 38 GB and unbounded — the same
  class of growth that filled Kit 0 on 09-01.
- pr_status_bridge: jobs named *docker* are not posted. Job containers
  have no daemon by design (I-5), so they are red on every commit; a
  permanent red X teaches everyone to ignore red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-23 02:47:28 -04:00
parent e4a15869c0
commit 45686283be
6 changed files with 65 additions and 1 deletions

View File

@@ -138,3 +138,12 @@ def test_pr_mirror_opened_once_and_closed_when_github_closes(fake, monkeypatch):
f2 = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 4, "title": "[GH#7] t"}]) f2 = fake(gh_prs=[_gh_pr(7)], wg_prs=[{"number": 4, "title": "[GH#7] t"}])
bridge.sync_prs("r") bridge.sync_prs("r")
assert f2.opened == [] and f2.closed == [] assert f2.opened == [] and f2.closed == []
def test_image_build_jobs_are_not_posted(fake):
"""No Docker daemon in job containers (I-5): a build job's red is structural."""
f = fake(
runs=[_run(1, "ci.yml", "Docker Build", "failure"), _run(2, "ci.yml", "docker", "failure")]
)
bridge.post_statuses("r", SHA)
assert f.posted == []

28
deploy/runner/prune.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Keep CI storage bounded (2026-09-23).
#
# Kit 0's 09-01 wipe began with CI `_work` dirs (74 GB) + Docker filling the
# disk. Windy Git's runners have no host `_work` dir — every job runs in a
# container inside the CI-only dind — so the thing that grows here is dind's
# image/volume store (38 GB when this was written, never pruned). This prunes
# ONLY that daemon, over its own socket. It never touches the host's Docker.
#
# In-use images/volumes are never removed, so a running job is safe.
set -euo pipefail
CAP_GB="${CI_STORAGE_CAP_GB:-60}"
D=(docker exec windy-git-runner-dind-1 docker)
"${D[@]}" container prune -f --filter until=6h >/dev/null
"${D[@]}" volume prune -af >/dev/null # job workspaces of finished jobs
"${D[@]}" image prune -af --filter until=168h >/dev/null
"${D[@]}" builder prune -af --filter until=168h >/dev/null 2>&1 || true
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
if (( used_gb > CAP_GB )); then
# Over the cap even after the age-based pass: drop every unused image. The
# next jobs re-pull (the act image is ~2 GB) — slower, never wrong.
"${D[@]}" image prune -af >/dev/null
used_gb=$(du -s --block-size=1G /var/lib/docker/volumes/windy-git-runner_dind-storage | cut -f1)
fi
echo "ci storage ${used_gb}G (cap ${CAP_GB}G)"
(( used_gb <= CAP_GB )) || { echo "STILL OVER CAP"; exit 1; }

View File

@@ -0,0 +1,6 @@
[Unit]
Description=Windy Git - prune CI-only dind storage (bounded, never the host daemon)
[Service]
Type=oneshot
ExecStart=/srv/windygit/src/deploy/runner/prune.sh

View File

@@ -0,0 +1,9 @@
[Unit]
Description=Windy Git - prune CI storage every 6 hours
[Timer]
OnCalendar=*-*-* 00/6:37:00
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -107,6 +107,10 @@ their CI permanently, not a stopgap:
- Covered repos: `BRIDGE_REPOS` in the script. Public repos are left out on - Covered repos: `BRIDGE_REPOS` in the script. Public repos are left out on
purpose; they run real GitHub Actions and two verdicts per commit is noise. purpose; they run real GitHub Actions and two verdicts per commit is noise.
- `skipped` jobs post nothing — no green for a job nobody ran. - `skipped` jobs post nothing — no green for a job nobody ran.
- **Image-build jobs** (name matches `docker`) post nothing: job containers
have no Docker daemon by design (I-5), so they are red on every commit. A
rootless builder (BuildKit rootless / buildx in the capped dind) is the open
decision that would bring them back.
**Onboarding another private repo** — the promotion steps below, then: **Onboarding another private repo** — the promotion steps below, then:

View File

@@ -30,6 +30,7 @@ from __future__ import annotations
import json import json
import os import os
import re
import sys import sys
import urllib.error import urllib.error
import urllib.request import urllib.request
@@ -60,6 +61,13 @@ STATE = {
} }
MIRROR_TAG = "[GH#" MIRROR_TAG = "[GH#"
# Image-build jobs cannot pass here BY DESIGN: job containers get no Docker
# daemon (I-5 — the host socket would hand every workflow root on Veron 1).
# Posting them would put a permanent red X on every commit, and a signal that is
# always red trains everyone to ignore red. Not posted until a rootless builder
# exists; that is a decision, recorded in docs/CUTOVER.md, not a failure.
NO_DAEMON_JOB = re.compile(r"docker", re.IGNORECASE)
def _call(base: str, token_header: str, method: str, path: str, body=None): def _call(base: str, token_header: str, method: str, path: str, body=None):
req = urllib.request.Request( req = urllib.request.Request(
@@ -142,7 +150,7 @@ def post_statuses(repo: str, sha: str) -> None:
break break
latest: dict[str, dict] = {} latest: dict[str, dict] = {}
for r in runs: for r in runs:
if r["head_sha"] != sha: if r["head_sha"] != sha or NO_DAEMON_JOB.search(r["name"]):
continue continue
ctx = f"windy-git/{r['workflow_id'].removesuffix('.yml')}/{r['name']}" ctx = f"windy-git/{r['workflow_id'].removesuffix('.yml')}/{r['name']}"
if ctx not in latest or r["id"] > latest[ctx]["id"]: if ctx not in latest or r["id"] > latest[ctx]["id"]: