From 390c1e7479d7fa6de65d1ca09eb31149c0c28d8a Mon Sep 17 00:00:00 2001 From: Grant Whitmer Date: Wed, 23 Sep 2026 01:29:57 -0400 Subject: [PATCH] ops: move tunnel metrics to 2001, sync windy-git into itself windygit-tunnel had crash-looped ~91k times: another project's cornercall-tunnel holds 127.0.0.1:2000, and cloudflared exits when it cannot bind its metrics port. Ingress only survived because a stray cloudflared.service ran the same config. That unit is now disabled and /etc/cloudflared/config.yml uses metrics 127.0.0.1:2001. Also add windy-git to the GitHub->Windy Git sync list; its self-hosted copy was stuck 3 commits behind (only check + canary workflows, no deploys, so syncing is safe). Co-Authored-By: Claude Opus 5.5 --- SUBSTRATE.md | 2 +- api/app/providers/registry.py | 2 +- docs/RUNBOOK-VERON.md | 10 +++++++++- scripts/sync_from_github.sh | 2 +- 4 files changed, 12 insertions(+), 4 deletions(-) diff --git a/SUBSTRATE.md b/SUBSTRATE.md index 282bd3e..2b43834 100644 --- a/SUBSTRATE.md +++ b/SUBSTRATE.md @@ -22,7 +22,7 @@ boot guard in `api/app/main.py` that refuses to start there in production. | 8600 | `windy-git-api` — our plane | | **3080** | Gitea — host 3000 and 3300 are taken by resident projects on Veron 1 | | 5432 | Postgres | -| 2000 | cloudflared metrics (probe target) | +| 2001 | cloudflared metrics — NOT 2000: `cornercall-tunnel` (another project) takes 2000, and a metrics bind failure kills the whole tunnel | ## Ingress — Cloudflare Tunnel `windy-git` diff --git a/api/app/providers/registry.py b/api/app/providers/registry.py index 7e27fa3..ff128ca 100644 --- a/api/app/providers/registry.py +++ b/api/app/providers/registry.py @@ -105,7 +105,7 @@ class DatabaseProvider(Provider): # TunnelProvider was removed deliberately. See the note in main.py: cloudflared -# binds 127.0.0.1:2000 on the HOST, and this process runs in a container whose +# binds 127.0.0.1:2001 on the HOST, and this process runs in a container whose # only route to the host is the bridge gateway (172.17.0.1), where nothing is # listening. Binding the metrics endpoint wider would fix the probe and make a # metrics bind failure able to take down ingress -- a worse trade than losing diff --git a/docs/RUNBOOK-VERON.md b/docs/RUNBOOK-VERON.md index ad16a2c..ca0b17d 100644 --- a/docs/RUNBOOK-VERON.md +++ b/docs/RUNBOOK-VERON.md @@ -22,7 +22,7 @@ boot in production if it finds itself on `72.60.118.54`. |---|---| | `127.0.0.1:3080` | Gitea (host 3000 is a resident node dev server; 3300 is nginx — **do not fight them for a port**) | | `127.0.0.1:8600` | windy-git API | -| `127.0.0.1:2000` | cloudflared metrics | +| `127.0.0.1:2001` | cloudflared metrics (`metrics:` in `/etc/cloudflared/config.yml`) — **not 2000**, see Troubleshooting | **No inbound port is opened.** cloudflared dials out, so the dynamic residential IP is irrelevant and there is no firewall hole to maintain. @@ -77,6 +77,14 @@ sudo ss -tlnp | grep -E "3080|8600" # both must be 127.0.0.1 **A hostname returns 530 or won't resolve** — the tunnel is down. `sudo systemctl restart windygit-tunnel`, then `journalctl -u windygit-tunnel -n 50`. +**`windygit-tunnel` crash-loops with `bind: address already in use` on the metrics +port** — cloudflared exits if it cannot bind `metrics:`, taking ingress with it. +Until 2026-09-23 this unit restarted ~91,000 times because another project's +`cornercall-tunnel` held 127.0.0.1:2000; ingress only survived because a stray +generic `cloudflared.service` ran the same config (now disabled). Windy Git's +metrics port is **2001**. `sudo ss -ltnp | grep :2001` names any squatter. +Keep exactly ONE unit running `/etc/cloudflared/config.yml`: `windygit-tunnel`. + **TLS handshake fails with `curl` exit 35 and no HTTP status at all** — someone added a **two-level** hostname. Free Universal SSL covers `windygit.com` and `*.windygit.com` only. The request dies before the tunnel is consulted, so it diff --git a/scripts/sync_from_github.sh b/scripts/sync_from_github.sh index 4c9c935..2fdd85f 100755 --- a/scripts/sync_from_github.sh +++ b/scripts/sync_from_github.sh @@ -38,7 +38,7 @@ FAILED=0 # Repos Windy Git tracks FROM GitHub. Remove a repo from this list at the moment # it flips to Windy-Git-first, or the sync will fight its authors and win. -REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent}" +REPOS="${SYNC_REPOS:-windy-calendar windy-search windy-registry Windy-Clone WindyCloud windy-cloud-sites windy-mind eternitas windy-agent windy-git}" mkdir -p "$WORK" log() { printf '[sync %s] %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }