G7.6: never let bookkeeping kill the monitor
All checks were successful
check / gate (push) Successful in 18s
canary / probe (push) Successful in 26s

An unwritable state path raised and took the whole canary down. That is the
worst possible trade for a monitoring tool: it reports nothing at all, and
reports it silently. State is an optimisation for transition detection; the
probing is the point.

Found by fat-fingering an env var, which is exactly how it would happen in
production.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Grant Whitmer
2026-08-12 15:42:58 -04:00
parent a094960da3
commit ef3450d87b
2 changed files with 22 additions and 2 deletions

View File

@@ -156,8 +156,20 @@ def load_state() -> dict:
def save_state(results: list[Result]) -> None:
with open(STATE_PATH, "w") as f:
json.dump({r.name: r.status for r in results}, f, indent=2)
"""Never let bookkeeping kill the monitor.
State is an optimisation — it lets the next run tell "still broken" from
"just broke". The probing is the valuable part. An unwritable path used to
raise here and take the whole canary down, which is the worst possible
trade: a monitoring tool that dies of a config problem reports nothing at
all, and reports it silently.
"""
try:
with open(STATE_PATH, "w") as f:
json.dump({r.name: r.status for r in results}, f, indent=2)
except OSError as exc:
print(f"!! could not save state to {STATE_PATH}: {exc}")
print(" (probes still ran; transition detection is degraded this run)")
def send_alert(subject: str, lines: list[str]) -> bool: