From the 2026-08-23 live dispatch of PR #535 (the restore-failed incident):
What happened. The verb froze a verified artifact (c3eade50) and kicked via launchctl kickstart -k. The new binary flapped 24× under launchd (ThrottleInterval=10) — dying silently between the skills-staging lines and the server's first output. The same binary ran clean in foreground with near-identical env; a DIFFERENT build of the SAME source (2b83d011) booted once-clean under launchd minutes later; and even the RESTORED previous binary flapped during the same window before recovering. Verdict: not the binary — an environmental race in the kick sequence.
Hypothesis. kickstart -k kills the old instance and starts the new one without waiting for the old process to exit. During the drain window (SQLite DB lock / port still held), the new instance dies silently; launchd throttles and retries into the same race; the verb's 120s health poll sees connection-refused and correctly declares failure — then the RESTORE kick hits the same race.
Fix direction. After kickstart: (1) wait for the pre-kick pid to actually exit (poll the pid, not the port) before starting the health window; (2) in the health poll, distinguish connection-refused-with-launchd-retrying (booting/throttled — keep waiting) from N consecutive silent-restart cycles (flapping — fail fast into restore); (3) after a restore-kick, wait for pid-exit too. Also worth logging launchctl print state + last-exit-status into the receipt detail on failure.
The incident itself validated the safety design: the verb detected real unhealthiness through two windows, restored the previous binary, wrote an honest receipt, and never lied. This issue is about making the kick sequence as robust as the rollback.
From the 2026-08-23 live dispatch of PR #535 (the restore-failed incident):
What happened. The verb froze a verified artifact (c3eade50) and kicked via
launchctl kickstart -k. The new binary flapped 24× under launchd (ThrottleInterval=10) — dying silently between the skills-staging lines and the server's first output. The same binary ran clean in foreground with near-identical env; a DIFFERENT build of the SAME source (2b83d011) booted once-clean under launchd minutes later; and even the RESTORED previous binary flapped during the same window before recovering. Verdict: not the binary — an environmental race in the kick sequence.Hypothesis.
kickstart -kkills the old instance and starts the new one without waiting for the old process to exit. During the drain window (SQLite DB lock / port still held), the new instance dies silently; launchd throttles and retries into the same race; the verb's 120s health poll sees connection-refused and correctly declares failure — then the RESTORE kick hits the same race.Fix direction. After kickstart: (1) wait for the pre-kick pid to actually exit (poll the pid, not the port) before starting the health window; (2) in the health poll, distinguish connection-refused-with-launchd-retrying (booting/throttled — keep waiting) from N consecutive silent-restart cycles (flapping — fail fast into restore); (3) after a restore-kick, wait for pid-exit too. Also worth logging
launchctl printstate + last-exit-status into the receipt detail on failure.The incident itself validated the safety design: the verb detected real unhealthiness through two windows, restored the previous binary, wrote an honest receipt, and never lied. This issue is about making the kick sequence as robust as the rollback.