feat: fail node probes on a stale tip

[scarmuega]

Why

The probe script only checks syncProgress within 99–100. It has two decimals, so on mainnet it still reads ≥ 99 with a tip weeks old. On 2026-09-30 node-mainnet-b02 stopped adopting blocks at 13:03 UTC and stayed Ready until it was restarted manually about 90 minutes later. node-mainnet-stable kept routing Ogmios sessions to it the whole time. For a few minutes it was the only endpoint, and every mainnet Ogmios pod served a tip 95+ blocks old.

What

• readiness.sh [MAX_TIP_AGE_SECONDS]: with the argument, the script also requires the tip slot to be at or after the slot of now - MAX_TIP_AGE_SECONDS. The conversion is cardano-cli query slot-number, which uses the node's own era history, so it works on every network whatever the slot length. The conversion fails past the node's forecast horizon (~36 h ahead of its tip on mainnet). That only happens to a stale node, so it counts as a failure.
• max_tip_age_seconds is a new optional field on readiness_probe, liveness_probe and startup_probe. Each probe passes its own value, validated at plan time as a positive whole number. Unset keeps the current behaviour, so no instance changes until it opts in. prime-testnet keeps its socket-only check.
• Mesh peer Service publish_not_ready_addresses = true: without it, a node whose readiness probe fails drops out of its headless nodes-<salt> record. Its siblings then cache the NXDOMAIN for up to 15 minutes, well past its recovery. The Service is only used for mesh peering, which doesn't need readiness gating.

Testing

• terraform test: 9/9 pass (5 existing + 4 new in tests/probes.tftest.hcl: opt-in default, per-probe values, prime-testnet unchanged, peer Service publishes not-ready addresses).
• Checked by hand that max_tip_age_seconds = 1.5 fails the plan with the validation message. It isn't a committed test, because expect_failures can't point at a child-module variable.
• Script run in debian:bookworm-slim (the node image's base) against a stub cardano-cli, 9 scenarios: unchanged no-argument path, fresh and stale tips, past-horizon failure, syncProgress below 99, non-integer and zero arguments rejected, --testnet-magic used for both queries.
• query slot-number exists with the same bare-number output in cardano-cli 10.15.1.0 and 11.0.0.0, the versions bundled in blinklabs-io/cardano-node:11.0.1 builds.

Rollout

A follow-up change in demeter-run/clusters bumps the stage4 module ref and opts the mainnet nodes in: readiness max_tip_age_seconds = 180 (period 60, failure 3), liveness max_tip_age_seconds = 900 (period 120, failure 10). Applying it updates the node-readiness ConfigMap, the StatefulSet probe commands, and the peer Services in place.

🤖 Generated with Claude Code