Skip to content

fix(mesh): a restart no longer forgets how long each peer has been known - #888

Merged
defenwycke merged 1 commit into
mainfrom
fix/peer-age-survives-restart
Sep 11, 2026
Merged

fix(mesh): a restart no longer forgets how long each peer has been known#888
defenwycke merged 1 commit into
mainfrom
fix/peer-age-survives-restart

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

Hold the merge until the v1.11.39 roll has finished. release.sh refuses to continue if main moves under bins/ or crates/. This ships as v1.11.40.

What happened

Found on the first v1.11.39 canary (vm5): its Wraith roster was only itself. It led all four tiers and reported degraded, while the not-yet-upgraded nodes held rosters of 7–8 that included vm5.

  • The coordinator roster requires 24h of maturity (EligibilityPolicy::maturity_secs).
  • It reads first_seen from the in-memory peer table.
  • That table is rebuilt from nothing at startup: every peer re-enters through Peer::new with first_seen = now. The database keeps the real first sighting across restarts (its upsert never overwrites first_seen) and was never read back.

So for a day after every restart, unattended upgrades included, a node judged every peer brand new, and its roster held itself alone. v1.11.38 had no maturity rule, which is why the old nodes' rosters were full.

The fix

On a peer's first health ping per process, HealthPingHandler::restore_peer_age reads its database record and PeerManager::backdate_first_seen moves the in-memory value back. It only ever moves earlier: a later value would make a known peer look newer, which is the defect. It costs one database read per peer per restart, and retries on a database error rather than marking the peer done. Every consumer of first_seen gets the real age, not just the roster.

⚠ Correction to #883

#883's description ruled maturity out because /api/v1/network/peers showed ~71-day first sightings. That endpoint reads the database (db.get_active_peers), not the in-memory table the roster reads. I measured the wrong store.

Verification

  • a_health_ping_after_a_restart_restores_the_peers_age drives a real ping through handle_message. Removing the one call in the ping path fails it (checked by hand), so it pins the wiring and not just the helper.
  • The database-restore unit test; a peer the database has never seen keeps its sighting; backdating only ever moves earlier, ignores 0, and never creates a peer.
  • ghost-consensus 391/391. The CI clippy invocation and cargo fmt --check are clean.

After it rolls

A restarted node's roster_size should match its peers within a ping or two (~1 minute), not 24 hours later. Compare roster_commitment across all eight.

The in-memory peer table is rebuilt from nothing at startup: every peer
re-enters through Peer::new with first_seen = now, and the database — which
keeps the real first sighting across restarts (its upsert never overwrites
first_seen) — was never read back. So for a day after every restart each peer
read as brand new.

Found on the v1.11.39 canary. The Wraith coordinator roster requires 24h of
maturity (eligibility::EligibilityPolicy), so a restarted node's roster was
itself alone: it led all four tiers and reported degraded, while the
not-yet-upgraded nodes (no maturity rule) held full rosters. It would have
repeated on every restart, unattended upgrades included.

On a peer's first health ping per process, its first sighting is restored from
the database. Only ever earlier (PeerManager::backdate_first_seen), one read per
peer per restart, retried on a database error.

⚠ Correction: #883's description ruled maturity out because
/api/v1/network/peers showed ~71-day first sightings. That endpoint reads the
DATABASE (db.get_active_peers), not the in-memory table the roster reads.
@defenwycke
defenwycke merged commit cbda4a1 into main Sep 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant