fix(mesh): a restart no longer forgets how long each peer has been known - #888
Merged
Conversation
The in-memory peer table is rebuilt from nothing at startup: every peer re-enters through Peer::new with first_seen = now, and the database — which keeps the real first sighting across restarts (its upsert never overwrites first_seen) — was never read back. So for a day after every restart each peer read as brand new. Found on the v1.11.39 canary. The Wraith coordinator roster requires 24h of maturity (eligibility::EligibilityPolicy), so a restarted node's roster was itself alone: it led all four tiers and reported degraded, while the not-yet-upgraded nodes (no maturity rule) held full rosters. It would have repeated on every restart, unattended upgrades included. On a peer's first health ping per process, its first sighting is restored from the database. Only ever earlier (PeerManager::backdate_first_seen), one read per peer per restart, retried on a database error. ⚠ Correction: #883's description ruled maturity out because /api/v1/network/peers showed ~71-day first sightings. That endpoint reads the DATABASE (db.get_active_peers), not the in-memory table the roster reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⛔ Hold the merge until the v1.11.39 roll has finished.
release.shrefuses to continue ifmainmoves underbins/orcrates/. This ships as v1.11.40.What happened
Found on the first v1.11.39 canary (vm5): its Wraith roster was only itself. It led all four tiers and reported
degraded, while the not-yet-upgraded nodes held rosters of 7–8 that included vm5.EligibilityPolicy::maturity_secs).first_seenfrom the in-memory peer table.Peer::newwithfirst_seen = now. The database keeps the real first sighting across restarts (its upsert never overwritesfirst_seen) and was never read back.So for a day after every restart, unattended upgrades included, a node judged every peer brand new, and its roster held itself alone. v1.11.38 had no maturity rule, which is why the old nodes' rosters were full.
The fix
On a peer's first health ping per process,
HealthPingHandler::restore_peer_agereads its database record andPeerManager::backdate_first_seenmoves the in-memory value back. It only ever moves earlier: a later value would make a known peer look newer, which is the defect. It costs one database read per peer per restart, and retries on a database error rather than marking the peer done. Every consumer offirst_seengets the real age, not just the roster.⚠ Correction to #883
#883's description ruled maturity out because
/api/v1/network/peersshowed ~71-day first sightings. That endpoint reads the database (db.get_active_peers), not the in-memory table the roster reads. I measured the wrong store.Verification
a_health_ping_after_a_restart_restores_the_peers_agedrives a real ping throughhandle_message. Removing the one call in the ping path fails it (checked by hand), so it pins the wiring and not just the helper.ghost-consensus391/391. The CI clippy invocation andcargo fmt --checkare clean.After it rolls
A restarted node's
roster_sizeshould match its peers within a ping or two (~1 minute), not 24 hours later. Compareroster_commitmentacross all eight.