fix(wraith): the coordinator roster converges — no local verdicts, no first-look freeze (#710) - #883
Merged
Merged
Conversation
… first-look freeze (#710) The night all eight mainnet nodes opted in (epoch 6711), rosters read 6/7/5/5/2/2/4/5 and the fleet elected two different coordinators for one epoch. Ten hours later it was still split. Two causes, both removed here. The view was computed once per epoch and held. Whatever the peer table looked like at that instant became the roster for the rest of the day, so a node restarted mid-epoch drew from a half-filled table and kept it. Measured: one node restarted while three others still had coordinating switched off, and its roster was still missing exactly those three ten hours later. The roster is now re-read on every refresh and the view rebuilt when it changes. The beacon and the seat target stay frozen for the epoch, so only the roster can move. Eligibility also read `qualified` and `archive` from this node's own verification ledger. No two nodes hold the same challenge history, so that input could never converge, whatever the timing. Both fields are removed from `NodeFacts`. The module doc said they were "a verdict the network reached together"; they were not. What is given up is a Sybil cost (a verified archive), which is stated in the module rather than implied. A roster that moves mid-epoch can unseat a node while it holds rounds that participants have committed inputs to, and `stop()` aborts them. So a node seated at any point in an epoch keeps serving until the epoch turns (`should_serve`); new wallets follow the current view. The mutation harness targeted the removed archive rule; it now targets the opt-in refusal instead. All 29 mutations caught.
defenwycke
added a commit
that referenced
this pull request
Sep 11, 2026
…every opted-in node At zero session demand — mainnet — `seats_for_demand` returned one seat, so one node carried all four denominations for a whole epoch while every other opted-in node sat idle and earned nothing. And the seat count was one more thing nodes had to agree on, sized from each node's own snapshot of gossiped session counters. Each tier now has its own leader. The chain beacon ranks every roster node per tier (`coordinator_order_for_tier`, which existed with no callers), and tiers are filled in turn by the best-ranked node not already leading one — so with eight nodes, every epoch has four different leaders. Ranks are independent and uniform and the rule never looks at who a node is, so every node is equally likely to lead every tier; 16,000 epochs hold every (node, tier) count within 10% of its share. Random each epoch, deliberately not a fixed rotation: a published schedule lets a malicious operator prepare for the round it will run. Leaders are known one anchor block before the epoch starts. Every opted-in node runs a coordinator. The ones not leading are every tier's failover path — the leader, then the tier's own ranking — walked in the same order by every wallet, so a dead leader's cohort moves together. That also retires #883's drain rule: a node never leaves its own roster, so a lead that moves mid-epoch leaves the old leader serving what it holds. The seat scheme is deleted rather than left beside the new one: `elect_coordinators`, `verify_election`, `shard_for`, `shard_key_for_tier_epoch`, `seat_try_order`, `pick_seat_endpoints`, `seats_for_demand`, `COORDINATOR_SEATS`. The published document carries `tiers`, `coordinators` (with `leads`), `my_tiers` and `my_endpoint`; `election_is_honest` recomputes the draw and refuses leaders, failover orders or `leads` that do not follow from the inputs, including the old seat-shaped document. The attestation swaps `my_seat` for `serving` + `my_tiers`; the challenger checks `attested_serving`.
defenwycke
added a commit
that referenced
this pull request
Sep 11, 2026
…every opted-in node (#885) At zero session demand — mainnet — `seats_for_demand` returned one seat, so one node carried all four denominations for a whole epoch while every other opted-in node sat idle and earned nothing. And the seat count was one more thing nodes had to agree on, sized from each node's own snapshot of gossiped session counters. Each tier now has its own leader. The chain beacon ranks every roster node per tier (`coordinator_order_for_tier`, which existed with no callers), and tiers are filled in turn by the best-ranked node not already leading one — so with eight nodes, every epoch has four different leaders. Ranks are independent and uniform and the rule never looks at who a node is, so every node is equally likely to lead every tier; 16,000 epochs hold every (node, tier) count within 10% of its share. Random each epoch, deliberately not a fixed rotation: a published schedule lets a malicious operator prepare for the round it will run. Leaders are known one anchor block before the epoch starts. Every opted-in node runs a coordinator. The ones not leading are every tier's failover path — the leader, then the tier's own ranking — walked in the same order by every wallet, so a dead leader's cohort moves together. That also retires #883's drain rule: a node never leaves its own roster, so a lead that moves mid-epoch leaves the old leader serving what it holds. The seat scheme is deleted rather than left beside the new one: `elect_coordinators`, `verify_election`, `shard_for`, `shard_key_for_tier_epoch`, `seat_try_order`, `pick_seat_endpoints`, `seats_for_demand`, `COORDINATOR_SEATS`. The published document carries `tiers`, `coordinators` (with `leads`), `my_tiers` and `my_endpoint`; `election_is_honest` recomputes the draw and refuses leaders, failover orders or `leads` that do not follow from the inputs, including the old seat-shaped document. The attestation swaps `my_seat` for `serving` + `my_tiers`; the challenger checks `attested_serving`.
defenwycke
added a commit
that referenced
this pull request
Sep 11, 2026
…own (#888) The in-memory peer table is rebuilt from nothing at startup: every peer re-enters through Peer::new with first_seen = now, and the database — which keeps the real first sighting across restarts (its upsert never overwrites first_seen) — was never read back. So for a day after every restart each peer read as brand new. Found on the v1.11.39 canary. The Wraith coordinator roster requires 24h of maturity (eligibility::EligibilityPolicy), so a restarted node's roster was itself alone: it led all four tiers and reported degraded, while the not-yet-upgraded nodes (no maturity rule) held full rosters. It would have repeated on every restart, unattended upgrades included. On a peer's first health ping per process, its first sighting is restored from the database. Only ever earlier (PeerManager::backdate_first_seen), one read per peer per restart, retried on a database error. ⚠ Correction: #883's description ruled maturity out because /api/v1/network/peers showed ~71-day first sightings. That endpoint reads the DATABASE (db.get_active_peers), not the in-memory table the roster reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #710. The first of two changes that ship in one release; the second spreads coordinator load evenly across every opted-in node.
What was measured
The night all eight mainnet nodes opted in (#708, epoch 6711), the rosters read 6/7/5/5/2/2/4/5 and the fleet elected two different coordinators for one epoch (vm1–3 → vm3, vm4–8 → vm5). Three checks over 25 minutes returned identical numbers. Ten hours later (09-11 08:35 UTC) there were still five distinct rosters. Only vm6 held all eight, and vm8 still elected a different coordinator from everyone else.
Two causes
1. The view was frozen at first look.
refresh_for_heightcomputed the roster once, when the epoch changed, and held it for the rest of the epoch (~a day). The roster was whatever the peer table held at that instant. vm8 restarted at 23:22 during the #708 rollout, while vm1–3 still had coordinating switched off. Ten hours later its roster was missing exactly vm1, vm2 and vm3. This was the dominant cause, and the measurements match it node for node.2. Eligibility read a local verdict.
qualifiedandarchivecame fromQualifiedCapabilityProvider, which reads this node's own verification ledger: the challenges it issued and the verdicts it holds. Challenge rotation samples a few peers per round, so no two nodes hold the same evidence. That input cannot converge, however long you wait. The module doc called it "a qualification verdict the network reached together", and it was not.What changes
Coordinator roster changed, with the commitment), so a split can be traced to the moment it happened.qualifiedandarchiveare removed fromNodeFacts, not defaulted off, so nobody can wire a local verdict back in without it showing up in review. Eligibility is now: opted in, dialable endpoint, 24h maturity, not absent for 7 days. Everything is declared by the node and gossiped.stop()aborts them.should_servekeeps the coordinator running for the rest of that epoch, and new wallets follow the current view.⚠ What is given up
A Sybil cost. Before this change an identity needed a verified archive to coordinate. Now it needs its identity proof-of-work, a day of maturity and an endpoint. That's weaker, and the module says so. It wasn't load-bearing for safety: a coordinator can deny service but can't take coins, because the round is atomic and blind-signed whichever node runs it. Misbehaviour is handled by the outpoint ban list, and identity by the coordinator challenge (#881/#882). A verdict may come back only if every node reads the same one (BFT-finalised or chain-anchored).
Verification
coordinator_election, 19/19 pass. The headline test,a_node_that_drew_early_converges_on_the_late_nodes_election, reproduces the vm8 case: a node that drew from a partial roster ends up with the same commitment, beacon and seating as one that drew from the full roster.scripts/mutate-wraith.shtargeted the removed archive rule and now targets the opt-in refusal: 29/29 caught.cargo fmt --check, the CI clippy invocation andcargo docwith-D warningsare all clean for both crates.Rolling it
roster_commitmentacross all eight (GET /api/v1/pool/coordinatoron :8080). This binary is the first to publish it. Equal everywhere means converged. A node that restarts should match the others within a few health pings, not at the next epoch.Not in this PR
roster_snapshot::lagged_snapshot_heighthas no callers. Left alone.