Skip to content

fix(wraith): the coordinator roster converges — no local verdicts, no first-look freeze (#710) - #883

Merged
defenwycke merged 1 commit into
mainfrom
fix/710-roster-converges
Sep 11, 2026
Merged

fix(wraith): the coordinator roster converges — no local verdicts, no first-look freeze (#710)#883
defenwycke merged 1 commit into
mainfrom
fix/710-roster-converges

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

Part of #710. The first of two changes that ship in one release; the second spreads coordinator load evenly across every opted-in node.

What was measured

The night all eight mainnet nodes opted in (#708, epoch 6711), the rosters read 6/7/5/5/2/2/4/5 and the fleet elected two different coordinators for one epoch (vm1–3 → vm3, vm4–8 → vm5). Three checks over 25 minutes returned identical numbers. Ten hours later (09-11 08:35 UTC) there were still five distinct rosters. Only vm6 held all eight, and vm8 still elected a different coordinator from everyone else.

Two causes

1. The view was frozen at first look. refresh_for_height computed the roster once, when the epoch changed, and held it for the rest of the epoch (~a day). The roster was whatever the peer table held at that instant. vm8 restarted at 23:22 during the #708 rollout, while vm1–3 still had coordinating switched off. Ten hours later its roster was missing exactly vm1, vm2 and vm3. This was the dominant cause, and the measurements match it node for node.

2. Eligibility read a local verdict. qualified and archive came from QualifiedCapabilityProvider, which reads this node's own verification ledger: the challenges it issued and the verdicts it holds. Challenge rotation samples a few peers per round, so no two nodes hold the same evidence. That input cannot converge, however long you wait. The module doc called it "a qualification verdict the network reached together", and it was not.

What changes

  • The roster is re-read on every refresh, and the view is rebuilt when the roster or an endpoint changes. The beacon and the seat target stay frozen for the epoch, so session-counter churn can't resize seats mid-epoch. A change is logged once (Coordinator roster changed, with the commitment), so a split can be traced to the moment it happened.
  • qualified and archive are removed from NodeFacts, not defaulted off, so nobody can wire a local verdict back in without it showing up in review. Eligibility is now: opted in, dialable endpoint, 24h maturity, not absent for 7 days. Everything is declared by the node and gossiped.
  • A lost seat is served until the epoch turns. A roster that can move mid-epoch can unseat a node while it holds rounds with committed inputs, and stop() aborts them. should_serve keeps the coordinator running for the rest of that epoch, and new wallets follow the current view.

⚠ What is given up

A Sybil cost. Before this change an identity needed a verified archive to coordinate. Now it needs its identity proof-of-work, a day of maturity and an endpoint. That's weaker, and the module says so. It wasn't load-bearing for safety: a coordinator can deny service but can't take coins, because the round is atomic and blind-signed whichever node runs it. Misbehaviour is handled by the outpoint ban list, and identity by the coordinator challenge (#881/#882). A verdict may come back only if every node reads the same one (BFT-finalised or chain-anchored).

Verification

  • 8 new tests in coordinator_election, 19/19 pass. The headline test, a_node_that_drew_early_converges_on_the_late_nodes_election, reproduces the vm8 case: a node that drew from a partial roster ends up with the same commitment, beacon and seating as one that drew from the full roster.
  • Mutations, run by hand. Restoring the freeze fails 5 tests, including the headline. Removing the drain rule fails both drain tests.
  • scripts/mutate-wraith.sh targeted the removed archive rule and now targets the opt-in refusal: 29/29 caught.
  • cargo fmt --check, the CI clippy invocation and cargo doc with -D warnings are all clean for both crates.

Rolling it

  • No height gate. The election is not consensus and does not feed payouts (the +4 weight move is still unbuilt, L2 removal: capability reweight, the CapabilityType wire hazard, and what it reclaims #736 step 4).
  • During the roll, nodes on the old binary keep their frozen rosters until they're restarted onto this one. That's the same split as today, not a new one.
  • After the roll, compare roster_commitment across all eight (GET /api/v1/pool/coordinator on :8080). This binary is the first to publish it. Equal everywhere means converged. A node that restarts should match the others within a few health pings, not at the next epoch.

Not in this PR

  • Load spread. At zero demand, one seat carries all four denominations for a day. That's the next PR.
  • A brand-new node counts itself as mature from its first second, while peers wait 24h, so for its first day its own roster can differ. It's transient and pre-existing.
  • roster_snapshot::lagged_snapshot_height has no callers. Left alone.

… first-look freeze (#710)

The night all eight mainnet nodes opted in (epoch 6711), rosters read
6/7/5/5/2/2/4/5 and the fleet elected two different coordinators for one
epoch. Ten hours later it was still split. Two causes, both removed here.

The view was computed once per epoch and held. Whatever the peer table looked
like at that instant became the roster for the rest of the day, so a node
restarted mid-epoch drew from a half-filled table and kept it. Measured: one
node restarted while three others still had coordinating switched off, and its
roster was still missing exactly those three ten hours later. The roster is now
re-read on every refresh and the view rebuilt when it changes. The beacon and
the seat target stay frozen for the epoch, so only the roster can move.

Eligibility also read `qualified` and `archive` from this node's own
verification ledger. No two nodes hold the same challenge history, so that
input could never converge, whatever the timing. Both fields are removed from
`NodeFacts`. The module doc said they were "a verdict the network reached
together"; they were not. What is given up is a Sybil cost (a verified
archive), which is stated in the module rather than implied.

A roster that moves mid-epoch can unseat a node while it holds rounds that
participants have committed inputs to, and `stop()` aborts them. So a node
seated at any point in an epoch keeps serving until the epoch turns
(`should_serve`); new wallets follow the current view.

The mutation harness targeted the removed archive rule; it now targets the
opt-in refusal instead. All 29 mutations caught.
@defenwycke
defenwycke merged commit aecd1e9 into main Sep 11, 2026
13 checks passed
defenwycke added a commit that referenced this pull request Sep 11, 2026
…every opted-in node

At zero session demand — mainnet — `seats_for_demand` returned one seat, so one
node carried all four denominations for a whole epoch while every other
opted-in node sat idle and earned nothing. And the seat count was one more
thing nodes had to agree on, sized from each node's own snapshot of gossiped
session counters.

Each tier now has its own leader. The chain beacon ranks every roster node per
tier (`coordinator_order_for_tier`, which existed with no callers), and tiers
are filled in turn by the best-ranked node not already leading one — so with
eight nodes, every epoch has four different leaders. Ranks are independent and
uniform and the rule never looks at who a node is, so every node is equally
likely to lead every tier; 16,000 epochs hold every (node, tier) count within
10% of its share.

Random each epoch, deliberately not a fixed rotation: a published schedule lets
a malicious operator prepare for the round it will run. Leaders are known one
anchor block before the epoch starts.

Every opted-in node runs a coordinator. The ones not leading are every tier's
failover path — the leader, then the tier's own ranking — walked in the same
order by every wallet, so a dead leader's cohort moves together. That also
retires #883's drain rule: a node never leaves its own roster, so a lead that
moves mid-epoch leaves the old leader serving what it holds.

The seat scheme is deleted rather than left beside the new one:
`elect_coordinators`, `verify_election`, `shard_for`, `shard_key_for_tier_epoch`,
`seat_try_order`, `pick_seat_endpoints`, `seats_for_demand`, `COORDINATOR_SEATS`.

The published document carries `tiers`, `coordinators` (with `leads`),
`my_tiers` and `my_endpoint`; `election_is_honest` recomputes the draw and
refuses leaders, failover orders or `leads` that do not follow from the inputs,
including the old seat-shaped document. The attestation swaps `my_seat` for
`serving` + `my_tiers`; the challenger checks `attested_serving`.
defenwycke added a commit that referenced this pull request Sep 11, 2026
…every opted-in node (#885)

At zero session demand — mainnet — `seats_for_demand` returned one seat, so one
node carried all four denominations for a whole epoch while every other
opted-in node sat idle and earned nothing. And the seat count was one more
thing nodes had to agree on, sized from each node's own snapshot of gossiped
session counters.

Each tier now has its own leader. The chain beacon ranks every roster node per
tier (`coordinator_order_for_tier`, which existed with no callers), and tiers
are filled in turn by the best-ranked node not already leading one — so with
eight nodes, every epoch has four different leaders. Ranks are independent and
uniform and the rule never looks at who a node is, so every node is equally
likely to lead every tier; 16,000 epochs hold every (node, tier) count within
10% of its share.

Random each epoch, deliberately not a fixed rotation: a published schedule lets
a malicious operator prepare for the round it will run. Leaders are known one
anchor block before the epoch starts.

Every opted-in node runs a coordinator. The ones not leading are every tier's
failover path — the leader, then the tier's own ranking — walked in the same
order by every wallet, so a dead leader's cohort moves together. That also
retires #883's drain rule: a node never leaves its own roster, so a lead that
moves mid-epoch leaves the old leader serving what it holds.

The seat scheme is deleted rather than left beside the new one:
`elect_coordinators`, `verify_election`, `shard_for`, `shard_key_for_tier_epoch`,
`seat_try_order`, `pick_seat_endpoints`, `seats_for_demand`, `COORDINATOR_SEATS`.

The published document carries `tiers`, `coordinators` (with `leads`),
`my_tiers` and `my_endpoint`; `election_is_honest` recomputes the draw and
refuses leaders, failover orders or `leads` that do not follow from the inputs,
including the old seat-shaped document. The attestation swaps `my_seat` for
`serving` + `my_tiers`; the challenger checks `attested_serving`.
defenwycke added a commit that referenced this pull request Sep 11, 2026
…own (#888)

The in-memory peer table is rebuilt from nothing at startup: every peer
re-enters through Peer::new with first_seen = now, and the database — which
keeps the real first sighting across restarts (its upsert never overwrites
first_seen) — was never read back. So for a day after every restart each peer
read as brand new.

Found on the v1.11.39 canary. The Wraith coordinator roster requires 24h of
maturity (eligibility::EligibilityPolicy), so a restarted node's roster was
itself alone: it led all four tiers and reported degraded, while the
not-yet-upgraded nodes (no maturity rule) held full rosters. It would have
repeated on every restart, unattended upgrades included.

On a peer's first health ping per process, its first sighting is restored from
the database. Only ever earlier (PeerManager::backdate_first_seen), one read per
peer per restart, retried on a database error.

⚠ Correction: #883's description ruled maturity out because
/api/v1/network/peers showed ~71-day first sightings. That endpoint reads the
DATABASE (db.get_active_peers), not the in-memory table the roster reads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant