feat(wraith): every tier gets its own coordinator, drawn evenly from every opted-in node

[defenwycke]

Follows #883 (the roster converges, merged). Same release, one roll.

The problem

At zero session demand, which is what mainnet has, seats_for_demand returned one seat. So one node carried all four denominations (100k / 1m / 10m / 100m) for a whole epoch (~a day) while every other opted-in node sat idle and earned nothing. Measured on the fleet: 8 nodes opted in, elected had one entry.

The seat count was also one more thing nodes had to agree on. It was sized from each node's own snapshot of gossiped session counters, so two nodes with identical rosters could still send the same tier to different coordinators.

What changes

Each tier gets its own leader, and they're different nodes. Every epoch the chain beacon ranks every roster node separately for each tier (coordinator_order_for_tier, which already existed and had no callers). Tiers are filled in turn, each by the best-ranked node not already leading one. With 8 nodes, every epoch has four different leaders.

• Unbiased. Ranks are independent and uniform, and the rule never looks at who a node is. Every node is equally likely to lead every tier. The test runs 16,000 epochs and holds every (node, tier) count within 10% of its fair share.
• Random each day, deliberately not a fixed rotation. An exact rotation would publish the schedule days ahead. A malicious operator could then prepare, for example lining up coins of the right denomination to thin out the round it will run. Here the leaders are known only one block (the anchor) before the epoch starts, and which tier a node gets is part of the draw.
• Variance is the price. With 8 nodes, a node leads on a given day with probability ½ and leads the 100m tier with probability ⅛. Over a year that's about 46 ± 6 100m-tier leads per node.

Every opted-in node runs a coordinator. Nodes that aren't leading are every tier's failover path. The order is the leader, then the rest of that tier's own ranking. Every wallet walks the same order, so a dead leader's cohort moves as one body to the same next node rather than scattering.

The seat scheme is deleted, not left beside the new one. That covers elect_coordinators, verify_election, shard_for, shard_key_for_tier_epoch, seat_try_order, pick_seat_endpoints, seats_for_demand and COORDINATOR_SEATS. The codebase records several times what a dead second scheme invites.

#883's drain rule goes too. It kept a node that lost its seat mid-epoch serving until the epoch turned. Now every opted-in node serves, and a node never leaves its own roster, so when a lead moves the old leader is still serving and finishes what it holds. The rule could no longer trigger.

The published document

GET /api/v1/pool/coordinator now carries tiers (each tier's leader and order), coordinators (every roster node with endpoint and leads), and my_tiers / my_endpoint. seats, my_seat and elected are gone. beacon, anchor_height, roster and roster_commitment are unchanged.

election_doc::election_is_honest recomputes the draw from the published inputs. It refuses a document whose leaders, failover orders or leads don't follow from them. That includes the old seat-shaped document, so an old node's answer makes a new wallet fall back rather than guess. The wallet takes its endpoints from election_doc::endpoints_for_tier, which recomputes them and never reads the claims.

The coordinator attestation (#881) swaps my_seat for serving + my_tiers, and the challenger (#882) checks attested_serving. Neither is deployed.

Verification

• wraith-protocol: tests for distinct leaders (2,000 epochs), evenness per node and per tier (16,000 epochs), no self-nomination, rotation, fewer nodes than tiers, the failover order, and duplicate ids.
• ghost-pool: the document a node publishes passes the exact check a wallet runs (election_is_honest, endpoints_for_tier); four tiers go to four nodes and every roster node serves; each node publishes its own my_tiers / my_endpoint. fix(wraith): the coordinator roster converges — no local verdicts, no first-look freeze (#710) #883's convergence tests are kept.
• Wallet: forged leader, forged leads, trimmed roster, substituted beacon and a seat-shaped document are all refused; four tiers resolve to four different endpoints; nodes with no endpoint are skipped.
• scripts/mutate-wraith.sh gains one mutation, independent picks in place of distinct leaders: 30/30 caught.
• Local: wraith-protocol 426, ghost-pool 501, ghost-verification 324 and the wallet resolver 20/20 pass. cargo fmt --check, the CI clippy invocation (all targets) and cargo doc -D warnings are clean for all four crates.

⛔ Before rolling

Port 9100 must be open on every node. #884 makes the fleet check fail until it is. Every opted-in node now runs a coordinator, and wallets have to be able to reach it. Measured 2026-09-11: ufw is default deny (incoming) everywhere and only vm4 and vm8 have a 9100 rule. Today's elected coordinator (vm3) answers 127.0.0.1:9100/health with 200, but from outside it times out.

No height gate: the election isn't consensus and doesn't feed payouts.