Skip to content

feat(wraith): a full coordinator spills its tier to the next node in the order - #886

Merged
defenwycke merged 1 commit into
mainfrom
feat/wraith-spill-when-full
Sep 11, 2026
Merged

feat(wraith): a full coordinator spills its tier to the next node in the order#886
defenwycke merged 1 commit into
mainfrom
feat/wraith-spill-when-full

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

Follows #885 (every tier gets its own coordinator). Same release.

What this does

Rising demand now reaches more coordinators. A tier's leader takes every round until it's running MAX_LIVE_ROUNDS_PER_TIER (4) at once. The next wallet is told coordinator_full, walks to the next node in the tier's order, and joins there. Every wallet on the tier walks the same order, so the overflow gathers into one real round on the next node rather than scattering. A busier tier spills further down the order, and more opted-in nodes get called and earn fees.

The demand-sized seat count it replaces never did this for a single tier. Each tier was routed to exactly one seat (shard(tier, epoch) mod seats), so seats beyond the four tiers could never get work. And the count came from each node's own snapshot of gossiped session counters, so nodes could disagree about it. Spilling needs no shared number: the only input is the answer the full coordinator gives.

The pieces

  • Protocol. MAX_LIVE_ROUNDS_PER_TIER and COORDINATOR_FULL. The registry counts live rounds inside the same lock as the insert, so two concurrent callers can't both see room for one more. A filling round with room is still joined at the limit; the limit only stops a new round. Live means filling (window open), locked or signing; a round that's broadcast is only waiting for confirmations and doesn't hold a place.
  • Coordinator. At the limit, find_or_create answers 503 + coordinator_full and creates nothing.
  • Wallet. The join step walks the coordinator and then its alternates, moving on only on coordinator_full or unreachable. Any other refusal is final. "Full" is read from the error code, not the status, so a proxy's bare 503 doesn't count.
    • The round then stays on the coordinator that accepted. PreparedMix.coordinator_url records it, and submit goes there. Alternates are kept separate from peers: a peer is a standby that mirrors this coordinator's sessions and can take over mid-round, while an alternate is an independent coordinator that has never heard of the session.
  • Daemon. Both mix entry points take alternates from the verified election: the rest of the tier's order after the chosen coordinator. A coordinator the user typed in by hand gets none, since spilling a user's explicit choice onto nodes they didn't pick would be the wallet overriding them.

⚠ Stated rather than discovered

  • The limit is a parameter, not a measurement. Nobody has measured what one coordinator can carry.
  • Fixed failover order was published but never walked. feat(wraith): deterministic seat failover that keeps the anonymity set together (#711) #873 returned fallbacks and only the CLI printed them; a mix took a single URL. The daemon now uses the order.
  • A leader can lie about being full. It can push wallets onward, including specific ones it chooses to refuse. It can't pick where they go, because the order comes from the chain, so steering a target to a partner needs the next node in that order to be colluding. It could already refuse service outright.

Verification

  • End to end over real HTTP (wraith_e2e): a wallet sent to a full leader enrols with the next node; and five wallets spilled off a full leader complete a whole round on the next node (join, sign, submit, broadcast), with the leader broadcasting nothing and creating nothing.
  • Mutations, run by hand. Sending the submit back to the leader fails the whole-round test. Treating "full" as final fails both spill tests.
  • Registry: refusal at the limit, an open round still joined at the limit, broadcast and expired rounds holding no place, the limit per tier, a configurable limit. Coordinator router: 503 + coordinator_full, nothing created. Wallet: a proxy's bare 503 isn't full; the code under another status isn't full; a pinned client keeps standbys only for its own coordinator. Daemon: alternates follow the chosen node; a hand-typed coordinator gets none.
  • scripts/mutate-wraith.sh gains "a full coordinator keeps opening rounds": 31/31 caught.
  • Local: 779 tests across 14 suites pass (wraith-protocol, wraith-coordinator, wraith-wallet-core, wraith-wallet-daemon). The CI clippy invocation (all targets, including ghost-pool) and cargo doc -D warnings are clean.

…the order

Rising demand now reaches more coordinators. A tier's leader takes every round
until it runs MAX_LIVE_ROUNDS_PER_TIER at once; the next wallet is answered 503
`coordinator_full`, walks to the next node in the tier's order and joins there.
Every wallet walks the same order, so the overflow gathers into one round on the
next node rather than scattering; a busier tier spills further down.

The demand-sized seat count this replaces never did that for a single tier —
each tier routed to exactly one seat — and it was sized from a figure nodes
could disagree about. Spilling needs no shared number: the only input is the
answer the full coordinator gives.

- Registry: live rounds (filling with an open window, locked, signing) are
  counted inside the same lock as the insert; an open round with room is still
  joined at the limit; a broadcast round holds no place.
- Coordinator: at the limit, find_or_create answers 503 + coordinator_full and
  creates nothing.
- Wallet: enrol walks the coordinator then its alternates, moving on only on
  coordinator_full (read from the error code, not the status) or unreachable;
  the round is then pinned to the acceptor (PreparedMix.coordinator_url) and
  submission goes there. Alternates are not standbys: they share no state.
- Daemon: alternates come from the verified election — the rest of the tier's
  order after the chosen coordinator; a hand-typed coordinator gets none.

#873 published the failover order and nothing walked it (the CLI printed it);
the daemon now uses it. The limit (4) is a parameter, not a measurement.
@defenwycke
defenwycke merged commit cc4361f into main Sep 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant