Skip to content

Stop blaming the guest id when two spine workers race - #231

Merged
defenwycke merged 2 commits into
mainfrom
fix/220-spine-race-message
Sep 8, 2026
Merged

Stop blaming the guest id when two spine workers race#231
defenwycke merged 2 commits into
mainfrom
fix/220-spine-race-message

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

Closes #220.

The two cases looked identical, and the message picked the wrong one

From the v0.20.0 cutover:

extend-spine failed for block 2:
  Build a host that matches the proof's guest (reproduce/Dockerfile), then retry.
Underlying verifier error: claim digest does not match the expected digest c08d5e0f…

There was no guest mismatch. Both workers ran the same binary, both had asserted binary and board agree at startup, and the next attempt succeeded — which a real mismatch cannot do, being deterministic.

The cause is the shape of the loop: fetch the head → wait on the GPU lock → fold. Another worker advances the spine inside that window, so the fold is computed against a head that no longer exists and the claim digest cannot match.

The fix goes in the caller, because only it knows

The host sees a claim-digest mismatch and nothing else; it cannot know which head the fold was computed against. The CLI can. On failure it now re-reads /api/spine:

  • head movedthe spine advanced under us ([1..N] -> [1..M]) — refolding from the new head, at info level, and the loop continues. Not an error — the other worker did the work we were about to do.
  • head unchanged → the existing build-mismatch advice, which is now actually the likely cause.

Only the message and the recovery change. A seam failure with an unmoved head still exits loudly, because that one really is information.

The operator half, stated where the trap is set

MODE=spine already says "only ever needs ONE". That is fleet-wide, not per box — and MODE=mixed allocates one spine worker per box, so mixed on N boxes gives N spine workers racing one spine. Nothing said so. run-workers.sh's header now does, and says to run mixed on one box and prove on the rest.

Verified: python3 -m ast parses, hazync --help runs, bash -n clean, all eight repo check scripts REAL_EXIT=0.

⚠ Independent of #230 — branched off main, not stacked on it, so both get their own CI.

A lost race and a guest mismatch produced the same message, and the message
described the wrong one. From the v0.20.0 cutover:

    extend-spine failed for block 2:
      Build a host that matches the proof's guest (reproduce/Dockerfile), then retry.
    Underlying verifier error: claim digest does not match the expected digest c08d5e0f…

There was no guest mismatch. Both workers ran the same binary, both had
asserted `binary and board agree` at startup, and the very next attempt
succeeded — which a genuine mismatch cannot do, because it is deterministic.

The real cause is the shape of the loop: fetch the head, wait on the GPU lock,
then fold. Another worker advances the spine inside that window, so the fold is
computed against a head that no longer exists and its claim digest cannot
match.

The caller is the only place that can tell the two apart, because only it knows
which head the fold was computed against. On failure it now re-reads
`/api/spine` and compares:

  * head moved   -> "the spine advanced under us … refolding from the new head",
                    at info level, and the loop continues. Not an error: the
                    other worker did the work we were about to do.
  * head unchanged -> the existing advice, which is now actually the likely cause.

Also states the operator half in `run-workers.sh`, where the trap is set.
`MODE=spine` says "only ever needs ONE" — that is **fleet-wide, not per box**,
and `MODE=mixed` allocates one spine worker per box, so `mixed` on N boxes
gives N spine workers racing one spine. The header now says so, and says to run
`mixed` on one box and `prove` on the rest.

⚠ Only the message and the recovery change. A seam failure with an unmoved head
still exits loudly, because that one really is information.

Verified: `python3 -m ast` parses, `hazync --help` runs, `bash -n` on
run-workers.sh, and all eight repo check scripts REAL_EXIT=0.

Claude-Session: https://claude.ai/code/session_01BGBba1FtGQjp2focJGWtjU
@defenwycke
defenwycke merged commit d502dfc into main Sep 8, 2026
6 checks passed
@defenwycke
defenwycke deleted the fix/220-spine-race-message branch September 8, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extend-spine blames the guest id when two spine workers race

1 participant