Two workers advancing the spine concurrently produce an error that points at the wrong cause.
What happened
During the v0.20.0 cutover, MODE=mixed was started on two GPU boxes. Mixed allocates "1 advancing the spine" each, so there were two spine workers against one spine — where run-workers.sh says plainly: MODE=spine … # just the spine (only ever needs ONE).
Observed on both boxes, exactly once each:
absorbing block 2 into [1..1] — one fold, no re-proving…
(waited 8s for the GPU — another worker was absorbing)
extend-spine failed for block 2:
…f1d7f7d86c24bacc4ce968c621faed4b15
Build a host that matches the proof's guest (reproduce/Dockerfile), then retry.
See PROVING.md -> "the guest image id (METHOD_ID) & reproducibility".
Underlying verifier error: claim digest does not match the expected digest c08d5e0f…
absorbing block 2 into [1..1] — one fold, no re-proving…
✓ spine now [1..2] — genesis-anchored
Why the message is wrong
The advice — "Build a host that matches the proof's guest" — describes a guest mismatch. There was none: both workers ran the same binary, both had already asserted binary and board agree on 3867611d at startup, and the very next attempt succeeded. A genuine guest mismatch is deterministic and cannot self-heal on retry.
The real cause is a lost race: the other worker advanced the spine first, so this fold was computed against a head that had already moved and its claim digest no longer matched the expected one.
Both boxes recovered on the next attempt, so this costs GPU time rather than correctness.
Suggested
The two cases are distinguishable and should not share a message:
- claim digest mismatch + the spine head moved since this fold began -> "the spine advanced under you; refolding from the new head", at info level, not an error
- claim digest mismatch + the head is unchanged -> the existing guest-mismatch advice, which is then actually the likely cause
The "(waited Ns for the GPU — another worker was absorbing)" line immediately above already knows contention occurred, so the signal is present at the point the message is chosen.
⚠ Operator note, separate from the bug: MODE=mixed on N boxes gives N spine workers. Only one may exist fleet-wide. Worth stating in run-workers.sh's header alongside the existing MODE docs, since the mixed description does not say it.
Two workers advancing the spine concurrently produce an error that points at the wrong cause.
What happened
During the v0.20.0 cutover,
MODE=mixedwas started on two GPU boxes. Mixed allocates "1 advancing the spine" each, so there were two spine workers against one spine — whererun-workers.shsays plainly:MODE=spine … # just the spine (only ever needs ONE).Observed on both boxes, exactly once each:
Why the message is wrong
The advice — "Build a host that matches the proof's guest" — describes a guest mismatch. There was none: both workers ran the same binary, both had already asserted
binary and board agree on 3867611dat startup, and the very next attempt succeeded. A genuine guest mismatch is deterministic and cannot self-heal on retry.The real cause is a lost race: the other worker advanced the spine first, so this fold was computed against a head that had already moved and its claim digest no longer matched the expected one.
Both boxes recovered on the next attempt, so this costs GPU time rather than correctness.
Suggested
The two cases are distinguishable and should not share a message:
The "(waited Ns for the GPU — another worker was absorbing)" line immediately above already knows contention occurred, so the signal is present at the point the message is chosen.
⚠ Operator note, separate from the bug:
MODE=mixedon N boxes gives N spine workers. Only one may exist fleet-wide. Worth stating inrun-workers.sh's header alongside the existing MODE docs, since the mixed description does not say it.