prover: route prove-chunk and agg-chunks through the #119 retry (#237) - #240
prover: route prove-chunk and agg-chunks through the #119 retry (#237)#240defenwycke wants to merge 1 commit into
Conversation
`prove_session` proves every segment behind a single call, so a transient #119 fault surfaced as one `Err` after the whole chunk's work and took the chunk with it. The only retry for that path lived in `scripts/gpu-benchmark.sh` -- outside the binary -- so anyone running `prove-chunk` by hand simply lost the chunk. Measured on 8x L40S (docs/history/BENCH_8xL40S_2026-09-08.md): a fault two-thirds through a chunk cost **33% of the block's critical path** -- 817 s wall against 614 s of useful work -- because recovery could only restart from segment zero. Observed rate was 5 faults in 80 chunk attempts (6.3%), a ~40% chance of at least one per 8-chunk block. Adds `prove_session_resilient`, which drives the segment loop itself so every segment goes through the existing `prove_segment_resilient` bounded retry, then assembles. A #119 fault now costs ONE segment instead of the chunk. This is option (2) from the issue. Assembly is not reimplemented: `assemble_from_segment_receipts` is the same code `prove_session` runs after its own loop, so the two paths cannot drift. `seg_distribute_cmd` already proves this shape yields a receipt that verifies against METHOD_ID. Applied to BOTH unprotected call sites. `agg-chunks` had the identical gap, and a fault there loses the most expensive single unit of work in the pipeline. `FoldProgress` is removed -- `prove_session_resilient` prints the progress itself, at the same cadence and in the same line format, since recorded benchmark logs are parsed for it. ⚠ This does not fix #119. It changes an intermittent prover fault from "lose the chunk" to "lose a few seconds". Retrying on "proof is invalid" is safe only because the guest and inputs are identical across attempts, so a genuinely invalid proof still fails every attempt and the bounded loop terminates -- that reasoning is written down at the helper. Refs #119.
Equivalence result — and the gate I promised in the description could not be run as writtenI said I would compare the receipts byte for byte and that the PR should not merge if they RISC0 seals are not reproducible. let mut rng = rand::rng();
let rand_z = ExtVal::random(&mut rng);It feeds a Schwartz–Zippel checksum in witgen, so the trace — and therefore the seal — differs on So the run below carries its own control: What was run
METHOD_ID gate first. Both binaries report canonical The promised gate, for the recordThree different digests, identical length. The gate that bindsEvery receipt run through both binaries'
All six verify against the canonical image id, with an identical journal and an identical segment Scope — what this does NOT cover
Nothing here blocks the merge on equivalence grounds. |
1 participant
Closes #237.
Implements option (2) from the issue: prove the segments explicitly so the existing per-segment retry applies, rather than wrapping the whole chunk.
The gap
prove_sessionproves every segment behind a single call, so a transient #119 fault surfaced as oneErrafter the whole chunk's work and took the chunk with it.prove_segment_resilientalready existed with five callers —prove-chunkwas not one of them, because it used a different API. The only retry covering this path lived inscripts/gpu-benchmark.sh, outside the binary, so anyone invokingprove-chunkby hand lost the chunk.Why it is worth doing properly
From last night's 8× L40S runs (#238):
Option (1) — a chunk-level retry — would have left that 33% in place. Under option (2) a fault costs one segment.
The change
Adds
prove_session_resilient, which drives the segment loop and routes each segment throughprove_segment_resilient, then assembles.Assembly is not reimplemented.
assemble_from_segment_receiptsis the same codeprove_sessionruns after its own loop, so the two paths cannot drift — andseg_distribute_cmdalready demonstrates this shape yields a receipt that verifies againstMETHOD_ID.Applied to both unprotected call sites.
agg-chunkshad the identicalprove_session().unwrap(), and a fault there loses the most expensive single unit of work in the pipeline.FoldProgressis removed — the helper prints progress itself, at the same cadence and the same line format, because recorded benchmark logs are parsed for it.Verification
⚠ This does not fix #119. It changes an intermittent prover fault from "lose the chunk" to "lose a few seconds".
⚠ Retrying on "proof is invalid" is safe only because the guest and inputs are identical across attempts — a genuinely invalid proof fails every attempt and the bounded loop still terminates. That reasoning is written down at the helper, per the issue's request.
Refs #119.