prove-chunk has no #119 retry, and it is the path people run by hand

[defenwycke]

prove-chunk is the one prove path with no #119 retry, and it is the path a person runs by hand.

What happened

Proving block 966,108 across 8 L40S, one chunk per card, using the released v0.21.0 binary. Chunk 2 died at segment ~65 of 271, about nine minutes in:

thread 'main' panicked at host/src/main.rs:2485:56:
called `Result::unwrap()` on an `Err` value: verify segment
Caused by:
    verification indicates proof is invalid

That is #119. The other seven chunks completed normally (13m29s wall). The card that hit it produced nothing at all and had to start from zero, which put ~10 minutes on the critical path of an 8-card run.

The gap

prove_segment_resilient() already exists and does exactly the right thing — bounded attempts, and it discriminates the transient fault from the ones that will fail identically every time:

let transient = msg.contains("verification indicates proof is invalid")
    || msg.contains("verify segment");
if !transient { panic!("{what}: not a #119 fault, not retrying: {msg}"); }

It has five callers. prove-chunk is not one of them. It uses a different API:

let receipt = server.prove_session(&ctx, &session).unwrap().receipt;
receipt.verify(METHOD_ID).unwrap();

prove_session proves every segment internally, so the per-segment helper is not reachable through it, and the bare unwrap() is the whole of the error handling.

The retry that does exist for this path lives in scripts/gpu-benchmark.sh — outside the binary, so only that script benefits. Anyone invoking prove-chunk directly gets a hard failure and loses the chunk.

Options

1. Wrap the prove_session call in a bounded retry inside the binary. Simplest, and matches what gpu-benchmark.sh does today. Cost: a Prover emits a succinct receipt that fails its own verify() for rare rand_z values — both backends, in preflight #119 fault re-proves the entire chunk — here that is ~9 minutes of good work discarded to re-do 65 segments.
2. Prove segments explicitly so prove_segment_resilient applies. More work, but a fault then costs one segment (~2 s) instead of a whole chunk. Five other paths already do this, so the shape is established.
3. Leave it, and document loudly that prove-chunk must be wrapped by the caller.

(2) is the right answer if the aim is that a contributor can prove a block by hand without knowing about #119. (1) is a small change that removes the sharp edge today.

⚠ This does not fix #119 and is not a substitute for it. It changes an intermittent prover fault from "lose the chunk" to "lose a few seconds".

⚠ Retrying on "proof is invalid" is safe here specifically because the guest and inputs are identical across attempts: a genuinely invalid proof fails every attempt and the bounded loop still terminates. That reasoning should stay written down wherever the retry lands.

Refs #119.

[defenwycke]

Measured the fault rate over last night's 8× L40S runs — this is not a rare event.

5 faults in 80 chunk attempts = 6.3% per chunk.

run	chunks	#119 faults
966,108 run 1 (8 chunks)	8	1
966,108 run 2 (8 chunks)	8	0
966,107 (8 chunks)	8	0
966,106 (8 chunks)	8	1
966,108 (16 chunks)	16	0
966,107 (16 chunks)	16	2
966,106 (16 chunks)	16	1

At 6.3% per chunk that is a ~40% chance of at least one fault per 8-chunk block, and ~64% at 16 chunks. Every fault in the four runs that had the shell retry armed recovered on attempt=2, so the fault is transient — but the cost is not.

What it costs

966,107 at 16 chunks, the two cards that faulted:

card	successful work	wall-clock
.150.120	614 s	817 s
.148.197	614 s	782 s

33% of the block's critical path, because prove-chunk restarts the faulted chunk from segment zero. A fault two-thirds of the way through a chunk throws away two-thirds of a chunk.

Run 1 of 966,108 is the worst case on a 1:1 mapping: seven chunks finished in 13m29s, the faulted one took 21m05s after restarting, and the block came in at 26m00s instead of 16m42s.

Two asks, in order of value

1. Retry inside the binary (this issue). At a 40%-per-block rate, an unwrapped prove-chunk is not usable unattended.
2. Resume rather than restart. Retry alone still pays for the discarded segments — the 33% above was with a working retry. If the already-proved segments can be kept and only the faulted one redone, the cost drops from a third of a block to roughly nothing.

Full evidence and per-chunk timings in #238.