Skip to content

gpu-benchmark.sh phase 3 cannot find its own receipts, and the fallback path holds cross-guest ones #229

Description

@defenwycke

Found while measuring block 962,000 on an L40S for v0.21.0. Phases 1 and 2 are fine — the failure
is phase 3, and the reason it is worth an issue is the second half rather than the first.

The bug

Phase 2 writes each receipt into the run directory, and does so correctly — all 16 land:

prove_0.log:  chunk 0 (435 inputs) proved in 242s -> /root/gpu-bench-core-.../receipts/chunk_0.bin
$ ls gpu-bench-core-20260907-2316/receipts | wc -l
16

Phase 3 (scripts/gpu-benchmark.sh:144) then does:

( cd "$P" && export HAZYNC_CHUNKS=16 HAZYNC_RECEIPTS="$OUT/receipts"; "$BIN" agg-chunks )

and dies immediately:

thread `main` panicked at host/src/main.rs:2609:72:
chunk receipt chunk_0.bin: No such file or directory (os error 2)
agg REAL_EXIT=101

agg-chunks does not honour HAZYNC_RECEIPTS. It resolves chunk_0.bin relative to the working
directory, which phase 3 has just set to $P (prover/). HAZYNC_RECEIPTS is referenced exactly
once in the whole script — on that line — so nothing else was ever going to make it take effect.

Workaround, which is how I got the number: cd "$OUT/receipts" and run agg-chunks there.
Result: BLOCK 962000 AGGREGATED in 1238.9s — succinct receipt VERIFIED.

⛔ The part that actually matters

Because the path is resolved from the working directory, an aggregate run from the wrong place does
not necessarily fail — it can silently pick up whatever chunk_*.bin happens to be there.

On the box I ran this on, /root/bench-default/receipts/ contains a full set of 16 chunk receipts
dated 2026-09-05, i.e. from a different guest, two re-baselines of context ago:

-rw-r--r-- 1 root root  511490 09-05 21:13 chunk_0.bin
...
count: 16

A complete set, plausible sizes, correct names. An aggregate that resolved to that directory would
fold cross-guest receipts and return a number that looks entirely reasonable. That is the failure
mode this repo keeps writing down: a harness result that is indistinguishable from a real one.

Suggested fix

  1. Make agg-chunks honour HAZYNC_RECEIPTS (or take the directory as an argument) so the script
    and the host agree on one contract — they currently disagree silently.
  2. Have agg-chunks refuse to aggregate receipts whose guest id is not the current one, rather
    than trusting the filename. The receipts carry their guest; the check is cheap and it is the only
    thing that makes a wrong-directory run loud instead of plausible.
  3. Failing both, phase 3 should assert the receipt count and their mtimes against the proves it just
    ran, and stop if they do not correspond.

Item 2 is the one worth doing. 1 and 3 fix this script; 2 fixes the class.

Evidence: ~/hazync-v0.21.0-release-evidence/gpu-benchmark-962000-{l40s,aggregate}.log.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions