agg-chunks: honour HAZYNC_RECEIPTS, and correct §7 bind advice - #230
Merged
Conversation
…d advice
Two unrelated fixes that share a cause: a written instruction that nothing
implemented.
## hazync#229 — `agg-chunks` never read `HAZYNC_RECEIPTS`
`scripts/gpu-benchmark.sh:144` has always exported `HAZYNC_RECEIPTS` for phase
3, and `grep -rn HAZYNC_RECEIPTS prover/host/src/` returned nothing. So
`read_chunk_receipts` resolved `chunk_0.bin` against its own CWD — which phase
3 sets to `prover/` — and the aggregate died on a run whose sixteen proves had
every one succeeded:
thread 'main' panicked at host/src/main.rs:2609:72:
chunk receipt chunk_0.bin: No such file or directory (os error 2)
`prove-chunk` writes each receipt to `$HAZYNC_OUT`, a full path, so the two
halves of a run could always disagree about where the receipts were. This adds
the missing half. It defaults to `.`, so behaviour is unchanged when nothing
sets it, and the script becomes correct as already written rather than needing
a `cd` around it.
Verified with a control, since "it compiles" is not evidence:
HAZYNC_RECEIPTS unset -> chunk receipt ./chunk_0.bin: No such file or directory
HAZYNC_RECEIPTS=<dir> -> chunk receipt <dir>/chunk_0.bin does not verify: claim digest …
The second run reaches the id check, which is the point: the path resolved. It
fails there only because that host was built on a laptop and carries a
non-canonical `METHOD_ID`, which is the guard doing its job on a real
cross-guest receipt.
⚠ I filed #229 claiming a wrong-directory aggregate could "fold cross-guest
receipts and return a plausible number". That was wrong, and the correction is
on the issue: `read_chunk_receipts` has verified every receipt against
`METHOD_ID` since hazync#153, unconditionally. The stale set on the bench box
would panic, not pass. The defect was only ever the path.
## hazync#218 — §7 told you to do something impossible
§7 said to set `COORD_BIND` to the coordinator's **private-network IP**. There
is no private network: the coordinator has `lo`, one PUBLIC `eth0` and
`docker0`, and the web box proxies to it over the public internet.
`COORD_BIND=127.0.0.1` would take the board dark. `0.0.0.0` is correct here and
the access control is the firewall, not the bind address.
Also records what is actually there, measured rather than assumed — and it is
not what #218's reconnaissance found. `ufw` is **active** with default-deny, not
inactive; `:8899` is already restricted to the web box. But the `INPUT` chain
carries hand-added rules ahead of ufw's, so one port has **two sources of
truth** and `ufw status` alone will mislead you.
⚠ Left for an operator, not changed here: one hand-added ACCEPT is for
`94.237.17.228`, which moved **0 packets in 90 s** against the web box's 1,690.
It is a stale allow for a recycled provider address. Editing live firewall
state is not something to bundle into a docs commit.
Verified: `cargo check --release` REAL_EXIT=0; all eight repo check scripts
REAL_EXIT=0.
Claude-Session: https://claude.ai/code/session_01BGBba1FtGQjp2focJGWtjU
This was referenced Sep 8, 2026
defenwycke
added a commit
that referenced
this pull request
Sep 8, 2026
Completes the 8-vs-16 comparison across all three blocks. | block | 8 chunks | 16 chunks | Δ | |---------|----------|-----------|-------| | 966,108 | 912.8 s | 905.5 s | -0.8% | | 966,107 | 807.8 s | 819.3 s | +1.4% | | 966,106 | 711.8 s | 711.9 s | +0.0% | ⏰ Chunk count does essentially nothing -- mean +0.2%, and the sign is not consistent. The mechanism shows in the ordering: the 16-chunk gain tracks the 8-chunk straggler monotonically (1.118 -> -0.8%, 1.059 -> 0.0%, 1.048 -> +1.4%). Splitting finer buys back imbalance and nothing else, and roughly pays for its own overhead when it does. ✅ All three blocks gave byte-identical digests at 8 and 16 chunks -- six proofs, three digests. The receipt is a function of the block, not of how the work was divided. ⇒ Keep HAZYNC_CHUNKS equal to the card count. Effort is better spent on #119 (33% of a block when it fires, 6.3% per chunk) than on chunk count. Also notes that HAZYNC_RECEIPTS (#230) is not in the v0.21.0 binary.
defenwycke
added a commit
that referenced
this pull request
Sep 8, 2026
* docs: benchmark of 8x L40S over three consecutive tip blocks First fleet measurement of near-tip blocks on the v0.21.0 CORE guest. Blocks 966,108 / 966,107 / 966,106, `HAZYNC_CHUNKS=8`, one chunk per card, aggregated over `seg-serve`. Measured, not projected: - compute total 912.8 s / 807.8 s / 711.8 s - straggler 1.118 / 1.048 / 1.059 -- block composition moves the chunk phase more than any knob we have tuned - the aggregate barely moves (203.8 / 205.8 / 189.8 s) across a 22% swing in chunk work; it tracks segment count, not layout - prevout count predicts chunk time, transaction count does not -- the three blocks order in reverse by transactions - both 966,108 runs produced an identical receipt digest despite different chunk timings and one restarted chunk - 966,106 hit #119 and the armed retry recovered it on `attempt=2`, the first live confirmation the wrapper works Card-count projection is anchored on the measured 8-card figure and puts sub-ten-minute blocks at ~13 L40S, not 10. * docs: measure 16 chunks on 8 cards -- it is a wash Answers the open question the doc previously left as "not measured": is `HAZYNC_CHUNKS=16` (two chunks per card) better than one chunk per card? Measured on 966,108, same fleet, same binary, same guest: - straggler falls 1.118 -> 1.085, exactly as summing two draws predicts - but total chunk work rises 1.3% (fixed per-chunk cost paid 16 times) - and the aggregate rises 2.3% (depth-4 join tree, 4.3 s of resolves) - net **905.5 s vs 912.8 s -- 0.8%, inside noise** All sixteen chunks `REAL_EXIT=0, attempt=1`. Chunk inputs sum to 8,562, matching the block's prevout count exactly. The 8- and 16-chunk runs produced the **identical receipt digest**, so the proof does not depend on how the work is divided -- determinism across a structural change, not just a timing one. Conclusion: match chunks to cards. Chunk count is nearly free to get wrong, which is a better reason for the default than load balance. * docs: 966,107 at 16 chunks, and the measured cost of #119 Second block re-run at 16 chunks. Two chunks hit #119 and the armed retry recovered both on `attempt=2`. - 16 chunks is *slightly worse* here: 819.3 s vs 807.8 s (+1.4%). The 8-chunk straggler was already 1.048, so there was no imbalance to win back and only the +1.2% per-chunk overhead showed up. The 16-chunk gain is bounded by how bad the 8-chunk straggler was. - digest matches the 8-chunk run of the same block, as with 966,108 - inputs sum to 7,961, matching the block's prevout count exactly ⛔ #119 cost 33% of the block -- 817 s wall against 614 s of successful work on the same card, because `prove-chunk` restarts from segment zero. Observed rate across the night is 4 faults in 64 attempts (~6%), which is a ~40% chance of at least one fault per 8-chunk block. The retry is not optional; it belongs in the binary (#237). * docs: 966,106 at 16 chunks, and the verdict on chunk count Completes the 8-vs-16 comparison across all three blocks. | block | 8 chunks | 16 chunks | Δ | |---------|----------|-----------|-------| | 966,108 | 912.8 s | 905.5 s | -0.8% | | 966,107 | 807.8 s | 819.3 s | +1.4% | | 966,106 | 711.8 s | 711.9 s | +0.0% | ⏰ Chunk count does essentially nothing -- mean +0.2%, and the sign is not consistent. The mechanism shows in the ordering: the 16-chunk gain tracks the 8-chunk straggler monotonically (1.118 -> -0.8%, 1.059 -> 0.0%, 1.048 -> +1.4%). Splitting finer buys back imbalance and nothing else, and roughly pays for its own overhead when it does. ✅ All three blocks gave byte-identical digests at 8 and 16 chunks -- six proofs, three digests. The receipt is a function of the block, not of how the work was divided. ⇒ Keep HAZYNC_CHUNKS equal to the card count. Effort is better spent on #119 (33% of a block when it fires, 6.3% per chunk) than on chunk count. Also notes that HAZYNC_RECEIPTS (#230) is not in the v0.21.0 binary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #229. Also does the docs half of #218 (the firewall change itself is left for an operator).
agg-chunksnever readHAZYNC_RECEIPTSscripts/gpu-benchmark.sh:144has exported it for phase 3 all along, andgrep -rn HAZYNC_RECEIPTS prover/host/src/returned nothing.read_chunk_receiptsresolvedchunk_0.binagainst its own CWD — which phase 3 sets toprover/— so the aggregate died on a run whose sixteen proves had every one succeeded.prove-chunkwrites to$HAZYNC_OUT, a full path, so the two halves of a run could always disagree about where the receipts were. This adds the missing half. Default is., so nothing changes when nothing sets it, and the script is correct as already written rather than needing acdwrapped around it.Verified with a control, because "it compiles" is not evidence:
The second reaches the id check — that is the point, the path resolved. It fails there only because that host was laptop-built and carries a non-canonical
METHOD_ID, i.e. the guard working on a genuinely cross-guest receipt.⚠ I was wrong in the issue, and the correction matters
I filed #229 saying a wrong-directory aggregate could "fold cross-guest receipts and return a plausible number". That is not true.
read_chunk_receiptshas verified every receipt againstMETHOD_IDsince #153, unconditionally — the stale set on the bench box would panic, not pass. The "fix the class" work I proposed was already done before I filed. The defect was only ever the path, and it fails loudly either way.§7 told you to do something impossible (#218)
§7 said to set
COORD_BINDto the coordinator's private-network IP. There is no private network:lo, one PUBLICeth0,docker0, and the web box proxies over the public internet.COORD_BIND=127.0.0.1takes the board dark;0.0.0.0is correct and the control is the firewall, not the bind.The RUNBOOK now also records what is measurably there — and it is not what #218's recon found.
ufwis active with default-deny, not inactive, and:8899is already restricted to the web box. But hand-added rules sit ahead of ufw's inINPUT, so one port has two sources of truth andufw statusalone will mislead you.Not changed here, deliberately
One hand-added ACCEPT is for
94.237.17.228, which moved 0 packets in 90 s while the web box moved 1,690 — a stale allow for a recycled provider address. Editing live firewall state does not belong in a docs commit, and #218 stays open for it.Verified:
cargo check --releaseREAL_EXIT=0, all eight repo check scripts REAL_EXIT=0.