docs: 8x L40S benchmark -- three tip blocks, 8 vs 16 chunks - #238
Merged
Conversation
First fleet measurement of near-tip blocks on the v0.21.0 CORE guest. Blocks 966,108 / 966,107 / 966,106, `HAZYNC_CHUNKS=8`, one chunk per card, aggregated over `seg-serve`. Measured, not projected: - compute total 912.8 s / 807.8 s / 711.8 s - straggler 1.118 / 1.048 / 1.059 -- block composition moves the chunk phase more than any knob we have tuned - the aggregate barely moves (203.8 / 205.8 / 189.8 s) across a 22% swing in chunk work; it tracks segment count, not layout - prevout count predicts chunk time, transaction count does not -- the three blocks order in reverse by transactions - both 966,108 runs produced an identical receipt digest despite different chunk timings and one restarted chunk - 966,106 hit #119 and the armed retry recovered it on `attempt=2`, the first live confirmation the wrapper works Card-count projection is anchored on the measured 8-card figure and puts sub-ten-minute blocks at ~13 L40S, not 10.
Answers the open question the doc previously left as "not measured": is `HAZYNC_CHUNKS=16` (two chunks per card) better than one chunk per card? Measured on 966,108, same fleet, same binary, same guest: - straggler falls 1.118 -> 1.085, exactly as summing two draws predicts - but total chunk work rises 1.3% (fixed per-chunk cost paid 16 times) - and the aggregate rises 2.3% (depth-4 join tree, 4.3 s of resolves) - net **905.5 s vs 912.8 s -- 0.8%, inside noise** All sixteen chunks `REAL_EXIT=0, attempt=1`. Chunk inputs sum to 8,562, matching the block's prevout count exactly. The 8- and 16-chunk runs produced the **identical receipt digest**, so the proof does not depend on how the work is divided -- determinism across a structural change, not just a timing one. Conclusion: match chunks to cards. Chunk count is nearly free to get wrong, which is a better reason for the default than load balance.
Second block re-run at 16 chunks. Two chunks hit #119 and the armed retry recovered both on `attempt=2`. - 16 chunks is *slightly worse* here: 819.3 s vs 807.8 s (+1.4%). The 8-chunk straggler was already 1.048, so there was no imbalance to win back and only the +1.2% per-chunk overhead showed up. The 16-chunk gain is bounded by how bad the 8-chunk straggler was. - digest matches the 8-chunk run of the same block, as with 966,108 - inputs sum to 7,961, matching the block's prevout count exactly ⛔ #119 cost 33% of the block -- 817 s wall against 614 s of successful work on the same card, because `prove-chunk` restarts from segment zero. Observed rate across the night is 4 faults in 64 attempts (~6%), which is a ~40% chance of at least one fault per 8-chunk block. The retry is not optional; it belongs in the binary (#237).
Completes the 8-vs-16 comparison across all three blocks. | block | 8 chunks | 16 chunks | Δ | |---------|----------|-----------|-------| | 966,108 | 912.8 s | 905.5 s | -0.8% | | 966,107 | 807.8 s | 819.3 s | +1.4% | | 966,106 | 711.8 s | 711.9 s | +0.0% | ⏰ Chunk count does essentially nothing -- mean +0.2%, and the sign is not consistent. The mechanism shows in the ordering: the 16-chunk gain tracks the 8-chunk straggler monotonically (1.118 -> -0.8%, 1.059 -> 0.0%, 1.048 -> +1.4%). Splitting finer buys back imbalance and nothing else, and roughly pays for its own overhead when it does. ✅ All three blocks gave byte-identical digests at 8 and 16 chunks -- six proofs, three digests. The receipt is a function of the block, not of how the work was divided. ⇒ Keep HAZYNC_CHUNKS equal to the card count. Effort is better spent on #119 (33% of a block when it fires, 6.3% per chunk) than on chunk count. Also notes that HAZYNC_RECEIPTS (#230) is not in the v0.21.0 binary.
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes up last night's fleet runs. Everything here is wall-clock on real hardware; nothing is projected unless it says so.
What was run
Eight UpCloud L40S boxes, v0.21.0 release binary verified against the signed
SHA256SUMS.txton every box, CORE guest37987b85. Blocks 966,108 / 966,107 / 966,106, each proved twice — once atHAZYNC_CHUNKS=8(one chunk per card) and once at 16 (two per card). Six proofs in all.Headline
⏰ Chunk count does essentially nothing — mean +0.2%, and the sign is not consistent. This answers the open question the doc originally left as "not measured".
The mechanism is visible in the ordering: the 16-chunk gain tracks the 8-chunk straggler monotonically (1.118 → −0.8%, 1.059 → 0.0%, 1.048 → +1.4%). Splitting finer buys back imbalance and nothing else, and roughly pays for its own overhead when it does. Recommendation: keep
HAZYNC_CHUNKSequal to the card count.Other measured results
prove-chunkrestarts a faulted chunk from segment zero. Observed rate 5 faults in 80 chunk attempts (6.3%) — a ~40% chance of at least one fault per 8-chunk block. The retry wrapper is not optional. This is the case for prove-chunk has no #119 retry, and it is the path people run by hand #237.Card counts
The projection is anchored on the measured 8-card figure, which is the only reason to trust it. It puts sub-ten-minute blocks at ~13 L40S, not 10.
Incidental finding
HAZYNC_RECEIPTS(#230) is not in the v0.21.0 binary — it merged after the tag. The aggregate readschunk_<i>.binfrom the working directory. Noted in the reproduction section.Refs #119, #237, #226, #230, #236.