diff --git a/docs/history/BENCH_8xL40S_2026-09-08.md b/docs/history/BENCH_8xL40S_2026-09-08.md new file mode 100644 index 0000000..3d91996 --- /dev/null +++ b/docs/history/BENCH_8xL40S_2026-09-08.md @@ -0,0 +1,349 @@ +# Eight L40S on one tip block — 2026-09-08 + +**What this is.** The first fleet measurement of *near-tip* blocks on the v0.21.0 CORE guest: three +consecutive blocks at 8 chunks, the same three re-run at 16, and what the numbers say about card +counts. Everything here is wall-clock on real hardware; nothing is scaled or inferred unless it says +so. + +## Setup + +| | | +|---|---| +| hardware | 8 × UpCloud `gpu-ubuntu-8cpu-64gb-fi-hel2` — 1× NVIDIA L40S (46 GB), 8 cores, 64 GB RAM, FI-HEL2 | +| binary | v0.21.0 release `hazync-host-x86_64-linux-gnu-cuda`, **verified against the signed `SHA256SUMS.txt` on every box** | +| guest | `37987b85…` (CORE canonical) — confirmed identical on all 8 | +| block | **966,108** — 3,493 txs, 8,562 prevouts | +| fixture | 4,840,086 bytes, byte-identical on all 8 (`fetch_block_rpc.py` from the archive node) | +| topology | `HAZYNC_CHUNKS=8`, one chunk per card, po2 21; aggregate via `seg-serve` + 8 `seg-connect` workers | +| env | `HAZYNC_FIELD_BIGINT2=1 HAZYNC_LIFTX_HINT=1 HAZYNC_ECMULT_WINDOW=21` | + +⚠ `HAZYNC_LIFTX_HINT=1` is **required**. Without it the host omits the pubkey hints the CORE guest +expects and the guest dies with `DeserializeUnexpectedEnd` — which reads as a corrupt fixture. + +## Run 1 — 20:10:53Z, hit by #119 + +Seven chunks finished cleanly in **13m29s**. Chunk 2 panicked at segment ~65 of 271, roughly nine +minutes in: + +``` +thread 'main' panicked at host/src/main.rs:2485:56: +called `Result::unwrap()` on an `Err` value: verify segment +Caused by: verification indicates proof is invalid +``` + +That is hazync#119. `prove-chunk` has **no built-in retry** — the retry lives in +`scripts/gpu-benchmark.sh`, outside the binary — so the card produced nothing and had to restart from +zero. Filed as **hazync#237**. + +| phase | | +|---|---| +| 7 chunks clean | 13m29s | +| chunk 2 after restart | 21m05s | +| aggregate | 223.3 s | +| **end-to-end** | **26m00s** | + +⚠ The original panic log was overwritten by the relaunch. The text above is quoted from hazync#237, +which was filed from it at the time. + +## Run 2 — 20:38:09Z, clean + +Same block, retry armed, **it never fired**. Every chunk `REAL_EXIT=0, attempt=1`. + +| chunk | box | segments | time | +|---|---|---|---| +| 4 | .150.120 | 289 | **709 s** ← critical path | +| 3 | .151.37 | 301 | 704 s | +| 2 | .148.65 | 271 | 632 s | +| 5 | .148.197 | 251 | 622 s | +| 7 | .148.182 | 263 | 614 s | +| 0 | .148.248 | 266 | 605 s | +| 6 | .150.87 | 259 | 604 s | +| 1 | .148.16 | 249 | 585 s | + +- total chunk work **5,075 card-seconds**, 2,149 segments +- mean 634.4 s, max 709 s ⇒ **straggler 1.118** + +Aggregate, 8 workers: + +``` +execution 19.2 s +worker wall 159.3 s 539 segments over the network +assembly 25.3 s last segment + join tree + resolves +TOTAL 203.8 s +>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID + digest 34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc +``` + +| | | +|---|---| +| chunk phase | 709 s (11m49s) | +| aggregate | 203.8 s (3m24s) | +| compute total | **912.8 s — 15m13s** | +| end-to-end incl. manual receipt collection | 16m42s | + +✅ **Both runs produced the identical digest** `34dbe8e4…`, across different chunk timings and a +restarted chunk. That is the determinism the whole scheme rests on, observed rather than assumed. + +## Three consecutive blocks, same fleet, same settings + +| block | txs | prevouts | segments | chunk work | mean | max | straggler | aggregate | compute total | +|---|---|---|---|---|---|---|---|---|---| +| 966,108 | 3,493 | 8,562 | 2,149 | 5,075 cs | 634.4 s | 709 s | **1.118** | 203.8 s | **912.8 s** | +| 966,107 | 4,574 | 7,961 | 1,943 | 4,594 cs | 574.3 s | 602 s | **1.048** | 205.8 s | **807.8 s** | +| 966,106 | 4,176 | 6,644 | 1,658 | 3,942 cs | 492.8 s | 522 s | **1.059** | 189.8 s | **711.8 s** | + +Digests, all verified against `METHOD_ID` by the coordinator itself: + +| block | receipt digest | +|---|---| +| 966,108 | `34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc` | +| 966,107 | `2ad4a28ac6bdfa793a5891313256ed4bcc8b3a2bb65e7a81851c9aeebb227997` | +| 966,106 | `448aa75fd8736b7953cc104cef82cfe07de6470e09c13bb6d949e3656d77dab4` | + +966,108 and 966,107 were clean — every chunk `REAL_EXIT=0, attempt=1`. **966,106 hit #119 on chunk 2 +and the armed retry recovered it on `attempt=2`**, which is the first live confirmation that the +retry wrapper works: same fault as run 1 of 966,108, but it cost one chunk restart inside the run +instead of a 26-minute end-to-end. That is the case hazync#237 asks the binary to handle itself. + +⏰ **Straggler varies more between blocks than any knob we have tuned.** 1.048 / 1.059 / 1.118 on +three consecutive blocks is a 6.7% swing in the chunk phase, from block composition alone. For comparison, +the entire packing-constant sweep in hazync#226 was worth ~3% end-to-end and the per-block-profile +ceiling was 0.5%. **Which block you get matters more than how you tune the packer.** + +⚠ The aggregate barely moves across all three (203.8 / 205.8 / 189.8 s) despite the chunk phase +falling 22% from 966,108 to 966,106. It is dominated by segment count, and segment count tracks total +work, not layout — 966,106 has 23% less chunk work but only 7% less aggregate. + +⏰ **Prevout count predicts the chunk phase; transaction count does not.** Ordered by transactions +the blocks run 966,107 > 966,106 > 966,108; ordered by chunk time they run exactly the *reverse*. +966,106 has 20% more transactions than 966,108 and a 26% shorter chunk phase. Prevouts are what the +guest verifies, and they track chunk time monotonically across all three. + +## What it says about card counts + +The two phases scale differently. Chunk work divides by N; the aggregate divides by N *except* a +~19 s execute that does not. + +``` +block ≈ (5,075 / N) × straggler + 19.2 + (184.6 / N × 8) +``` + +| cards | projected block | +|---|---| +| 8 | 15.2 min *(measured: 15m13s)* | +| 10 | 12.3 min | +| 12 | 10.3 min | +| **13** | **~9.8 min** | +| 16 | 7.8 min | + +⇒ **Sub-ten-minute blocks need ~13 L40S**, not 10. The formula reproduces the measured 8-card figure, +which is the only reason to trust it at all. + +⚠ Two caveats, both in the pessimistic direction. Straggler is likely to get *worse* above 8 chunks — +fewer inputs per chunk means more relative variance — and only N=8 has been measured. The ~19 s +execute is removed from the critical path by hazync#236 (merged after these runs), worth ~2% at N=8 +and more as N grows. + +## Why 8 chunks and not 16 — now measured + +The 16-chunk figure in `docs/BUILDS.md` comes from `gpu-benchmark.sh`, which proves all chunks +**sequentially on one card**. There, chunk count does not affect load balance at all — the reported +straggler is a *prediction* of how 16 chunks would spread across 16 cards, not a measurement of +anything. + +So we ran it: same fleet, same block, `HAZYNC_CHUNKS=16`, **two chunks per card, proved serially**. + +### 966,108 at 16 chunks + +| card | chunks | times | card total | +|---|---|---|---| +| .148.182 | 15, 7 | 302 + 395 | **697 s** ← critical path | +| .148.197 | 13, 5 | 342 + 336 | 678 s | +| .150.87 | 14, 6 | 319 + 345 | 664 s | +| .148.248 | 0, 8 | 296 + 354 | 650 s | +| .150.120 | 12, 4 | 307 + 338 | 645 s | +| .148.16 | 1, 9 | 317 + 316 | 633 s | +| .151.37 | 11, 3 | 287 + 312 | 599 s | +| .148.65 | 10, 2 | 299 + 274 | 573 s | + +All sixteen `REAL_EXIT=0, attempt=1`; no #119, no retries. Chunk inputs sum to **8,562**, matching the +block's prevout count exactly. + +### The comparison + +| | 8 chunks (1/card) | 16 chunks (2/card) | | +|---|---|---|---| +| total chunk work | 5,075 cs | 5,139 cs | **+1.3%** | +| segments (chunk phase) | 2,149 | 2,156 | +0.3% | +| mean per card | 634.4 s | 642.4 s | +1.3% | +| max per card | 709 s | 697 s | **−1.7%** | +| **straggler** | **1.118** | **1.085** | **−3.0%** | +| aggregate segments | 539 | 541 | +0.4% | +| aggregate tree depth | 3 | 4 | | +| aggregate resolves | — | 4.3 s (16 pushed) | | +| aggregate total | 203.8 s | 208.5 s | **+2.3%** | +| **compute total** | **912.8 s** | **905.5 s** | **−0.8%** | + +✅ **Both chunkings produced the identical digest** `34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc`. +The receipt does not depend on how the work was divided — the same guarantee as the run-1/run-2 match +above, but across a *structural* change rather than a timing one. + +### Reading it + +⏰ **On this block, sixteen chunks is a wash: 0.8% faster, inside run-to-run noise.** The load-balance +prediction was right — the straggler fell 1.118 → 1.085, exactly as summing two draws should — but the +win is spent twice over: + +1. **Chunk work rises 1.3%.** Each chunk carries a fixed per-chunk cost (execute, hint assembly, + receipt framing) that does not shrink when the chunk does. Sixteen chunks pay it sixteen times. +2. **The aggregate rises 2.3%.** Sixteen receipts is a depth-4 join tree instead of depth-3, and adds + 4.3 s of resolves. The aggregate is the phase that *dislikes* more chunks. + +The two overheads cancel the balance gain. Whether that holds generally is the question the other two +blocks answer below. + +### 966,107 at 16 chunks — and what #119 costs + +Same treatment on the next block. **Two chunks hit hazync#119** (chunk 12 on `.150.120`, chunk 13 on +`.148.197`); the armed retry recovered both on `attempt=2`. + +| card | chunk times (successful attempt) | successful work | **wall-clock** | +|---|---|---|---| +| .150.120 | 310 + 304 | 614 s | **817 s** ← #119 | +| .148.197 | 311 + 303 | 614 s | **782 s** ← #119 | +| .148.182 | 293 + 293 | 586 s | 586 s | +| .148.65 | 294 + 285 | 579 s | 580 s | +| .148.16 | 281 + 292 | 573 s | 574 s | +| .151.37 | 287 + 285 | 572 s | 572 s | +| .150.87 | 280 + 289 | 569 s | 569 s | +| .148.248 | 272 + 270 | 542 s | 543 s | + +Inputs sum to **7,961**, again matching the block's prevout count exactly. Aggregate: + +``` +execution 20.2 s 570 segments, 281.9 MB, depth 4 +worker wall 159.5 s +assembly 25.7 s of which resolves 4.3 s +TOTAL 205.3 s +>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID + digest 2ad4a28ac6bdfa793a5891313256ed4bcc8b3a2bb65e7a81851c9aeebb227997 +``` + +Digest matches the 8-chunk run of the same block. Two chunkings, two receipts, one digest — as with +966,108. + +⛔ **#119 cost 33% of the block.** Critical path 817 s as-run against 614 s of successful work on the +same card. `prove-chunk` restarts a faulted chunk **from segment zero**, so a fault two-thirds through +throws away two-thirds of a chunk. On a 1:1 mapping that is the whole block's critical path; here it +was absorbed slightly better only because the card had a second chunk to hide behind. + +⏰ **Observed #119 rate: 5 faults in 80 chunk attempts across the night — 6.3%.** With 8 chunks that is +a ~40% chance of at least one fault per block; with 16, ~64%. It is not a rare event, and the retry +wrapper is not optional. This is the case for **hazync#237** — the retry belongs in the binary, and +ideally it should resume rather than restart. + +| 966,107 | 8 chunks | 16 chunks (fault-free equiv.) | 16 chunks (as-run) | +|---|---|---|---| +| total chunk work | 4,594 cs | 4,649 cs (**+1.2%**) | — | +| straggler | **1.048** | 1.057 | — | +| critical path | 602 s | 614 s | 817 s | +| aggregate | 205.8 s | 205.3 s | 205.3 s | +| **compute total** | **807.8 s** | **819.3 s (+1.4%)** | 1,022.3 s | + +⇒ On 966,107, sixteen chunks is *slightly worse*. Its 8-chunk straggler was already 1.048 — there was +no imbalance left to win back, so only the +1.2% overhead showed up. **The 16-chunk gain is bounded by +how bad the 8-chunk straggler was**, and that is a property of the block, not a knob. + +### 966,106 at 16 chunks + +One #119, on chunk 14 (`.150.87`), recovered on `attempt=2`. Everything else `attempt=1`. + +| card | chunk times | successful work | wall-clock | +|---|---|---|---| +| .150.120 | 257 + 256 | **513 s** ← critical path | 513 s | +| .148.197 | 255 + 255 | 510 s | 510 s | +| .150.87 | 247 + 255 | 502 s | **593 s** ← #119 | +| .148.65 | 246 + 246 | 492 s | 492 s | +| .148.182 | 232 + 255 | 487 s | 487 s | +| .148.16 | 243 + 241 | 484 s | 484 s | +| .148.248 | 242 + 240 | 482 s | 481 s | +| .151.37 | 246 + 232 | 478 s | 478 s | + +Inputs sum to **6,644** — the block's prevout count. Aggregate: + +``` +execution 18.6 s 519 segments, 258.0 MB, depth 4 +worker wall 156.1 s +assembly 24.3 s of which resolves 4.5 s +TOTAL 198.9 s +>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID + digest 448aa75fd8736b7953cc104cef82cfe07de6470e09c13bb6d949e3656d77dab4 +``` + +Digest matches the 8-chunk run. + +## The verdict on chunk count + +Three blocks, each proved twice — once at 8 chunks (1 per card) and once at 16 (2 per card), same +fleet, same binary, same guest. 16-chunk figures are the fault-free equivalent, so #119 does not +contaminate the comparison. + +| block | 8 chunks | 16 chunks | Δ | 8-chunk straggler | 16-chunk straggler | +|---|---|---|---|---|---| +| 966,108 | 912.8 s | **905.5 s** | **−0.8%** | 1.118 | 1.085 | +| 966,107 | **807.8 s** | 819.3 s | +1.4% | 1.048 | 1.057 | +| 966,106 | 711.8 s | 711.9 s | +0.0% | 1.059 | 1.040 | + +⏰ **Chunk count does essentially nothing: mean +0.2% over three blocks, and the sign is not even +consistent.** Whichever you pick, you get the same block time. + +The mechanism is visible in the ordering. **The 16-chunk gain tracks how bad the 8-chunk straggler +was, monotonically:** + +| 8-chunk straggler | 16-chunk Δ | +|---|---| +| 1.118 (worst balance) | −0.8% (best gain) | +| 1.059 | +0.0% | +| 1.048 (best balance) | +1.4% (worst) | + +Splitting finer buys back imbalance, and nothing else. When there is imbalance to buy back it roughly +pays for its own overhead; when there is not, you pay the overhead for nothing. That overhead is +consistent across all three blocks: **+0.15% to +1.3% more chunk work** (a fixed per-chunk cost paid +twice as many times) and **−0.2% to +4.8% on the aggregate** (depth-4 join tree, ~4.4 s of resolves +for 16 receipts instead of 8). + +✅ **All three blocks produced byte-identical digests at 8 and 16 chunks.** Six proofs, three digests. +The receipt is a function of the block, not of how the work was divided — which is what makes chunk +count a free operational choice rather than a consensus one. + +⇒ **Keep `HAZYNC_CHUNKS` equal to the card count.** Not because it balances best, but because it is +the simplest mapping and the alternative is worth nothing. Effort spent on chunk count is effort not +spent on #119 (which costs 33% of a block when it fires, at a 6.3% per-chunk rate) or on the +aggregate's fixed ~19 s execute. + +⚠ This says nothing about 16 chunks on **16 cards**, which is a different question — there the chunk +phase would genuinely halve and only the +1.3% and the deeper tree would be paid. The card-count +projection above already assumes that regime. + +## Reproducing + +```sh +# per card i in 0..7 +HAZYNC_FIELD_BIGINT2=1 HAZYNC_LIFTX_HINT=1 HAZYNC_ECMULT_WINDOW=21 \ +HAZYNC_BLOCK=block_966108.json HAZYNC_CHUNKS=8 HAZYNC_CHUNK=$i \ +HAZYNC_OUT=chunk_$i.bin ./hazync-host-x86_64-linux-gnu-cuda prove-chunk $i + +# aggregate: coordinator with the 8 receipts in CWD, then a worker per card +HAZYNC_BLOCK=block_966108.json HAZYNC_CHUNKS=8 HAZYNC_AGG=1 HAZYNC_PORT=9160 \ + ./hazync-host-x86_64-linux-gnu-cuda seg-serve +./hazync-host-x86_64-linux-gnu-cuda seg-connect :9160 +``` + +⛔ Wrap `prove-chunk` in a retry that re-runs on `verification indicates proof is invalid` +(hazync#237). Without it a single #119 fault costs the whole chunk, as it did in run 1. + +Raw logs: `~/hazync-v0.21.0-release-evidence/bench-8xL40S-2026-09-08/`. + +For the 16-chunk runs, substitute `HAZYNC_CHUNKS=16` and give each card two chunk indices, `i` and +`i+8`, proved serially. ⚠ `HAZYNC_RECEIPTS` (hazync#230) is **not** in the v0.21.0 binary — the +aggregate reads `chunk_.bin` from the working directory, so `cd` into the receipt directory.