Skip to content

docs: 8x L40S benchmark -- three tip blocks, 8 vs 16 chunks #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
defenwycke merged 4 commits into from
Sep 8, 2026
Merged

docs: 8x L40S benchmark -- three tip blocks, 8 vs 16 chunks #238

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c8424a2
docs: benchmark of 8x L40S over three consecutive tip blocks
defenwycke Sep 8, 2026
d87406c
docs: measure 16 chunks on 8 cards -- it is a wash
defenwycke Sep 8, 2026
0cf9b21
docs: 966,107 at 16 chunks, and the measured cost of #119
defenwycke Sep 8, 2026
ddd95b5
docs: 966,106 at 16 chunks, and the verdict on chunk count
defenwycke Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 349 additions & 0 deletions docs/history/BENCH_8xL40S_2026-09-08.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,349 @@
# Eight L40S on one tip block — 2026-09-08

**What this is.** The first fleet measurement of *near-tip* blocks on the v0.21.0 CORE guest: three
consecutive blocks at 8 chunks, the same three re-run at 16, and what the numbers say about card
counts. Everything here is wall-clock on real hardware; nothing is scaled or inferred unless it says
so.

## Setup

| | |
|---|---|
| hardware | 8 × UpCloud `gpu-ubuntu-8cpu-64gb-fi-hel2` — 1× NVIDIA L40S (46 GB), 8 cores, 64 GB RAM, FI-HEL2 |
| binary | v0.21.0 release `hazync-host-x86_64-linux-gnu-cuda`, **verified against the signed `SHA256SUMS.txt` on every box** |
| guest | `37987b85…` (CORE canonical) — confirmed identical on all 8 |
| block | **966,108** — 3,493 txs, 8,562 prevouts |
| fixture | 4,840,086 bytes, byte-identical on all 8 (`fetch_block_rpc.py` from the archive node) |
| topology | `HAZYNC_CHUNKS=8`, one chunk per card, po2 21; aggregate via `seg-serve` + 8 `seg-connect` workers |
| env | `HAZYNC_FIELD_BIGINT2=1 HAZYNC_LIFTX_HINT=1 HAZYNC_ECMULT_WINDOW=21` |

⚠ `HAZYNC_LIFTX_HINT=1` is **required**. Without it the host omits the pubkey hints the CORE guest
expects and the guest dies with `DeserializeUnexpectedEnd` — which reads as a corrupt fixture.

## Run 1 — 20:10:53Z, hit by #119

Seven chunks finished cleanly in **13m29s**. Chunk 2 panicked at segment ~65 of 271, roughly nine
minutes in:

```
thread 'main' panicked at host/src/main.rs:2485:56:
called `Result::unwrap()` on an `Err` value: verify segment
Caused by: verification indicates proof is invalid
```

That is hazync#119. `prove-chunk` has **no built-in retry** — the retry lives in
`scripts/gpu-benchmark.sh`, outside the binary — so the card produced nothing and had to restart from
zero. Filed as **hazync#237**.

| phase | |
|---|---|
| 7 chunks clean | 13m29s |
| chunk 2 after restart | 21m05s |
| aggregate | 223.3 s |
| **end-to-end** | **26m00s** |

⚠ The original panic log was overwritten by the relaunch. The text above is quoted from hazync#237,
which was filed from it at the time.

## Run 2 — 20:38:09Z, clean

Same block, retry armed, **it never fired**. Every chunk `REAL_EXIT=0, attempt=1`.

| chunk | box | segments | time |
|---|---|---|---|
| 4 | .150.120 | 289 | **709 s** ← critical path |
| 3 | .151.37 | 301 | 704 s |
| 2 | .148.65 | 271 | 632 s |
| 5 | .148.197 | 251 | 622 s |
| 7 | .148.182 | 263 | 614 s |
| 0 | .148.248 | 266 | 605 s |
| 6 | .150.87 | 259 | 604 s |
| 1 | .148.16 | 249 | 585 s |

- total chunk work **5,075 card-seconds**, 2,149 segments
- mean 634.4 s, max 709 s ⇒ **straggler 1.118**

Aggregate, 8 workers:

```
execution 19.2 s
worker wall 159.3 s 539 segments over the network
assembly 25.3 s last segment + join tree + resolves
TOTAL 203.8 s
>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID
digest 34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc
```

| | |
|---|---|
| chunk phase | 709 s (11m49s) |
| aggregate | 203.8 s (3m24s) |
| compute total | **912.8 s — 15m13s** |
| end-to-end incl. manual receipt collection | 16m42s |

✅ **Both runs produced the identical digest** `34dbe8e4…`, across different chunk timings and a
restarted chunk. That is the determinism the whole scheme rests on, observed rather than assumed.

## Three consecutive blocks, same fleet, same settings

| block | txs | prevouts | segments | chunk work | mean | max | straggler | aggregate | compute total |
|---|---|---|---|---|---|---|---|---|---|
| 966,108 | 3,493 | 8,562 | 2,149 | 5,075 cs | 634.4 s | 709 s | **1.118** | 203.8 s | **912.8 s** |
| 966,107 | 4,574 | 7,961 | 1,943 | 4,594 cs | 574.3 s | 602 s | **1.048** | 205.8 s | **807.8 s** |
| 966,106 | 4,176 | 6,644 | 1,658 | 3,942 cs | 492.8 s | 522 s | **1.059** | 189.8 s | **711.8 s** |

Digests, all verified against `METHOD_ID` by the coordinator itself:

| block | receipt digest |
|---|---|
| 966,108 | `34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc` |
| 966,107 | `2ad4a28ac6bdfa793a5891313256ed4bcc8b3a2bb65e7a81851c9aeebb227997` |
| 966,106 | `448aa75fd8736b7953cc104cef82cfe07de6470e09c13bb6d949e3656d77dab4` |

966,108 and 966,107 were clean — every chunk `REAL_EXIT=0, attempt=1`. **966,106 hit #119 on chunk 2
and the armed retry recovered it on `attempt=2`**, which is the first live confirmation that the
retry wrapper works: same fault as run 1 of 966,108, but it cost one chunk restart inside the run
instead of a 26-minute end-to-end. That is the case hazync#237 asks the binary to handle itself.

⏰ **Straggler varies more between blocks than any knob we have tuned.** 1.048 / 1.059 / 1.118 on
three consecutive blocks is a 6.7% swing in the chunk phase, from block composition alone. For comparison,
the entire packing-constant sweep in hazync#226 was worth ~3% end-to-end and the per-block-profile
ceiling was 0.5%. **Which block you get matters more than how you tune the packer.**

⚠ The aggregate barely moves across all three (203.8 / 205.8 / 189.8 s) despite the chunk phase
falling 22% from 966,108 to 966,106. It is dominated by segment count, and segment count tracks total
work, not layout — 966,106 has 23% less chunk work but only 7% less aggregate.

⏰ **Prevout count predicts the chunk phase; transaction count does not.** Ordered by transactions
the blocks run 966,107 > 966,106 > 966,108; ordered by chunk time they run exactly the *reverse*.
966,106 has 20% more transactions than 966,108 and a 26% shorter chunk phase. Prevouts are what the
guest verifies, and they track chunk time monotonically across all three.

## What it says about card counts

The two phases scale differently. Chunk work divides by N; the aggregate divides by N *except* a
~19 s execute that does not.

```
block ≈ (5,075 / N) × straggler + 19.2 + (184.6 / N × 8)
```

| cards | projected block |
|---|---|
| 8 | 15.2 min *(measured: 15m13s)* |
| 10 | 12.3 min |
| 12 | 10.3 min |
| **13** | **~9.8 min** |
| 16 | 7.8 min |

⇒ **Sub-ten-minute blocks need ~13 L40S**, not 10. The formula reproduces the measured 8-card figure,
which is the only reason to trust it at all.

⚠ Two caveats, both in the pessimistic direction. Straggler is likely to get *worse* above 8 chunks —
fewer inputs per chunk means more relative variance — and only N=8 has been measured. The ~19 s
execute is removed from the critical path by hazync#236 (merged after these runs), worth ~2% at N=8
and more as N grows.

## Why 8 chunks and not 16 — now measured

The 16-chunk figure in `docs/BUILDS.md` comes from `gpu-benchmark.sh`, which proves all chunks
**sequentially on one card**. There, chunk count does not affect load balance at all — the reported
straggler is a *prediction* of how 16 chunks would spread across 16 cards, not a measurement of
anything.

So we ran it: same fleet, same block, `HAZYNC_CHUNKS=16`, **two chunks per card, proved serially**.

### 966,108 at 16 chunks

| card | chunks | times | card total |
|---|---|---|---|
| .148.182 | 15, 7 | 302 + 395 | **697 s** ← critical path |
| .148.197 | 13, 5 | 342 + 336 | 678 s |
| .150.87 | 14, 6 | 319 + 345 | 664 s |
| .148.248 | 0, 8 | 296 + 354 | 650 s |
| .150.120 | 12, 4 | 307 + 338 | 645 s |
| .148.16 | 1, 9 | 317 + 316 | 633 s |
| .151.37 | 11, 3 | 287 + 312 | 599 s |
| .148.65 | 10, 2 | 299 + 274 | 573 s |

All sixteen `REAL_EXIT=0, attempt=1`; no #119, no retries. Chunk inputs sum to **8,562**, matching the
block's prevout count exactly.

### The comparison

| | 8 chunks (1/card) | 16 chunks (2/card) | |
|---|---|---|---|
| total chunk work | 5,075 cs | 5,139 cs | **+1.3%** |
| segments (chunk phase) | 2,149 | 2,156 | +0.3% |
| mean per card | 634.4 s | 642.4 s | +1.3% |
| max per card | 709 s | 697 s | **−1.7%** |
| **straggler** | **1.118** | **1.085** | **−3.0%** |
| aggregate segments | 539 | 541 | +0.4% |
| aggregate tree depth | 3 | 4 | |
| aggregate resolves | — | 4.3 s (16 pushed) | |
| aggregate total | 203.8 s | 208.5 s | **+2.3%** |
| **compute total** | **912.8 s** | **905.5 s** | **−0.8%** |

✅ **Both chunkings produced the identical digest** `34dbe8e42b24a50123ea739f070cb8e45d364b2a2dd9024fe141e302305fc9cc`.
The receipt does not depend on how the work was divided — the same guarantee as the run-1/run-2 match
above, but across a *structural* change rather than a timing one.

### Reading it

⏰ **On this block, sixteen chunks is a wash: 0.8% faster, inside run-to-run noise.** The load-balance
prediction was right — the straggler fell 1.118 → 1.085, exactly as summing two draws should — but the
win is spent twice over:

1. **Chunk work rises 1.3%.** Each chunk carries a fixed per-chunk cost (execute, hint assembly,
receipt framing) that does not shrink when the chunk does. Sixteen chunks pay it sixteen times.
2. **The aggregate rises 2.3%.** Sixteen receipts is a depth-4 join tree instead of depth-3, and adds
4.3 s of resolves. The aggregate is the phase that *dislikes* more chunks.

The two overheads cancel the balance gain. Whether that holds generally is the question the other two
blocks answer below.

### 966,107 at 16 chunks — and what #119 costs

Same treatment on the next block. **Two chunks hit hazync#119** (chunk 12 on `.150.120`, chunk 13 on
`.148.197`); the armed retry recovered both on `attempt=2`.

| card | chunk times (successful attempt) | successful work | **wall-clock** |
|---|---|---|---|
| .150.120 | 310 + 304 | 614 s | **817 s** ← #119 |
| .148.197 | 311 + 303 | 614 s | **782 s** ← #119 |
| .148.182 | 293 + 293 | 586 s | 586 s |
| .148.65 | 294 + 285 | 579 s | 580 s |
| .148.16 | 281 + 292 | 573 s | 574 s |
| .151.37 | 287 + 285 | 572 s | 572 s |
| .150.87 | 280 + 289 | 569 s | 569 s |
| .148.248 | 272 + 270 | 542 s | 543 s |

Inputs sum to **7,961**, again matching the block's prevout count exactly. Aggregate:

```
execution 20.2 s 570 segments, 281.9 MB, depth 4
worker wall 159.5 s
assembly 25.7 s of which resolves 4.3 s
TOTAL 205.3 s
>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID
digest 2ad4a28ac6bdfa793a5891313256ed4bcc8b3a2bb65e7a81851c9aeebb227997
```

Digest matches the 8-chunk run of the same block. Two chunkings, two receipts, one digest — as with
966,108.

⛔ **#119 cost 33% of the block.** Critical path 817 s as-run against 614 s of successful work on the
same card. `prove-chunk` restarts a faulted chunk **from segment zero**, so a fault two-thirds through
throws away two-thirds of a chunk. On a 1:1 mapping that is the whole block's critical path; here it
was absorbed slightly better only because the card had a second chunk to hide behind.

⏰ **Observed #119 rate: 5 faults in 80 chunk attempts across the night — 6.3%.** With 8 chunks that is
a ~40% chance of at least one fault per block; with 16, ~64%. It is not a rare event, and the retry
wrapper is not optional. This is the case for **hazync#237** — the retry belongs in the binary, and
ideally it should resume rather than restart.

| 966,107 | 8 chunks | 16 chunks (fault-free equiv.) | 16 chunks (as-run) |
|---|---|---|---|
| total chunk work | 4,594 cs | 4,649 cs (**+1.2%**) | — |
| straggler | **1.048** | 1.057 | — |
| critical path | 602 s | 614 s | 817 s |
| aggregate | 205.8 s | 205.3 s | 205.3 s |
| **compute total** | **807.8 s** | **819.3 s (+1.4%)** | 1,022.3 s |

⇒ On 966,107, sixteen chunks is *slightly worse*. Its 8-chunk straggler was already 1.048 — there was
no imbalance left to win back, so only the +1.2% overhead showed up. **The 16-chunk gain is bounded by
how bad the 8-chunk straggler was**, and that is a property of the block, not a knob.

### 966,106 at 16 chunks

One #119, on chunk 14 (`.150.87`), recovered on `attempt=2`. Everything else `attempt=1`.

| card | chunk times | successful work | wall-clock |
|---|---|---|---|
| .150.120 | 257 + 256 | **513 s** ← critical path | 513 s |
| .148.197 | 255 + 255 | 510 s | 510 s |
| .150.87 | 247 + 255 | 502 s | **593 s** ← #119 |
| .148.65 | 246 + 246 | 492 s | 492 s |
| .148.182 | 232 + 255 | 487 s | 487 s |
| .148.16 | 243 + 241 | 484 s | 484 s |
| .148.248 | 242 + 240 | 482 s | 481 s |
| .151.37 | 246 + 232 | 478 s | 478 s |

Inputs sum to **6,644** — the block's prevout count. Aggregate:

```
execution 18.6 s 519 segments, 258.0 MB, depth 4
worker wall 156.1 s
assembly 24.3 s of which resolves 4.5 s
TOTAL 198.9 s
>>> PUSH-TRANSPORT RECEIPT VERIFIED against METHOD_ID
digest 448aa75fd8736b7953cc104cef82cfe07de6470e09c13bb6d949e3656d77dab4
```

Digest matches the 8-chunk run.

## The verdict on chunk count

Three blocks, each proved twice — once at 8 chunks (1 per card) and once at 16 (2 per card), same
fleet, same binary, same guest. 16-chunk figures are the fault-free equivalent, so #119 does not
contaminate the comparison.

| block | 8 chunks | 16 chunks | Δ | 8-chunk straggler | 16-chunk straggler |
|---|---|---|---|---|---|
| 966,108 | 912.8 s | **905.5 s** | **−0.8%** | 1.118 | 1.085 |
| 966,107 | **807.8 s** | 819.3 s | +1.4% | 1.048 | 1.057 |
| 966,106 | 711.8 s | 711.9 s | +0.0% | 1.059 | 1.040 |

⏰ **Chunk count does essentially nothing: mean +0.2% over three blocks, and the sign is not even
consistent.** Whichever you pick, you get the same block time.

The mechanism is visible in the ordering. **The 16-chunk gain tracks how bad the 8-chunk straggler
was, monotonically:**

| 8-chunk straggler | 16-chunk Δ |
|---|---|
| 1.118 (worst balance) | −0.8% (best gain) |
| 1.059 | +0.0% |
| 1.048 (best balance) | +1.4% (worst) |

Splitting finer buys back imbalance, and nothing else. When there is imbalance to buy back it roughly
pays for its own overhead; when there is not, you pay the overhead for nothing. That overhead is
consistent across all three blocks: **+0.15% to +1.3% more chunk work** (a fixed per-chunk cost paid
twice as many times) and **−0.2% to +4.8% on the aggregate** (depth-4 join tree, ~4.4 s of resolves
for 16 receipts instead of 8).

✅ **All three blocks produced byte-identical digests at 8 and 16 chunks.** Six proofs, three digests.
The receipt is a function of the block, not of how the work was divided — which is what makes chunk
count a free operational choice rather than a consensus one.

⇒ **Keep `HAZYNC_CHUNKS` equal to the card count.** Not because it balances best, but because it is
the simplest mapping and the alternative is worth nothing. Effort spent on chunk count is effort not
spent on #119 (which costs 33% of a block when it fires, at a 6.3% per-chunk rate) or on the
aggregate's fixed ~19 s execute.

⚠ This says nothing about 16 chunks on **16 cards**, which is a different question — there the chunk
phase would genuinely halve and only the +1.3% and the deeper tree would be paid. The card-count
projection above already assumes that regime.

## Reproducing

```sh
# per card i in 0..7
HAZYNC_FIELD_BIGINT2=1 HAZYNC_LIFTX_HINT=1 HAZYNC_ECMULT_WINDOW=21 \
HAZYNC_BLOCK=block_966108.json HAZYNC_CHUNKS=8 HAZYNC_CHUNK=$i \
HAZYNC_OUT=chunk_$i.bin ./hazync-host-x86_64-linux-gnu-cuda prove-chunk $i

# aggregate: coordinator with the 8 receipts in CWD, then a worker per card
HAZYNC_BLOCK=block_966108.json HAZYNC_CHUNKS=8 HAZYNC_AGG=1 HAZYNC_PORT=9160 \
./hazync-host-x86_64-linux-gnu-cuda seg-serve
./hazync-host-x86_64-linux-gnu-cuda seg-connect <coordinator>:9160
```

⛔ Wrap `prove-chunk` in a retry that re-runs on `verification indicates proof is invalid`
(hazync#237). Without it a single #119 fault costs the whole chunk, as it did in run 1.

Raw logs: `~/hazync-v0.21.0-release-evidence/bench-8xL40S-2026-09-08/`.

For the 16-chunk runs, substitute `HAZYNC_CHUNKS=16` and give each card two chunk indices, `i` and
`i+8`, proved serially. ⚠ `HAZYNC_RECEIPTS` (hazync#230) is **not** in the v0.21.0 binary — the
aggregate reads `chunk_<i>.bin` from the working directory, so `cd` into the receipt directory.
Loading