Ghost's next build: the four levers reopened by "fastest wins"

[defenwycke]

Ghost is defined as the fastest non-Core code we can get. That definition reopens levers previously closed on cost/benefit — each was rejected because the fidelity spend was not worth the gain, and under "fastest wins" that reasoning no longer applies.

Baseline to beat: GHOST = 1,198,904,653 cycles, 11.467x, straggler 1.407, 5 cards (docs/BUILDS.md).

⏰ Ghost sits at 4.05 of 5 cards. A straggler of 1.35 instead of 1.407 makes it four. That is a 4% packing improvement worth 20% of the fleet, and it is the cheapest item here.

---

1 · Ghost packer calibration — do this first

Worth a card. Cheapest thing on the list. Never been done.

Ghost has never had its cost model calibrated. Core's refit took its straggler 1.557 → 1.311; the same refit applied to Ghost made it worse — 1.407 → 1.884, and 1.996 with the field backend.

The likely cause: I rescaled Ghost's Schnorr constant by the old 13.77x ECDSA ratio, but the liftx hint removes pubkey decompression from both curves (taproot key-path spends call lift_x too), so that ratio should have narrowed. Keeping it over-prices Schnorr inputs.

Method: HAZYNC_CHUNKS=16 HAZYNC_PROFILE_EXEC=1 host chunk-profile, least-squares fit (ecdsa, schnorr, bytes, inputs) → cycles — and this time fit the two curves separately rather than rescaling a stale ratio. The constants are already runtime-overridable (HAZYNC_COST_EC_OP etc.).

⛔ Also unmeasured: GHOST's straggler at default constants. It is assumed 1.407 from the no-backend arm. That assumption is what says 5 cards rather than 4.

2 · MSM batch verification

Previously: rejected. Sized at 9.24x over a whole block but 4.3x at chunk scale after the checks it cannot remove — worth one card, 5 → 4, in exchange for removing libsecp from per-signature verification and a Fiat-Shamir soundness argument needing outside review.

Now: that card counts. The primitive is already in-tree, tested, off by default (prover/methods/guest/src/msm.rs). → docs/history/MSM_BATCH_VERIFY.md

⚠ Interacts with (1): both target the same card. Do the calibration first and re-size MSM against what remains.

3 · Wholesale bigint2

Previously: the middle path was chosen because wholesale "buys at most ONE card" for 15% more speed. Now: 15% is 15%.

⚠ Re-measure rather than trust the old figure — it was taken before the field backend, which changes what the non-ECDSA residual costs.

4 · G3 Schnorr and scalar-inverse re-measure

Both were sized before the liftx hint landed. The hint changes the ECDSA:Schnorr cost ratio (see 1), so their contributions have moved and neither has been re-measured in the current build.

---

Method

Each lever measured independently against the pinned GHOST baseline, in execute mode on CPU, with:

• the journal digest byte-identical to 4fb3e3c5…4656d — a mismatch voids the arm
• a two-sided binary symbol assertion before any number counts. Eight silent no-ops were caught this way in one session; every one would have reported a moved METHOD_ID and a clean digest while measuring nothing.
• the straggler read from measured cycles, never predicted

Wall-clock on a GPU confirms at the end, not per lever.

[defenwycke]

Two updates from measurement, both of which make item 1 (Ghost packer calibration) more clearly the right first move — and one correction to this issue’s own baseline.

Correction: the straggler is worse than stated

This issue quotes 1.407. That figure was assumed from the no-backend arm and was never measured for the backend arm (BUILDS.md §4.2 flags it as exactly that). Measured on two L40S with real proving, Ghost’s straggler is 1.438.

So the gap is 1.438 → 1.35, not 1.407 → 1.35. The card is still there and still the cheapest item here; it is just slightly further away than the issue claims.

New evidence: the byte term is ~2x over-weighted

Measured 2026-09-05, #139 arm, block 741000, 16 chunks, execute mode. Chunks 1 and 11 are a controlled pair — identical 42 inputs and 42 EC verifies, differing only in bytes (765,282 vs 37,933):

chunk  1 (fat)   10,039,314 cycles
chunk 11 (lean)   7,763,901 cycles     ratio 1.293x measured, 1.574x modelled

⇒ COST_PER_INPUT_BYTE (6) is ~2x the measured 3.13 cycles/byte. Least squares over all 16 chunks, intercept forced to zero because the packer cannot express one:

cycles = 106,177*ec + 2.62*bytes + 90,395*inputs      mean |error| 5.2%   (current constants: 13.1%)

An over-weighted byte term starves byte-heavy chunks of inputs and over-loads lean ones — which is precisely how a straggler sits at 1.438 while the predicted straggler looks healthy. The packer balances its own model perfectly, and that is what makes a wrong model invisible.

⛔ This is not Ghost’s answer yet. It was measured on the #139 arm, not the full Ghost arm, and block 741000 carries 2 Schnorr verifies out of 723 — so it cannot inform the curve split, which is the dimension this issue correctly identifies as the problem. It is corroboration that the model is wrong, not a fit you can ship.

⛔ Method warning for whoever runs the profile

An experiment arm built from env vars alone is silently stock. This cost three consecutive profile runs; only a symbol gate caught it. Three independent reasons:

1. The lever is a macro, but the code arrives with patches/0005, applied by provision-vps.sh — not cargo.
2. build.rs tracks secp translation units, not headers. Patch 0005 edits ecdsa_impl.h. Fix: touch ~/hazync-build/secp256k1/src/*.c.
3. rerun-if-env-changed cannot fire if the var was already set by a previous run — there is no change to detect.

Gate on an exact symbol match for hazync_ecmult_verify (per-arm symbols are in scripts/gpu-benchmark.sh WANT/DENY). A gate on a symbol no build defines rejects everything and reads exactly like a failed build.

Also relevant to sizing: the aggregate is 25% of Ghost’s budget and has since measured 405.6 s on two workers against the 627 s assumed in BUILDS.md §4.3 — possibly a second card, already paid for.

Full plan, with the closed-lever table and what would falsify it, lands as docs/GHOST_NEXT_BUILD.md on feat/liftx-accel.

[defenwycke]

Status against the four items, so this issue is not mistaken for finished.

1 · Packer calibration — ✅ MEASURED, ⛔ not shipped

Two L40S, block 962,000, 16 chunks, real GPU proving, one arm per box, differing only in the four HAZYNC_COST_* values:

arm	straggler	max	mean
shipped defaults	1.462x	149 s	102 s
refit	1.189x	124 s	104 s

This issue’s hypothesis was right: the liftx hint removes decompression from both curves, so the 13.77x ratio should have narrowed. Measured, it narrows to 1.97x. Fitted on block 965,500 (7,889 inputs, 7.7% Schnorr, per-chunk spread 0–40.5% — the first block on hand that can identify the split at all; 962,000 is 2.7% and 741000 carries 2 Schnorr verifies out of 723):

HAZYNC_COST_EC_OP=85636  HAZYNC_COST_SCHNORR_OP=168542
HAZYNC_COST_INPUT_BYTE=2 HAZYNC_COST_INPUT_BASE=53162

That clears the ≤1.35 threshold BUILDS.md §4.2 states for the fourth card. ⚠ I have not re-derived the card arithmetic — read it as "the straggler target is met", not as a card count.

⛔ It first measured 1.567x — worse than doing nothing — and that was reported. It was one chunk: 31 of 32 timings across both boxes sit at 2.4–3.0 s/segment; hz-b chunk 1 came in at 4.89 (171 s for 35 segments). Re-run twice: 94 s, 94 s. It never reproduced. Two hypotheses were refuted on the way — the segment straggler said the opposite (57 → 49 max segments), and "the boxes differ in speed" is true but irrelevant, since an identical 33-segment chunk is 87 s vs 90 s and a straggler is a ratio, so box speed cancels out entirely.

Correction to this issue’s own baseline: it quotes straggler 1.407, which BUILDS.md §4.2 flags as assumed and never measured for the backend arm. Measured is 1.438 (and 1.462 on this run).

Remaining: ship the constants. They are per-build-mode and Ghost deliberately ships defaults today, so this is a decision, not a copy-paste. Deliberately kept out of the v0.20.0 release.

2 · MSM — excluded by decision, not by evidence. Unchanged.

3 · Wholesale bigint2 — ⛔ the premise is wrong

This item says "15% is 15%". That 15% was never produced by a run. hazync_ecdsa_verify_full existed from 3615a8d (2026-08-28) and nothing ever called it — no patch on any branch added a call site until patches/0014 (see 9b767b5). It also likely double-counts, because wholesale subsumes the modular inversion that patches/0008 accelerates separately. The middle path’s 8.00x at proving time is measured; the wholesale figure is not.

4 · G3 / scalar-inverse — not re-measured

⚠ One correction here too: the scalar inverse’s 12.61% IS a real measurement (9b767b5, modular inversion). CORE_VS_GHOST.md’s 1.082x is a different quantity — its step in Ghost’s cumulative chain. Both are real; they measure different things. G3 has no standalone figure at all: it was measured together with the decompression memo (1.253x combined).

Full record: docs/GHOST_NEXT_BUILD.md on measure/ghost-calibration.