[Rui Ren (rui-ren)]

Dependency and scope

Draft correctness follow-up to #2890, based on its exact head
f1a3c6c79532bc2db8b809ce99d1357a453bd722 through the existing upstream mirror
rui-ren/bolt-stack-20261104/06-drafter-graph-replay.
The original #2867 -> #2868 -> #2890 branches and PRs are unchanged.
This is not the separate-prefill change in closed #2891.

Published source is the already-qualified commit
c0fd01914265d5157d6da1376c38b4df521a76c5: four files, including documentation
and regression tests. No source was changed or rebased for publication.
This is a fork-based dependent PR; no native stack registration is claimed.

Issue

The pinned native H128 Flash attention route partitions split-K work using the
padded endpoint of each fixed-R8 call. A speculative block crossing a 128-key
arithmetic boundary can therefore use different BF16 attention arithmetic from
target-only row-zero decoding, despite identical queries and valid-prefix KV.
Small numerical differences can change predictions and break ordinary/k7 token
identity. Changing acceptance logic alone does not address the source.

Change

• Segment verification at that arithmetic boundary while continuing to execute
  the native fixed-R8 graph.
• Preserve and merge the generator's own complete recurrent snapshots, logits,
  prediction IDs and auxiliary rows on device before acceptance. Do not copy
  ordinary/reference state or change attention kernels, weight precision,
  padding policy or cache ownership.
• Count both actual target passes for a segmented round; retain one proposal's
  acceptance/round accounting.
• Leave prefill and ordinary decoding unchanged.
• Add an arithmetic-boundary unit regression, identical-candidate replay and
  independently advancing ordinary/k7 live-state/reset regressions.
• Align initializer-sharing assertions with actual configured source-region,
  file identity, dtype, shape and device keys, including GenAI path strings.
  Those follow-up changes affect tests, not inference.

Validation and provenance

• The four committed file hashes were rechecked against the preserved production
  build receipt before publication.
• Fresh publication-time native regression:
  NemotronBolt.VerificationKeepsQueryEndpointsInOneKeyTile: 1 passed,
  zero skips. Git whitespace checks passed.
• This exact GenAI source was held constant in the completed H100 comparison
  of the native/private ORT package and [CUDA] Consolidate source-exact BF16 decoding into existing contrib operators onnxruntime#33940's equivalent
  public package. Two reversed-order passes over the frozen 30 real prompts
  checked 240 complete measured generations. Every complete generated
  sequence matched the frozen quantized ordinary-control sequence, and complete
  speculative counter dictionaries matched across compared runs.
• The same comparison passed complete packed-value fidelity, 1,782 full
  tensor/state comparisons per variant, and the full 512-token LCB145
  ordinary/k7 probe. The public ORT package required no additional GenAI change.
• GenAI shared-library SHA256:
  5859c053086adc11578bbfb618dfeb84d01447f570fea4bd99a9fe5acbd865b0.
• GenAI CUDA-library SHA256:
  eb33580c8eca54cfe37f5da62539a31a2dcb5f677175aefdb55ff480d29d3469.
• Accepted real30 statistics SHA256:
  76d4aa528bc00aa0264c44415551a1df4f63147679ea853569f0f97cb0906cbd.

The throughput difference in the ORT comparison is not a speedup caused by
this GenAI fix: both arms used this same source and binaries. Segmentation can
require an extra real target pass; correctness and truthful counters take
precedence over historical non-lossless k7 timings.

Limits

This is qualification of the pinned Bolt/H128 route and frozen quantized
control, not a generic attention-backend guarantee, new independent HuggingFace
accuracy qualification, full benchmark-suite accuracy rerun or main-branch CI
acceptance. The build was incremental with recorded inputs, not a fresh full
build from main. The new GPU fixtures require explicit model/token fixtures;
the publication-time one-test CPU run is not presented as executing them.
Keep this PR draft pending normal review and CI.

[Rui Ren (rui-ren)]

Qualified opt-in ordinary decode: about 27% faster

Updated the actual stack33967 tip
in-place, without changing its seven-member order, lower heads, PR bases, or draft status.
Companion runtime: #2916.

The ordinary decoder was computing eight target rows while committing only the first.
The opt-in package keeps R8 prefill and R1 ordinary decode in one session, with shared
backing buffers and separate captured row geometries. Attention remains physically R8
to preserve Flash/cache arithmetic; other target work runs only the logical rows.

R1 also exposed a real Mamba inconsistency: its readout reduced two partials while R8
reduced four. The native correction retains four-part readout for multi-slot replay.
The new nonzero-state regression fails the old kernel and passes the corrected one.
Original R8 and single-slot R1 dispatch are unchanged.

Unprofiled frozen real30, six H100s, two reversed same-GPU passes, 240 complete measured
generations; original warmups, mode order, EOS/512 cap.

Actual input tokens	Before ordinary tok/s	Hybrid ordinary tok/s	Gain	Original R8/k7 tok/s
178-254	94.40	119.95	+27.1%	361.46 → 361.57
427-932	94.01	119.59	+27.2%	298.68 → 298.94
1028-1663	93.89	118.90	+26.6%	324.25 → 324.31

Ordinary TTFT medians: short 357.35→359.00ms, medium 826.97→831.29ms,
long 2153.15→2161.92ms. The earlier static-R1 version's prefill regression is not
the package being qualified here.

Separate same-GPU synthetic-2048 CUDA profiles, 64 captured target passes:
IQ2 expert work 3.259→1.496ms/pass, Mamba 0.824→0.423ms/pass.
These are device-work attribution, not an additional throughput claim.

Correctness: all240 exact same-mode sequences and complete counters match;
388 complete live committed-state boundaries (66.43GB compared), capture off/on,
two resets, prefix through130; original R8 ordinary/k7 retains1782 complete finite
tensor hashes. Three native kernel tests, seven Mamba operator tests, fifteen
GenAI tests execute without skips. The new replay test also passes CUDA memcheck
with zero errors (100 repetitions).

Scope: hybrid is ordinary-only. Both speculative arms use the original R8 package.
The benchmark explicitly keeps two per-mode models resident in each arm; it is not
a unified hybrid-speculative package or a k7 speedup. No precision/cache/mask workaround,
no weight repack, no independent HF rerun, no clean-build/full-CI claim.

Reproduce the exact tested graph/config with:

python examples/python/nemotron-bolt-hybrid.py --input <packed-r8> --output <fresh-same-filesystem-hybrid>

Requires onnx-ir/numpy/ml-dtypes and both updated runtimes. External-data validation
stays enabled; payloads are hardlinked inside the output package, not copied.

Pins:

• ORT: 0357a7dfe47d3c383bf36be62015162b6a612d5e
• Compiled/common GenAI: d73a6e627d6d20a5a1688b9fbd46237069a7be89
• Published GenAI: db6ffd29fc96e319f645ff461efa05c643235f29 (converter/docs-only child)
• CUDA provider: fac33c75d0336236736aa96456268057b628d93d0b9901d49598e275bf335275
• GenAI shared library: f977660a408bc0a01a31151c7280282ba443fa256785680ae3e841654eec9be7
• Tested hybrid graph: aaf5d6ae06b6740ab5b3af523359581930d2b6c42c38a18529a22737c2cfe530
• Config: 77f2d920d2b04e17884c3d2ee691d9053225b957c047454ebd0a4ea3649dac44
• Unchanged packed payload: 06d3d508e8d11235d0b770484b6b59764c2f297c3383b6d0de75648d26c11867

Full commands, per-case exact tokens/counters, raw kernel traces, provider maps,
UUID/admission receipts, and rejection evidence remain on the validation host under
/opt/bolt-runtime-fixes-20261108-v1/evidence/ordinary-r1-v1. Sealed manifest SHA256: 412b3898f09565557b01e82906358f017f9df3227db2bf330850245a52d32bbc.
Root-disk binaries were reloaded and verified with full ordinary/k7 generation;
the large model payload still resides on the existing tmpfs, not reboot-persistent.
Original package/deployment untouched; no merge.