feat(mp): per-row mixed-precision wiring + Qwen3 PPL sweep by Allenjin123 · Pull Request #6 · CrucibleComputingGroup/scmp_llm

Allenjin123 · 2026-05-26T19:55:04Z

Summary

Wires scmp_kernels.mp.MPConfig (fixed-fraction quantile) into SCLinear.forward and sc_eager_attention_forward. Per-row dispatch on token rows (Linear) and per-(B, H) query rows (attention), with the global sc_stoc_len as fallback when no MP config is attached. Effective avg stoc_len is tracked and printed alongside PPL.
Adds env loader apply_mp_config_from_env (MP_CONFIG_JSON=...) and three reference configs mp_{a,m,c}.json (avg_sl ≈ 67 / 96 / 112).
tests/run_mp_sweep.sh is a single reproducer: defaults regenerate the overnight 2026-05-26 5-model × 3-MP sweep; flags narrow models / MP / context / token cap / SQ / halve / etc. Header has Great Lakes salloc + tmux instructions.
benchmark/ppl/_mp_overnight*.sh + _env_common.sh: per-node driver, 5-node fan-out, and SUMMARY.txt aggregator that the reproducer chains to.
Bumps the kernels submodule to a576b83 (PR #16 Owen-in-rescale) which the MP path relies on.
Commits the previously-uncommitted SmoothQuant glue (model/smoothquant_apply.py, benchmark/ppl/calibrate_smoothquant.py) that ppl.py already imports — without these, the existing ppl.py on main cannot run.
MP.md documents invocation, JSON schema, implementation notes, and the two known limitations (AdaptiveMPConfig calibrated-thresholds path not wired; attention MP scales with B·H kernel launches).

Headline result

Halved (sl_max=128) + SQ@0.5 + per_row attention, 65 k tokens, sc_prec=8:

model	MP_c avg=112	MP_m avg=96	MP_a avg=67	FP16
4B	×1.097 (12.37)	×1.204 (13.57)	×2.842 (32.0)	11.27
8B	×1.105 (12.27)	×1.167 (12.96)	×1.690 (18.8)	11.10
14B	×0.999 (9.84)	×1.036 (10.20)	×1.291 (12.72)	9.85
30B-A3B	×1.096 (8.79)	×1.161 (9.32)	×1.614 (12.95)	8.02
32B	×1.014 (8.83)	×1.042 (9.07)	×1.228 (10.70)	8.71

MP_c (~12% cycle savings) is essentially lossless on ≥14B models; 14B at ×0.999, 32B at ×1.014. MP_m (~25% savings) is the sweet spot on big models. MP_a (~48% savings) collapses 4B quality.

Raw numbers: benchmark/ppl/_mp_overnight_20260526_040536/SUMMARY.txt.

Excluded from this PR

act_scales_Qwen_Qwen3-*.pt (≈150 MB total) — calibration artifacts, regenerable via calibrate_smoothquant.py.
Older one-off orchestrators (_alpha_sweep.sh, _moe_legacy.sh, _halved_fix_4b.sh, _overnight_*.sh) and their run-dirs — pre-existing, unrelated to MP.

Test plan

MP_CONFIG_JSON round-trip: python -c "from loader import apply_mp_config_from_env; ..." loads mp_m.json cleanly.
Smoke run: tests/run_mp_sweep.sh --models 4B --mp mp_m --max-tokens 4096 finishes < 5 min with finite PPL.
Smoke run with MP_CONFIG_JSON unset → falls back to fixed sc_stoc_len=128 (no per-row dispatch).
Verify model/smoothquant_apply.py / calibrate_smoothquant.py were genuinely orphan-untracked before this PR (git log --all --oneline -- model/smoothquant_apply.py returns empty on main).
Confirm the bumped kernels submodule SHA matches the PR #16 merge on the kernel repo.

🤖 Generated with Claude Code

Wires scmp_kernels.mp.MPConfig (fixed-fraction quantile) into the SC core so each token row in Linear and each query row in attention can use a different stoc_len. MPConfig is loaded from an MP_CONFIG_JSON env var and attached to model.config.sc_mp_config; both SCLinear.forward and sc_eager_attention_forward dispatch per row when present, falling back to the global sc_stoc_len when not. - model/sc_common.py: per-row classification + per-level sc_matmul dispatch in SCLinear; per-(B,H) loop in attention; weighted-mean effective stoc_len tracker (mp_tracker_*). - loader.py: apply_mp_config_from_env() reads MP_CONFIG_JSON and instantiates MPConfig. AdaptiveMPConfig is recognised in scmp_kernels.mp but the JSON loader rejects it for now (calibrated thresholds not wired yet). - benchmark/ppl/ppl.py: invoke MP loader, reset the tracker per SC run, and print avg_sl alongside the PPL row. - benchmark/ppl/mp_{a,m,c}.json: three reference MP configs (avg_sl ≈ 67 / 96 / 112). - tests/run_mp_sweep.sh: single-script reproducer; defaults reproduce the overnight 2026-05-26 result, with flags to narrow models/MP/ctx/tokens. - benchmark/ppl/_mp_overnight*.sh + _env_common.sh: per-node driver, 5-node fan-out, and SUMMARY.txt aggregator that the reproducer chains to. - MP.md: invocation guide + implementation notes + known limitations (no AdaptiveMPConfig path; attention MP scales with B·H). - kernels submodule: bump 59531ea -> a576b83 to pull in PR #16 (Owen-in-rescale) which the MP path relies on. - model/smoothquant_apply.py + benchmark/ppl/calibrate_smoothquant.py: add the previously-uncommitted SmoothQuant glue that ppl.py imports. Headline result (halved sl_max=128 + SQ@0.5 + per_row attention, 65k tokens): model | MP_c (avg=112) | MP_m (avg=96) | MP_a (avg=67) | FP16 4B | x1.097 | x1.204 | x2.842 | 11.27 8B | x1.105 | x1.167 | x1.690 | 11.10 14B | x0.999 | x1.036 | x1.291 | 9.85 30B-A3B | x1.096 | x1.161 | x1.614 | 8.02 32B | x1.014 | x1.042 | x1.228 | 8.71 MP_c (~12% cycle savings) is essentially lossless on >=14B models. MP_m (~25% savings) is the sweet spot for big models. MP_a (~48% savings) collapses 4B quality and should not be used. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR integrates per-row mixed-precision (MP) dispatch into the SC compute paths (Linear + eager attention), adds env-driven MP config loading, and introduces scripts/docs to reproduce and summarize a Qwen3 perplexity sweep (including SmoothQuant calibration/apply glue that benchmark/ppl/ppl.py relies on).

Changes:

Wire MPConfig-based per-row dispatch into SCLinear.forward and sc_eager_attention_forward, plus MP effective-stoc_len tracking for reporting.
Add SmoothQuant calibration + application utilities, and hook them into the PPL benchmark.
Add reproducer/driver scripts, JSON configs, and documentation for the MP PPL sweep workflow.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`tests/run_mp_sweep.sh`	Single-node reproducer script for the 5-model × 3-MP PPL sweep (plus calibration and summary steps).
`MP.md`	Documents MP invocation, JSON schema, and implementation/limitations.
`model/smoothquant_apply.py`	Implements SmoothQuant activation-scale calibration and per-layer scale attachment to `SCLinear`.
`model/sc_common.py`	Adds MP import wiring, per-row dispatch in `SCLinear` and attention matmuls, and MP stoc-len tracking helpers.
`loader.py`	Adds env parsing for `SC_HALVE_BIPOLAR_STOC_LEN` and `MP_CONFIG_JSON` → `model.config.sc_mp_config`.
`benchmark/ppl/ppl.py`	Applies SmoothQuant/MP from env, resets MP tracker per run, and prints avg effective `stoc_len` for MP.
`benchmark/ppl/mp_a.json`	Reference MP config (more aggressive avg `stoc_len`).
`benchmark/ppl/mp_m.json`	Reference MP config (mid avg `stoc_len`).
`benchmark/ppl/mp_c.json`	Reference MP config (conservative avg `stoc_len`).
`benchmark/ppl/calibrate_smoothquant.py`	CLI script to generate per-model activation scales for SmoothQuant.
`benchmark/ppl/_mp_overnight.sh`	Per-node driver for running all MP configs serially for a single model.
`benchmark/ppl/_mp_overnight_summary.sh`	Aggregates per-(model, MP) logs into a summary table.
`benchmark/ppl/_mp_overnight_dispatch.sh`	Great Lakes multi-node dispatcher for fanning the sweep across nodes.
`benchmark/ppl/_env_common.sh`	Common env setup sourced by the overnight scripts.
`benchmark/ppl/_mp_overnight_20260526_040536/SUMMARY.txt`	Checked-in reference summary output from the overnight run.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    """Process-wide accumulator for effective per-row stoc_len.
+
+    Set ``model.config.sc_mp_track = True`` to record. Use
+    ``mp_tracker_snapshot()`` to read and reset between sweep runs.


+def _record_assignment(assignment) -> None:
+    """Accumulate weighted-sum of per-row stoc_len for reporting.
+
+    Each level's stoc_len is the *effective* cycle count seen by the kernel
+    (MP levels are already in the halved space when halve_bipolar_stoc_len
+    is on — they're just sliced from [0, 2**(sc_prec-1)]).
+    """
+    state = _mp_tracker()
+    for sl, idxs in assignment.level_row_indices.items():
+        n = int(idxs.numel())
+        if n == 0:
+            continue
+        state["weighted_sl"] += float(sl) * n
+        state["rows"] += n


+# Aggregate per-(model, MP) log lines into SUMMARY.txt.
+# Run after all .done markers land — or anytime as a partial summary.
+set -e
+OUTDIR="${1:-$(cat /home/allenjin/Projects/scmp_llm/benchmark/ppl/_mp_overnight_latest.path 2>/dev/null | sed 's/^out=//')}"


+conda activate annstention
+
+# Turbo for read-mostly HF cache (10x faster cold reads vs GPFS).
+export HF_HOME=/nfs/turbo/coe-nbleier/allenjin/hf_cache
+export TRANSFORMERS_CACHE=$HF_HOME
+export HF_DATASETS_CACHE=/nfs/turbo/coe-nbleier/allenjin/hf_datasets
+
+# Scratch for write-heavy temp (pip, builds, GPU compile cache).
+mkdir -p /scratch/nbleier_owned_root/nbleier_owned1/shared_data/allenjin/tmp
+export TMPDIR=/scratch/nbleier_owned_root/nbleier_owned1/shared_data/allenjin/tmp
+
+# SC kernel knobs locked in for the MP sweep.
+unset SC_DISABLE_OWEN          # Owen-in-rescale scramble must be enabled.
+export SC_SCRAMBLE_RESCALE=1   # PR #16 path: scramble in rescale, not RNG.
+
+cd /home/allenjin/Projects/scmp_llm


+unset SC_DISABLE_OWEN          # Owen-in-rescale scramble must be enabled.
+export SC_SCRAMBLE_RESCALE=1   # PR #16 path: scramble in rescale, not RNG.
+
+cd /home/allenjin/Projects/scmp_llm


+launch_one gl1802 50893492 sweep_4b   Qwen/Qwen3-4B-Instruct-2507
+launch_one gl1803 50710402 sweep_8b   Qwen/Qwen3-8B
+launch_one gl1806 50708292 sweep_30b  Qwen/Qwen3-30B-A3B-Instruct-2507
+launch_one gl1806 50893493 sweep_14b  Qwen/Qwen3-14B
+
+# 32B waits for its SmoothQuant calibration to land act_scales on disk.
+SQ32_PATH="$HERE/act_scales_Qwen_Qwen3-32B.pt"
+launch_one gl1809 50712265 sweep_32b  Qwen/Qwen3-32B "$SQ32_PATH"


+mkdir -p /scratch/nbleier_owned_root/nbleier_owned1/shared_data/allenjin/tmp \
+  || true
+export TMPDIR=/scratch/nbleier_owned_root/nbleier_owned1/shared_data/allenjin/tmp


+    base=$(basename "$log" .log)
+    model="${base%_mp_*}"
+    mp="${base##*_}"
+    fp16=$(grep -E "^FP16 baseline " "$log" | awk '{print $3}' | tail -1)
+    line=$(grep -E "^SC MP " "$log" | tail -1)
+    if [[ -n "$line" ]]; then
+      ppl=$(echo "$line" | awk '{
+        for (i=1;i<=NF;i++) if ($i ~ /^[0-9]+(\.[0-9]+)?$/) {print $i; exit}
+      }')
+      ratio=$(echo "$line" | grep -oE '×[0-9.]+ vs fp16' | tr -d '×' | awk '{print $1}')
+      avg=$(echo "$line" | grep -oE 'avg_sl=[0-9.]+' | sed 's/avg_sl=//')
+      printf "%-44s  %-8s  %10s  %10s  %10s  %8s  %s\n" \
+        "$model" "$mp" "${fp16:--}" "${ppl:--}" "${ratio:--}" "${avg:--}" "$(basename "$log")"
+    elif grep -qE "FAIL|Traceback" "$log"; then
+      printf "%-44s  %-8s  %10s  %10s  %10s  %8s  %s\n" \
+        "$model" "$mp" "${fp16:--}" "FAIL" "-" "-" "$(basename "$log")"
+    else
+      printf "%-44s  %-8s  %10s  %10s  %10s  %8s  %s\n" \
+        "$model" "$mp" "${fp16:--}" "RUNNING" "-" "-" "$(basename "$log")"


heroarmor · 2026-05-26T21:20:52Z

LGTM

heroarmor

LGTM

Bring the branch up to date with main (PR #6 squash, docs/readme #8, bipolar-halving + offload-safe SC replacement d134e4a). Conflict resolution (all verified by line-level diff, not assumption): - model/sc_common.py, loader.py, benchmark/ppl/ppl.py: kept this branch's versions — each is a verified superset of main (adds AdaptiveMPConfig / calibrated-table + STE + cross-layer support on top of main's MPConfig). - tests/run_mp_sweep.sh: kept this branch's 531-line cross-layer rewrite (0d035b3); main's 216-line copy is the older same-lineage PR-#6 script, no independent feature lost. - kernels submodule: fc8d756 -> 41c16be (drop-formula-path branch: the matmul.py import hotfix + AdaptiveMPConfig formula-path removal). - Re-dropped benchmark/ppl/_mp_overnight_20260526_040536/SUMMARY.txt that the merge re-introduced (honors 5870a65). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Allenjin123 requested review from Copilot and heroarmor May 26, 2026 19:59

Copilot started reviewing on behalf of Allenjin123 May 26, 2026 20:00 View session

Copilot AI reviewed May 26, 2026

View reviewed changes

heroarmor approved these changes May 26, 2026

View reviewed changes

heroarmor merged commit 6ea08ef into main May 26, 2026
1 check passed

Allenjin123 mentioned this pull request Jun 3, 2026

mp: cross-layer SC mixed-precision wiring + merge main #9

Open

Navigation Menu

Uh oh!

feat(mp): per-row mixed-precision wiring + Qwen3 PPL sweep - #6

feat(mp): per-row mixed-precision wiring + Qwen3 PPL sweep#6
heroarmor merged 1 commit into
mainfrom
feat/mp-config-wiring

Allenjin123 commented May 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

heroarmor commented May 26, 2026

Uh oh!

heroarmor left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Allenjin123 commented May 26, 2026

Summary

Headline result

Excluded from this PR

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

heroarmor commented May 26, 2026

Uh oh!

heroarmor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants