check_perlayer_mse: skip empty-input hook calls (Qwen3 MoE)

[Allenjin123]

Summary

Follow-up to #1. check_perlayer_mse.py crashes on Qwen3 MoE checkpoints because the SC linears inside experts that receive zero routed tokens in a given step are still called, with an input of shape (0, hidden). The hook's (out - ref).abs().max() then raises:

RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0.

F.mse_loss handles numel()==0 cleanly, but the unreduced .max() on an empty tensor is undefined and PyTorch refuses to guess. Dense Qwen3 never triggers this; the 30B-A3B Phase 2 run in the multi-model sweep was the only case that hit it.

Fix is an early-return from the hook when x.numel() == 0 — there is no SC error to record for an unused expert call.

Test plan

• Qwen/Qwen3-30B-A3B-Instruct-2507, sc_prec=8 stoc_len=256 on gl1810 (RTX PRO 6000 Blackwell) — completes without raising; 3669 SC matmul calls captured across 48 layers × 128 experts.
• Aggregated MSE table renders cleanly; down_proj dominates (mean MSE 2.3e-3, max 3.7e-1) as expected from the outlier-driven pattern.
• Top-10 worst single matmuls all concentrate in late-layer experts (layers.46/47.mlp.experts.*.down_proj), the MoE analog of the late-layer down_proj story already documented for dense Qwen3 / llama.

Sample output (Qwen3-30B-A3B-Instruct-2507, stoc_len=256)

Captured 3669 SC matmul calls. model=Qwen/Qwen3-30B-A3B-Instruct-2507  sc_prec=8 stoc_len=256

projection    count     mean MSE      max MSE   mean max|Δ|   max max|Δ|   mean |x|    max |x|
--------------------------------------------------------------------------------------------------------------
q_proj           48   3.5632e-04   4.5403e-03        0.0959       0.5391     0.0600    49.2812
k_proj           48   2.9245e-04   4.0065e-03        0.0736       0.3477     0.0600    49.2812
v_proj           48   6.5802e-04   9.0080e-03        0.0710       0.4674     0.0600    49.2812
o_proj           48   1.0434e-04   4.6457e-03        0.1454       5.5000     0.0228     8.4531
gate_proj      1159   4.5253e-04   6.5630e-03        0.0737       0.4766     0.3075   154.2500
up_proj        1159   4.6053e-04   6.5198e-03        0.0683       0.6562     0.3075   154.2500
down_proj      1159   2.3462e-03   3.6788e-01        0.1651      18.2500     0.2101   566.5000

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a crash in check_perlayer_mse.py for Qwen3 MoE models where experts that receive zero routed tokens produce empty-input hook calls, causing .abs().max() on a 0-element tensor to raise.

Changes:

• Early-return from the post-hook when x.numel() == 0 to skip empty expert calls.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[heroarmor]

Clean, targeted fix for a real Qwen3-MoE-only crash. Confirmed:

• Empty-input case is genuine for MoE experts with 0 routed tokens; .abs().max()/.mean() on numel()==0 tensors raises (PyTorch refuses to guess a default for unreduced ops). F.mse_loss handles it but is unused on that path.
• Early-return is the right semantic (no SC error to record for an unrouted expert), and the printed 'Captured N' count becomes meaningful instead of being inflated by no-op calls.
• Disjoint from PR #3 (PR #3 touches root check_perlayer_mse.py; this touches model_qwen4b/check_perlayer_mse.py).
• Test evidence: 3669 calls captured on Qwen3-30B-A3B with late-layer down_proj dominating MSE — matches the established outlier pattern from dense Qwen3 and llama.