Skip to content

fix(mappers): refuse the pre-norm TransformerLens block hooks; map blocks.N.hook_in and unembed.hook_in - #27

Merged
hijohnnylin merged 3 commits into
mainfrom
tlens-prenorm-hooks
Sep 22, 2026
Merged

hijohnnylin merged 3 commits into
mainfrom
tlens-prenorm-hooks

Conversation

@hijohnnylin

Copy link
Copy Markdown
Contributor

Fixes #24 and fixes #25, both reported by @speediedan. The two commits on their branch (refusing the pre-norm block hooks; pinning the sublayer input names) are squashed into the first commit here with co-author credit.

What was wrong

TransformerLens' block-level blocks.N.hook_mlp_in and hook_attn_in fire on the residual before the layer norm. tlens_hook_to_point resolved them to mlp_in / attn_in, which are the norm's output, so a caller translating those names got the wrong tensor. point_to_tlens_hook("attn_in") emitted hook_attn_in for the same reason. Separately, blocks.N.hook_in and unembed.hook_in were unknown names and raised.

The rows were a copy of the TL3 TransformerBridge alias table, which TransformerLens itself corrected in TransformerLensOrg/TransformerLens#1257 and #1319 (see #1317). Our copy stayed. The mapper had only round-trip name tests, so nothing compared tensors.

Only the translation layer is affected. Canonical point names captured the right tensor before and after. None of the deployed Neuronpedia Python apps pass TransformerLens names through the mapper.

Changes

interp_engine/mappers.py

  • hook_mlp_in / hook_attn_in are refused with a message naming the TL3 bridge's mlp.hook_in / attn.hook_in and the legacy hook_resid_mid / hook_resid_pre.
  • blocks.N.hook_in -> resid_pre on a single-stream trunk; refused on a hyper-connection trunk.
  • unembed.hook_in -> final_norm (global, takes no layer). ln_final.hook_normalized is refused with the fold_ln reason.
  • point_to_tlens_hook("attn_in") now emits attn.hook_in.

Tests

  • tests/test_parity_gpt2.py: a full-table test walks every mapped TL hook on gpt2, resolves it, captures with eager and compares tensors against HookedTransformer and TransformerBridge. A second test shows numerically that the block-level hook_mlp_in / hook_attn_in are not mlp_in / attn_in.
  • tests/test_mappers.py: the new mappings and refusals.

Validator

  • attn_in and mlp_in are scored on tlens_v3 through attn.hook_in / mlp.hook_in (spec.py, tlens_v3_engine.py, scoring test).
  • The eager reference and tlens_v3 column were re-captured on 2x H200 for the 36-model sweep (Kimi-K2.6 excepted). Result: attn_in 120 PASS / 4 N/A, mlp_in 132 PASS / 7 N/A, no FAIL. No existing cell changed verdict.

Docs and visualizer

  • docs/ENGINE_HOOK_MAPPINGS.md: corrected rows and a section on why the block-level names are refused.
  • visualizer-web/data/engines.ts: TL naming and caveats follow the mapper.

Second commit, unrelated: visualizer-web/data/gpus.generated.ts regenerated from interp_engine.memory (39 verified runs had not been published; make viz-gpus-check failed on main).

Release

Patch bump (1.11.2). The refusals change behaviour for anyone passing hook_mlp_in / hook_attn_in through the mapper, but the old result was the wrong tensor.

hijohnnylin and others added 3 commits September 22, 2026 16:00
… the block and unembed inputs

TransformerLens' block-level `hook_mlp_in` and `hook_attn_in` fire on the residual BEFORE the
layer norm, so resolving them to `mlp_in` / `attn_in` (the norm's output) returned the wrong
tensor, and `point_to_tlens_hook("attn_in")` emitted `hook_attn_in` for the same reason. Both
names are now refused with a message that names the TL3 bridge's `mlp.hook_in` / `attn.hook_in`
and the legacy `hook_resid_mid` / `hook_resid_pre`. `blocks.N.hook_in` maps to `resid_pre` on a
single-stream trunk and is refused on a hyper-connection one; `unembed.hook_in` maps to
`final_norm`; `ln_final.hook_normalized` is refused with the fold_ln reason.

The rows were a faithful copy of the TL3 bridge's own alias table, which TransformerLens fixed in
#1257 and #1319 while our copy stayed. The mapper had only round-trip name tests, so a gpt2 parity
test now walks every mapped hook against HookedTransformer and TransformerBridge and compares the
tensors, and a second test shows numerically that the block-level names are not the sublayer
inputs.

Validator: `attn_in` and `mlp_in` are scored on `tlens_v3` through `attn.hook_in` / `mlp.hook_in`
(120 and 132 PASS, no FAIL). The eager reference and tlens_v3 column were re-captured on 2x H200
for the sweep; no existing cell changed verdict. Docs, the visualizer's naming data and the
capability spec follow.

Fixes #24
Fixes #25

Co-authored-by: Daniel Dale <danny.dale@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
39 verified runs recorded on disk had not been published; `make viz-gpus-check` failed on main.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ared against

TransformerLens 4.0.0 (released after v1.11.1) removed HookedTransformer, so the parity extra
and the validator now resolve >=3.0,<4. CI installs the extra unlocked and picked 4.0 up; the
port to the bridge-only API is separate work.

Co-authored-by: Cursor <cursoragent@cursor.com>
@hijohnnylin

Copy link
Copy Markdown
Contributor Author

TransformerLens 4.0.0 shipped after v1.11.1 and removed HookedTransformer; CI installs the parity extra unlocked and picked it up, which failed the cpu job's import gate. 5a7a5f7 pins transformer-lens>=3.0,<4 in the parity extra and the validator (both lockfiles refreshed). Porting the gate to TL 4's bridge-only API is separate work.

@hijohnnylin
hijohnnylin merged commit 63e28d5 into main Sep 22, 2026
10 checks passed
@hijohnnylin
hijohnnylin deleted the tlens-prenorm-hooks branch September 22, 2026 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant