Repository navigation
fix(mappers): refuse the pre-norm TransformerLens block hooks; map blocks.N.hook_in and unembed.hook_in - #27
Merged
Merged
Conversation
… the block and unembed inputs
TransformerLens' block-level `hook_mlp_in` and `hook_attn_in` fire on the residual BEFORE the
layer norm, so resolving them to `mlp_in` / `attn_in` (the norm's output) returned the wrong
tensor, and `point_to_tlens_hook("attn_in")` emitted `hook_attn_in` for the same reason. Both
names are now refused with a message that names the TL3 bridge's `mlp.hook_in` / `attn.hook_in`
and the legacy `hook_resid_mid` / `hook_resid_pre`. `blocks.N.hook_in` maps to `resid_pre` on a
single-stream trunk and is refused on a hyper-connection one; `unembed.hook_in` maps to
`final_norm`; `ln_final.hook_normalized` is refused with the fold_ln reason.
The rows were a faithful copy of the TL3 bridge's own alias table, which TransformerLens fixed in
#1257 and #1319 while our copy stayed. The mapper had only round-trip name tests, so a gpt2 parity
test now walks every mapped hook against HookedTransformer and TransformerBridge and compares the
tensors, and a second test shows numerically that the block-level names are not the sublayer
inputs.
Validator: `attn_in` and `mlp_in` are scored on `tlens_v3` through `attn.hook_in` / `mlp.hook_in`
(120 and 132 PASS, no FAIL). The eager reference and tlens_v3 column were re-captured on 2x H200
for the sweep; no existing cell changed verdict. Docs, the visualizer's naming data and the
capability spec follow.
Fixes #24
Fixes #25
Co-authored-by: Daniel Dale <danny.dale@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
39 verified runs recorded on disk had not been published; `make viz-gpus-check` failed on main. Co-authored-by: Cursor <cursoragent@cursor.com>
…ared against TransformerLens 4.0.0 (released after v1.11.1) removed HookedTransformer, so the parity extra and the validator now resolve >=3.0,<4. CI installs the extra unlocked and picked 4.0 up; the port to the bridge-only API is separate work. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
TransformerLens 4.0.0 shipped after v1.11.1 and removed |
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #24 and fixes #25, both reported by @speediedan. The two commits on their branch (refusing the pre-norm block hooks; pinning the sublayer input names) are squashed into the first commit here with co-author credit.
What was wrong
TransformerLens' block-level
blocks.N.hook_mlp_inandhook_attn_infire on the residual before the layer norm.tlens_hook_to_pointresolved them tomlp_in/attn_in, which are the norm's output, so a caller translating those names got the wrong tensor.point_to_tlens_hook("attn_in")emittedhook_attn_infor the same reason. Separately,blocks.N.hook_inandunembed.hook_inwere unknown names and raised.The rows were a copy of the TL3
TransformerBridgealias table, which TransformerLens itself corrected in TransformerLensOrg/TransformerLens#1257 and #1319 (see #1317). Our copy stayed. The mapper had only round-trip name tests, so nothing compared tensors.Only the translation layer is affected. Canonical point names captured the right tensor before and after. None of the deployed Neuronpedia Python apps pass TransformerLens names through the mapper.
Changes
interp_engine/mappers.pyhook_mlp_in/hook_attn_inare refused with a message naming the TL3 bridge'smlp.hook_in/attn.hook_inand the legacyhook_resid_mid/hook_resid_pre.blocks.N.hook_in->resid_preon a single-stream trunk; refused on a hyper-connection trunk.unembed.hook_in->final_norm(global, takes no layer).ln_final.hook_normalizedis refused with the fold_ln reason.point_to_tlens_hook("attn_in")now emitsattn.hook_in.Tests
tests/test_parity_gpt2.py: a full-table test walks every mapped TL hook on gpt2, resolves it, captures with eager and compares tensors againstHookedTransformerandTransformerBridge. A second test shows numerically that the block-levelhook_mlp_in/hook_attn_inare notmlp_in/attn_in.tests/test_mappers.py: the new mappings and refusals.Validator
attn_inandmlp_inare scored ontlens_v3throughattn.hook_in/mlp.hook_in(spec.py,tlens_v3_engine.py, scoring test).tlens_v3column were re-captured on 2x H200 for the 36-model sweep (Kimi-K2.6 excepted). Result:attn_in120 PASS / 4 N/A,mlp_in132 PASS / 7 N/A, no FAIL. No existing cell changed verdict.Docs and visualizer
docs/ENGINE_HOOK_MAPPINGS.md: corrected rows and a section on why the block-level names are refused.visualizer-web/data/engines.ts: TL naming and caveats follow the mapper.Second commit, unrelated:
visualizer-web/data/gpus.generated.tsregenerated frominterp_engine.memory(39 verified runs had not been published;make viz-gpus-checkfailed onmain).Release
Patch bump (1.11.2). The refusals change behaviour for anyone passing
hook_mlp_in/hook_attn_inthrough the mapper, but the old result was the wrong tensor.