Decide a per-rule cutoff burn-down: the cutoff currently hides 3,867 violations

[tarekziade]

Reported by developers

Was there any reason for the cutoff date because it might be too gracious and hiding too much across models

For example autodocstrings should be at top level models but based on older models we skip too graciously

Opening this to answer the question with numbers and to make the burn-down a deliberate decision per rule rather than a standing one-date policy.

Why the cutoff exists

cutoff_date grandfathers models contributed before a convention was introduced, so a rule does not have to ship an allowlist of every model that predates it. Without it the alternative is a ~500-entry allowlist_models on most rules.

What it hides today

Two full runs over a transformers checkout of 2026-08-25 (510 models), all rules enabled: 429 findings as shipped, 4,296 with every cutoff_date set to 1900-01-01. So the cutoff currently hides 3,867 violations. Per rule:

rule	hidden	rule	hidden	rule	hidden
TRF041	1744	TRF037	118	TRF024	25
TRF045	735	TRF023	118	TRF028	17
TRF057	393	TRF036	115	TRF056	17
TRF027	149	TRF034	96	TRF030	10
TRF029	141	TRF019	66	TRF026	9
		TRF035	64	TRF042	5
		TRF033	38	TRF025	4

(TRF041 #52, TRF029 #53, TRF034 #54 and TRF035 #55 have separate false-positive issues open; their numbers here are before those fixes. The inheritance defect behind the "parent is grandfathered" complaint is #56.)

Proposal

cutoff_date is already per rule, so this does not need a mechanism change — it needs a decision per rule. For rules whose fix is mechanical and reviewable in bulk, run a one-time cleanup PR in transformers and then delete that rule's cutoff_date:

• TRF057 — @auto_docstring on public classes and methods, 393 findings. This is the one the feedback asks for, and adding the decorator is mechanical.
• TRF028 — license headers, 17 findings. Trivially mechanical.
• TRF045 — output_attentions / output_hidden_states / return_dict in forward, 735 findings. Larger and behavioural, so a candidate for staging rather than one sweep.

Rules whose remaining findings are judgement calls (TRF041, TRF036, TRF037) should keep the cutoff until their false-positive issues land, and be re-measured afterwards.

Context worth recording

transformers' repo-local utils/rules.toml has been accumulating model-wide allowlist entries to unblock new-model PRs — TRF018 has 16 repo-only entries (gemma4, kimi_k25, olmo3, radio, sam3_lite_text, sam3_tracker, voxtral_realtime, ...), TRF034 six, TRF019 six, TRF041 three, TRF016 two.

That is strictly worse than the cutoff: a cutoff exempts a model for code written before the convention existed, whereas an allowlist entry mutes the rule for that model forever, including code written tomorrow. It is also a good measure of how much friction the current defaults are causing model adders.

[tarekziade]

Re-measured with all six open PRs applied

I merged the six open false-positive PRs — #59 (TRF041), #60 (TRF029), #61 (TRF034), #62 (TRF035), #63 (inherited cutoff), #64 (--rules-toml cutoff override) — into one integration branch and re-ran the same test: a full run over the same transformers checkout (510 models, 81ec4fd), all rules enabled, once as shipped and once with every cutoff_date replaced by 1900-01-01.

	as shipped	cutoffs at epoch	hidden by the cutoff
main	429	4,296	3,867
all six PRs	426	3,369	2,943

The cutoff now hides 924 fewer violations, a 24% cut, without any burn-down work in transformers. The main row reproduces this issue's numbers exactly, per rule.

Per rule, hidden before → after (rules not listed are unchanged):

rule	before	after
TRF041	1744	904	−840
TRF035	64	10	−54
TRF029	141	124	−17
TRF034	96	83	−13
TRF045	735	735	—
TRF057	393	393	—
TRF027	149	149	—
TRF037	118	118	—
TRF023	118	118	—
TRF036	115	115	—
TRF019	66	66	—
TRF033	38	38	—
TRF024	25	25	—
TRF028	17	17	—
TRF056	17	17	—
TRF030	10	10	—
TRF026	9	9	—
TRF042	5	5	—
TRF025	4	4	—
TRF032	3	3	—

(TRF032's 3 are missing from the table in the issue body; everything else matches.)

In the shipped run, three findings go and none appear: TRF041 in axk2 and cohere_compass, TRF035 in step3p7.

Two things this measurement does not show

#63 contributes nothing here, by construction. It reads a rule's cutoff against the file that defines an inherited base class, so setting every cutoff to 1900-01-01 — which grandfathers nobody — switches it off. Measured the way that defect is actually visible (every model treated as newly added, parents keeping their real contribution dates) it removes 16 TRF034 findings. It also makes tipsv2_dpt's model-wide TRF034 allowlist entry unnecessary, and dropping that entry is why TRF034's number here falls by 13 rather than 16: in the epoch-cutoff run those three inherited DPT layers come back.

#64 is what makes the burn-down proposal executable at all. The plan in this issue is "run a cleanup PR, then delete that rule's cutoff_date". On main that deletion is a silent no-op through --rules-toml, which is the path transformers uses (utils/check_modeling_structure.py --rules-toml utils/rules.toml):

# main
$ mlinter --enable-all-trf-rules                                  # 429
$ mlinter --enable-all-trf-rules --rules-toml no-cutoff.toml      # 429   <- key deleted, no effect
$ mlinter --enable-all-trf-rules --rules-toml epoch-cutoff.toml   # 4296

# with the six PRs
$ mlinter --enable-all-trf-rules --rules-toml no-cutoff.toml      # 3369  <- key deleted, exemption gone
$ mlinter --enable-all-trf-rules --rules-toml epoch-cutoff.toml   # 3369

So before #64, deleting a cutoff date would have looked like a completed burn-down step while changing nothing.

What this does to the per-rule decision

• TRF035 — 10 left, from 64. This one has effectively burnt itself down: six bare # noqa, two F841 in vivit, two E712 in esmfold. Cheaper than TRF028 now, and worth adding to the same first sweep.
• TRF041 — 904, from 1744. Still the biggest number by far, but it is now made of genuine architecture forks (scale_embedding 50, use_weighted_layer_sum 22, auxiliary_loss 18, ...) rather than problem_type and hidden_act boilerplate. Each one wants a human to name the checkpoints, so it stays a judgement-call rule — but the size of the judgement is halved.
• TRF029 — 124, and TRF034 — 83. Both now target what they were written for; neither is trivially mechanical.
• TRF045 (735) and TRF057 (393) are untouched by these PRs and remain the two candidates the burn-down proposal turns on.

One note on TRF057, since the feedback quoted in this issue and in #56 points at it: I looked at whether the inheritance fix should apply to it and concluded it should not. Its modular findings are inherited in origin but local in fix — Glm4vForConditionalGeneration and its forward are written in modular_glm4v.py, and the rule already says to add the decorator there — so exempting them would drop the requirement from 42 brand-new public model classes and methods. That is the "we skip too graciously" side of the complaint rather than the mis-attribution side, which is what #63 fixes.