Does T5 actually utilize Encoder prefixes in PrefixTuning? past_key_values seems ignored by the Encoder.

[david900125]

System Info

• PEFT version: 0.18.0
• Transformers version: 4.57.3
• Model: T5 / Flan-T5

Description

I am exploring Prefix-Tuning on a Seq2Seq model (T5/Flan-T5) and I noticed a potential discrepancy between the paper's implementation and the current codebase regarding the Encoder's prefix.

According to the [Prefix-Tuning paper](https://arxiv.org/abs/2101.00190), prefixes should be prepended to both the Encoder and the Decoder for Encoder-Decoder architectures.

I configured my PEFT model with num_transformer_submodules=2 to generate prompts for both components.

However, tracing the code:

1. In peft/src/peft/peft_model.py (PeftModelForSeq2SeqLM):
   The code generates the prompt and passes it into kwargs["past_key_values"]:

if peft_config.peft_type == PeftType.PREFIX_TUNING:
    # overwrite past_kv in kwargs
    kwargs["past_key_values"] = self.get_prompt(batch_size)
    return self.base_model(
        input_ids=input_ids,
        ...
        **kwargs, # past_key_values is inside here
    )

2. In transformers/models/t5/modeling_t5.py (T5Model):
   The forward method separates the execution flow. It calls self.encoder without passing past_key_values.
   Reference Line: [modeling_t5.py#L991 (v5.0.0rc1)](https://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/t5/modeling_t5.py#L991)

encoder_outputs = self.encoder(
    input_ids=input_ids,
    attention_mask=attention_mask,
    inputs_embeds=inputs_embeds,
    output_attentions=output_attentions,
    output_hidden_states=output_hidden_states,
    return_dict=return_dict,

)

Question

It seems that even if num_transformer_submodules=2 is set, the past_key_values intended for the Encoder are effectively discarded before reaching the encoder's attention layers.

1. Is my understanding correct that currently, PrefixTuning on T5 effectively only works on the Decoder?
2. Is there an internal mechanism in transformers that automatically handles this injection that I might have missed?

Any clarification would be appreciated. Thanks!

[BenjaminBossan]

Thanks for opening this discussion. Your observation is correct, in encoder-decoder models, prefix tuning does not affect the encoder. The kv cache (aka past_key_values) is not applied to the encoder since it is not "causal", but prefix tuning relies on the kv cache to inject its embeddings.

I have asked internally if there is a different way we could adapt the encoder, but possibly there is no solution. I'll update once I have an answer.

[BenjaminBossan]

@david900125 After discussion with my colleague, who checked whether something can be done on transformers to enable past_key_values in the encoder part, she said there is no easy solution. Basically, we would have to go through each and every encoder-decoder model architecture and adjust it. This is too much work for a rather niche application, so we decided not to support it.

This means that encoder-decoder models don't work with prefix tuning as described in the paper. We should document this in PEFT, LMK if you want to create a PR for that, otherwise I'll take it. Note, though, that we can still use prefix tuning with encoder-decoder models and it can still be fine-tuned, it just doesn't conform to the paper description.

[david900125]

Hi @BenjaminBossan,

Thank you so much for the thorough investigation and confirmation. It is very helpful to have a definitive answer on this behavior.

I completely understand that refactoring the encoder-decoder architectures in transformers is a massive undertaking and out of scope. Clarifying this in the documentation is definitely the right move to prevent confusion for future users.

I will leave the documentation PR to you, as you likely have a standard format for such warnings.
Thanks again for your help!

[BenjaminBossan]

PR is in the works: #2989. Edit: merged.

[SahilKumar75]

Your suspicion is correct the T5 encoder does not actually use the prefix past_key_values that PEFT generates. This is a known architectural gap that has been documented across several issues and PRs.

Why the encoder ignores past_key_values

In HuggingFace's T5 implementation, past_key_values is designed for the decoder only. The encoder runs a single forward pass at the start of generation and produces hidden states that the decoder attends to via cross-attention. There is no generation loop for the encoder, so KV caching was never implemented there. When PEFT calls get_prompt() and passes the result as past_key_values into the seq2seq model, T5's internal routing only forwards those values into the decoder. The encoder receives the parameter but doesn't concatenate it to its self-attention keys and values.

This was explicitly confirmed in transformers issue #15591 where the original reporter had the exact same use case (Li & Liang 2021 prefix tuning) and a maintainer confirmed the encoder cannot use past_key_values without modifying T5Attention to concatenate them manually.

How this deviates from the original paper

The Li & Liang (2021) prefix tuning paper (figure 2) shows both encoder and decoder receiving separate trainable prefixes. PEFT's implementation for SEQ_2_SEQ_LM only applies the prefix to the decoder — this was documented in peft issue #1188. The implementation is closer to P-tuning v2 than the original prefix tuning paper.

Current state

There have been active PRs to address this (transformers PR #34312, peft PR #2096) but the conclusion from the PEFT team was that encoder-decoder models work fundamentally differently for prefix tuning and the fix needs to live on the PEFT side rather than transformers. The transformers PR was closed with a note that a PEFT-side fix was found instead.

Practical implication

If you are training with PrefixTuningConfig(task_type=TaskType.SEQ_2_SEQ_LM), the encoder sees no prefix at all — only the decoder is getting the virtual tokens prepended to its attention. So if your task relies heavily on encoder-side conditioning (e.g., summarization, translation where input understanding matters more than generation style), the prefix is doing less than you expect.

Workarounds

If you genuinely need encoder side prefixes, the options are:

1. Prepend virtual token embeddings directly to the encoder input before the first layer (not via past_key_values). This is what some custom implementations like UnifiedSKG do they modify the encoder hidden states directly rather than using the KV cache mechanism.

2. Use modules_to_save or LoRA on the encoder layers instead of prefix tuning if you want the encoder to be adapted.

3. If your goal is prompt style conditioning, prepend real token ids (soft prompt tokens from the vocabulary) to the input sequence this works natively with T5's encoder without any architectural changes.

So to directly answer your question: no, the encoder prefixes are not being utilized in PEFT's T5 prefix tuning. The past_key_values passed to the model go to the decoder only.