Repository navigation
Gemma 4 assistant-head speculative decoding (MTP) #2439
Michael Burch (michaelburch)
started this conversation in
New features / APIs
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Add support for MTP with Gemma 4.
Create a branch that adds assistant-head speculative decoding for Gemma 4, reusing the existing MTP path. Tested end-to-end, lossless vs greedy. 4 commits:
Extract shared MTP generator base — refactor, no behavior change. Splits MtpGenerator into MtpGeneratorInterface + MtpGeneratorBase so a second generator can share the plumbing. Independently useful.
Add Gemma 4 assistant-head speculative decoding — new Gemma4AssistantGenerator, config schema (max_logits_sequence_length, shared_kv_layers, main_inputs_embeds), plus logits/hidden_states/multi_modal plumbing.
Export script + docs — python.
Simplify Logits::Update — complexity cleanup on code touched by (2).
Questions:
Export path. builder.py has no Gemma 4 support — it stops at Gemma3ForConditionalGeneration. My export lives in python and uses an external exporter. Should it go in builder.py instead?
Direction. README lists speculative decoding as roadmap. Is an architecture-specific assistant head the intended shape, or is a general mechanism planned?
Config schema. The three new genai_config.json fields are public surface — naming and placement acceptable?
Shared code. Changes to logits/hidden_states/multi_modal affect all models. Prefer those split into their own PR?
Happy to land (1) alone first if the rest needs iteration.
All reactions