[Pierluigi Urru (stwykd)]

Summary

Adds a config-driven llava model type. It runs the existing vision -> embedding -> decoder pipeline
(MultiModalLanguageModel / MultiModalPipelineState) for VLMs whose fixed-resolution vision encoder
emits a fixed number of features per image, and those features replace an image placeholder token that is
expanded to that many positions (LLaVA-style). There's no image_sizes, no HD crops and no mRoPE; the
decoder is a regular inputs_embeds decoder with past/present KV (e.g. GroupQueryAttention).

Today such an export has no type to use: gemma3 hardcodes its own image tokens and \n\n framing,
phi3v needs image_sizes/num_img_tokens from Phi3ImageTransform, and the Qwen-VL family assumes mRoPE.

Config

"model": {
  "type": "llava",
  "image_token_id": 32000,
  "vision": {
    "filename": "vision.onnx",
    "config_filename": "processor_config.json",
    "num_visual_tokens": 576,
    "inputs": { "pixel_values": "pixel_values" },
    "outputs": { "image_features": "image_features" }
  },
  "embedding": { "filename": "embedding.onnx", "inputs": { "input_ids": "input_ids", "image_features": "image_features" }, "outputs": { "inputs_embeds": "inputs_embeds" } },
  "decoder": { "...": "as for any inputs_embeds decoder" }
}

• The prompt holds one image_token_id token per image. The processor tokenizes the prompt, checks that
  count against the number of images and expands each token to vision.num_visual_tokens copies.
  num_visual_tokens (an existing field) is optional: by default it's read from the static token
  dimension of the vision output [num_images, num_tokens, hidden]. If both are set they must agree.
• Image preprocessing is the ort-extensions pipeline in vision.config_filename, which must produce float32
  [N, C, H, W]. The processor checks that, and the channel, height and width against the vision input's static
  dimensions, with an error pointing at the processor config (a pipeline left uint8 used to read past the buffer).
  Pixels are cast to the vision input type when it's float32, float16 or bfloat16.
• The embedding model gets the expanded input_ids and the image features. It puts the feature rows at
  the image-token positions, in order (HF LLaVA's masked scatter).

Fixes this needed (separate commits)

• Text-only prompts on rank-3 image features. With no images, MultiModalFeatures dropped the batch
  dimension, so [num_images, num_tokens, hidden] became [0, hidden] and ORT failed with "Invalid rank
  for input: image_features". The batch dimension is now dropped only for a negative batch size.
  Every in-tree test model uses rank-2 features, so none of them changes.
• Generator.set_active_adapter on multimodal pipelines. MultiModalPipelineState didn't override
  SetActiveAdapter. The adapter went to the pipeline state's own run options, which no session uses,
  so it was silently ignored for every VLM type. It's now forwarded to the decoder state. This covers the
  models that run through MultiModalPipelineState; pipelined decoder configs (decoder.pipeline) aren't changed.
• A renamed vision input. The processors hand the pixels over under model.vision.inputs.pixel_values,
  but the image count only looked for the default pixel_values name, so with a renamed input it fell to 0
  and the vision model failed on a [0, ...] output. It now matches the configured name too (this also fixes
  a renamed rank-3 gemma3 export).

Tests

• test/python/create/create_dummy_llava_models.py generates two tiny models, checked in as
  test/models/llava and test/models/llava-rank2 (~14 KB each; the second emits [total_tokens, hidden]
  features and sets num_visual_tokens). The generator reproduces them byte for byte. The decoder weights
  each embedding by its absolute position, so its logits change when image features are swapped or moved.
• test/python/models/test_llava_models.py (20 tests):
  • token expansion and text-only prompts;
  • generation identical to plain ONNX Runtime sessions driving the same graphs (1 image, 2 images
    in both orders, text-only, and 1/2/0 images on the rank-2 model). The reference's inputs are built
    without the processor under test: the prompt tokenized and expanded in the test, and each image
    preprocessed on its own. Swapping the two images changes the generated tokens;
  • a renamed vision input. This test fails without the renamed-input fix;
  • errors for token/image count mismatch, num_visual_tokens mismatch, a missing image_token_id,
    and a processor config that leaves the pixels uint8 or channel-last;
  • an active adapter reaching the decoder. This test fails without the adapter fix.
• Existing multimodal Python tests (lfm2-vl, gemma4, mistral3, qwen fara/3.5, lfm2-audio,
  decoder-input-ids), the API, component-session, non-generative and adapter graph-capture tests, and
  C++ unit_tests all pass on Linux x64 CPU.
• Also checked end to end on a real fixed-resolution VLM export on CPU (fp32) and WebGPU (fp16 and fp32,
  int8 KV cache, LoRA adapter). Greedy tokens are identical to a plain ONNX Runtime reference driving the
  same graphs (CPU fp32, with and without the adapter, rerun after these changes).

Notes

• This PR doesn't depend on anything in ort-extensions and builds against the current cmake/deps.txt pin.
• Some encoders are trained with pad-to-square before the resize, and ort-extensions has no such
  transform. Companion PR: Add a PadToSquare image transform onnxruntime-extensions#1146 adds PadToSquare; after a cmake/deps.txt bump it
  can go in processor_config.json.
• If Pipeline-as-Config (Copilot design of Pipeline-as-Config #2114, Pipeline-as-Config: Declarative model dispatch replacing model_type string registry #2115/Pipeline-as-Config: structural model dispatch (#2114) #2210) lands, it would subsume this kind of type registration.
  This change is small and usable now.

cc kunal-vaishnavi Tianlei Wu (@tianleiwu)

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Non-default vision input names break rank-3 LLaVA generation by producing zero-sized feature buffers, while order-insensitive test fixtures can miss incorrect image-feature ordering.

2 open issues

• Configured vision input name yields zero batch size · New
• Decoder ignores image feature ordering · New

What changed in this PR

Adds a config-driven llava model type for fixed-resolution vision-language models, reusing the existing vision → embedding → decoder pipeline.

Changes:

• Registers LLaVA support and adds image preprocessing, placeholder expansion, and configuration validation.
• Preserves rank-3 image features for text-only prompts and forwards active adapters to the decoder.
• Adds a small model fixture, its generator, and Python regression tests.

File	Description
test/​python/​models/​test_llava_models.py	Tests preprocessing, generation, validation, and adapters.
test/​python/​create/​create_dummy_llava_models.py	Generates the small LLaVA test fixture.
test/​models/​llava/​tokenizer.json	Defines the fixture vocabulary and image token.
test/​models/​llava/​tokenizer_config.json	Configures fixture tokenizer metadata.
test/​models/​llava/​processor_config.json	Defines fixed-resolution image transforms.
test/​models/​llava/​genai_config.json	Configures the three-session test pipeline.
src/​models/​preprocessing/​multi_modal_processor.cpp	Registers the LLaVA processor.
src/​models/​preprocessing/​llava_image_processor.h	Declares the processor and token-expansion helper.
src/​models/​preprocessing/​llava_image_processor.cpp	Implements preprocessing and placeholder expansion.
src/​models/​multi_modal.h	Declares multimodal adapter forwarding.
src/​models/​multi_modal.cpp	Forwards active adapters to the decoder.
src/​models/​model_type.h	Registers llava as a vision-language model.
src/​models/​io/​multi_modal_features.cpp	Preserves feature rank for zero-image inputs.
README.md	Lists LLaVA-style model support.
.gitignore	Allows tracking the LLaVA test fixtures.

_{Review effort: Balanced}

[Sushanth Rajasankar (sushraja-msft)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).

[Pierluigi Urru (stwykd)]

Baiju Meswani (@baijumeswani) kunal-vaishnavi could you take a look when you get a chance? Sushanth suggested you. merged main in since the integration tests were failing on the older base, could someone /azp run again?