Skip to content

Add Qwen 3.5 for Video Caption - #2344

Open
meatybobby wants to merge 3 commits into
mainfrom
bobchen/qwen_3_5
Open

Add Qwen 3.5 for Video Caption#2344
meatybobby wants to merge 3 commits into
mainfrom
bobchen/qwen_3_5

Conversation

@meatybobby

Copy link
Copy Markdown
Contributor

Description

Add Qwen 3.5 for Video Caption

Usage

# Add snippet demonstrating usage

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

@meatybobby
meatybobby requested review from a team, abhinavg4 and suiyoubi as code owners August 25, 2026 20:28
@copy-pr-bot

copy-pr-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds Qwen 3.5 support to video caption preparation, generation, enhancement, documentation, and tutorial configuration. It also introduces unrelated interleaved image-cache tests that currently target a nonexistent production API.

  • Adds Qwen3.5 model IDs, revisions, pixel parameters, and non-thinking chat-template behavior.
  • Exposes Qwen3.5 through video captioning and enhancement CLI choices.
  • Expands model and stage tests for the new variant.
  • Adds interleaved decoded-image cache tests without the corresponding production implementation.

Confidence Score: 4/5

The PR should not merge until the newly added interleaved cache tests are removed or the production cache API they exercise is implemented.

The Qwen3.5 integration is coherently wired across preparation, generation, enhancement, documentation, and CLI selection, but the new interleaved tests deterministically invoke a method absent from every production stage class and will fail the test suite.

Files Needing Attention: tests/stages/interleaved/test_stages.py and nemo_curator/stages/interleaved/stages.py

Important Files Changed

Filename Overview
nemo_curator/models/prompt_formatter.py Adds Qwen3.5 processor routing and disables thinking when formatting its video prompts.
nemo_curator/models/qwen_vl.py Registers the Qwen3.5-9B checkpoint, pinned revision, and multimodal pixel parameters.
nemo_curator/models/qwen_lm.py Registers Qwen3.5-27B for caption enhancement, disables multimodal slots by default, and formats prompts without reasoning traces.
tutorials/video/getting-started/video_split_clip_example.py Exposes Qwen3.5 as a caption-generation and caption-enhancement CLI choice.
tests/stages/interleaved/test_stages.py Adds three cache tests that call an absent iter_decoded_images production method and therefore fail at runtime.

Sequence Diagram

sequenceDiagram
    participant Clip as Video window
    participant Prep as Caption preparation
    participant VLM as Qwen3.5-9B
    participant LM as Qwen3.5-27B
    Prep->>Prep: Format video prompt with thinking disabled
    Clip->>Prep: Sampled video frames
    Prep->>VLM: Prompt and multimodal video data
    VLM-->>Clip: Generated caption
    Clip->>LM: Caption enhancement prompt
    LM-->>Clip: Enhanced caption
Loading

Reviews (1): Last reviewed commit: "Remove unused files" | Re-trigger Greptile

df = task.to_pandas()
image_mask = df["modality"] == "image"

first = list(stage.iter_decoded_images(task=task, df=df, row_mask=image_mask))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Missing decoded-image cache method

When the new interleaved tests run, they invoke iter_decoded_images, but neither the filter classes nor their base classes define that method, causing all three tests to fail with AttributeError.

Knowledge Base Used: Interleaved and synthetic data curation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant