ViSTR-Bench: Can MLLMs Reason from
Continuous Visual Cues in Dynamic Scenes?

Han Li^1,2  Si Liu¹  Zehao Huang  Dongxin Lyu³  Longfei Xu¹  Jiahui Fu¹ 
Daxin Tian^1,2  Yuliang Xiu³  Naiyan Wang

¹Beihang University  ²Zhongguancun Academy  ³Westlake University

We introduce the Visual Spatial-Temporal Reasoning Benchmark (ViSTR-Bench), a novel evaluation suite designed to systematically assess whether MLLMs can perform qualitative reasoning from continuous visual cues in dynamic scenes.

---

📢 News

• [2026-07-23] The paper, dataset, and evaluation code have been released.

💡 Highlights

• We establish a four-dimensional evaluation framework covering Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics, and construct a benchmark comprising 15 diverse subtasks and 1,340 high-quality video question-answer pairs across tabletop, indoor, and outdoor scenarios.
• We conduct extensive evaluations across proprietary, open-source, and specialized spatial MLLMs, revealing that current models, despite their progress in general video question answering and static spatial reasoning, still fall substantially short of human performance in reasoning about dynamic physical scenes.
• We provide systematic diagnostic analyses of text-based Chain-of-Thought prompting strategies, visual input formats, and model failure modes, together with pilot studies showing that explicitly providing task-relevant spatial and motion evidence can substantially improve selected reasoning tasks.

🛠️ Usage

Installation

conda create -n vistr python=3.10 -y
conda activate vistr

# Install PyTorch matching your CUDA/driver setup.
pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0

pip install -r requirements.txt

# Optional: install Flash Attention if supported by your GPU/CUDA stack.
pip install flash-attn --no-build-isolation

Data Preparation

Set the dataset repository ID and local path in conf/config.yaml:

datasets:
  repo_name: "<HF_DATASET_REPO>"
  datasets_path: "./data/ViSTR-Bench"
  video_num_frames: 16
  video_frame_cache: ""

Download the dataset with:

python download_datasets.py

You can also download it manually:

hf download <HF_DATASET_REPO> \
  --repo-type dataset \
  --local-dir ./data/ViSTR-Bench

The required local layout is:

data/ViSTR-Bench/
  data.json
  data/
    Motion_Perception/
    Outcome_Prediction/
    Physical_Dynamics/
    Spatial_Relations/

Model Preparation

The local model root is controlled by model.download_path in conf/config.yaml:

model:
  download_path: "./model"
  device: "cuda"
  torch_dtype: "bfloat16"
  trust_remote_code: true

Download a single model:

python download_model.py --model OpenGVLab/InternVL3_5-8B

Download all configured models listed in models::

python download_model.py --batch

Local Evaluation

Supported Hugging Face model names:

OpenGVLab/InternVL3_5-8B
OpenGVLab/InternVL3_5-14B
OpenGVLab/InternVL3_5-30B-A3B
OpenGVLab/InternVL3_5-38B
lmms-lab/LLaVA-OneVision-1.5-4B-Instruct
lmms-lab/LLaVA-OneVision-1.5-8B-Instruct
RUBBISHLIKE/spacer-sft
internlm/spatial-ssrl
hongxingli/spatialladder
rayruiyang/vst-7b-rl

Run local evaluation:

bash start_eval.sh

API Evaluation

test_api.py provides a lightweight runner for OpenAI-compatible GPT models and Gemini models.

Set API credentials before running:

export OPENAI_API_KEY=<your-openai-api-key>
export GEMINI_API_KEY=<your-gemini-api-key>

Run GPT evaluation:

python test_api.py --api gpt --model <model> --data-root ./data/ViSTR-Bench

Run Gemini evaluation:

python test_api.py --api gemini --model <model> --data-root ./data/ViSTR-Bench

For additional options and parameter settings, run:

python test_api.py --help

📝 Citation

If you find this work useful, please consider citing our paper:

@article{vistr-bench,
  title={ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?},
  author={Li, Han and Liu, Si and Huang, Zehao and Lyu, Dongxin and Xu, Longfei and Fu, Jiahui and Tian, Daxin and Xiu, Yuliang and Wang, Naiyan},
  journal={arXiv preprint arXiv:2607.20868},
  year={2026}
}

📄 License

This project is licensed under the Apache-2.0 License. See LICENSE.txt for more information.

🙏 Acknowledgements

We thank the authors of the following open-source projects:

• VSI-Bench
• Dyn-Bench
• OmniSpatial