Challenge 5: Uncalibrated Monocular VSLAM

This repository addresses an off-device monocular VSLAM pipeline for smartphone video streams with unknown intrinsics. The goal is to recover a high-precision ego-trajectory and a dense 3D point cloud from raw video, and to benchmark the result against ARCore and other state-of-the-art methods.

The rendered final report and update-meeting slides are available on the GitHub Pages.

https://github.com/JanDuchscherer104/prml-vslam/raw/main/docs/figures/evidence/tum-rgbd-loop.mp4

https://github.com/JanDuchscherer104/prml-vslam/raw/main/docs/figures/evidence/teddy-vista-loop.mp4

Status

Implemented or functional:

• Streamlit workbench pages for Record3D capture, ADVIO datasets, pipeline runs, and metrics review
• ADVIO local dataset readiness checks, selective downloads, and replay tooling
• Custom Record3D dataset with 8 scenes, captured with an iPhone 17 Pro Max.
• TOML-backed run planning and persisted pipeline request loading
• Separate offline and streaming runner paths
• ViSTA-SLAM, MASt3R-SLAM, and LingBot-Map integration
• Rerun live streaming and .rrd file export
• Explicit trajectory evaluation when benchmark inputs are available
• cloud and efficiency evaluation execution

Not yet implemented or limited:

• reference reconstruction stage

Quick Entry

uv sync --extra dev
uv lock --check
make ci

Launch the Streamlit workbench:

uv sync --extra streaming
uv run prml-vslam app

For dataset-backed pipelines or sweeps, download the raw datasets and build the normalized VSLAM datastore first. The runbook is in SETUP.md#dataset-downloads-and-vslam-datastore.

Plan or run a persisted pipeline request (after setting up ViSTA-SLAM as per SETUP.md):

uv run prml-vslam plan-run-config .configs/pipelines/advio-15-offline-vista.toml
uv run prml-vslam run-config .configs/pipelines/advio-15-offline-vista.toml

After completing the LingBot setup, plan or run the LingBot full pipeline:

uv run prml-vslam plan-run-config .configs/pipelines/lingbot-full.toml
uv run prml-vslam run-config .configs/pipelines/lingbot-full.toml

Each run-config invocation writes a timestamped command log under .logs/runs/<run-id>/, where <run-id> is the filesystem-safe run identifier derived from the config's experiment_name. Log filenames use YYYY-MM-DD_HH:MM:SS_<run-id>.log.

See SETUP.md for environment setup and src/prml_vslam/pipeline/README.md for pipeline TOML details.

Challenge Context

Professional SLAM systems usually require rigid factory calibration. Consumer frameworks like ARCore are stable due to real-time sensor fusion, but often fail when processing raw video retrospectively. In particular, they struggle with global metric consistency and high-fidelity dense mapping when camera intrinsics are unknown.

The system should build on existing monocular dense VSLAM methods such as ViSTA-SLAM, MASt3R-SLAM, or LingBot-Map, take a smartphone monocular video stream as input, autonomously handle unknown intrinsics, and output a high-precision trajectory together with a dense 3D point cloud.

Evaluation

• use ADVIO plus custom self-recorded dataset with raw video and odometry logs.
• Identify suitable metrics for pose drift and reconstruction fidelity.
• Evaluate at least two state-of-the-art VSLAM methods.
• Compare trajectories against available references and optional ARCore baselines.
• Compare dense geometry against reference reconstructions and optional ARCore maps when those baselines exist.
• Develop a custom capture or logging workflow for raw video and optional baseline data
• Measure efficiency in terms of latency and memory consumption

Deliverables

• A reproducible off-device monocular VSLAM pipeline for raw smartphone video with unknown intrinsics.
• Benchmark results for at least two methods, including trajectory and dense reconstruction quality.
• A custom dataset capture workflow or app with raw video and ARCore baseline logs.
• An evaluation report covering accuracy, reconstruction quality, latency, and memory consumption.
• A final recommendation for the most suitable pipeline in this challenge setting.

Starting Points

• Ground-truth 3D point clouds: COLMAP + Meshroom or COLMAP + 3DGS
• Point cloud comparison: CloudCompare, with metrics for example from Open3D
• Trajectory comparison: evo
• Papers and upstream methods: ViSTA-SLAM, MASt3R-SLAM, and LingBot-Map