SLAMASS — Semantic SLAM Operator UI for the Unitree Go2

Hackathon project from HACK26 (Dimensional Inc. robotics hackathon, March 2026). A live web operator UI for the Unitree Go2 that fuses a SLAM-backed 2-D occupancy map with floating semantic point-of-interest cards generated from OpenAI vision and live YOLO object detections, plus an LLM operator-chat agent that can search, navigate, and control the robot through natural language.

SLAMASS UI at the end of the hackathon — POV of the Go2 (left) showing the team posing in front of the robot, the live SLAM occupancy map (centre) with the robot pose and semantic POIs, and the operator timeline (right) tracking session events.

SLAMASS = "Simultaneously Located And Mapped Semantic Segmenting map" — in practice: a navigable SLAM map with floating semantic POI windows anchored into the world.

---

What it does

• Live first-person camera feed from the Go2 streamed at ~10 Hz.
• Persistent 2-D BEV occupancy map built from the robot's live SLAM costmap, downsampled and stored across runs under ~/.local/state/dimos/slamass/.
• Manual Inspect Now flow — operator clicks a button, the current POV image is sent to OpenAI vision with a structured prompt; the resulting semantic description is anchored to the robot's pose at the moment of capture and shown as a floating card on the map.
• Live YOLO object layer — repeated detections of the same class are promoted into persistent map objects, each with its own thumbnail, "Go To" pose, and delete control. Toggle live/pause and layer visibility from the UI.
• Operator chat agent — an LLM agent (powered by an MCP-style tool layer) that can:
  • search the semantic memory ("did we map any chairs?")
  • manipulate the UI (zoom, highlight, focus on a POI)
  • inspect the current view on demand
  • navigate to a saved POI or YOLO object
  • toggle the YOLO layer / pause ingestion
  • save the current map
  • call a curated subset of the Go2's MCP robot tools (sit, stand, dance, etc.)
• Teleop on/off for keyboard driving when the agent isn't in control.

---

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                       Web UI (React + Vite)                          │
│         POV pane          ·   Map pane    ·    Operator rail         │
│         JPEG @ ~10 Hz         BEV + POIs       Timeline / chat       │
└────────────────────────────┬─────────────────────────────────────────┘
                             │ HTTP / WebSocket
                             ▼
┌──────────────────────────────────────────────────────────────────────┐
│                  Sidecar service  (`dimos-slamass`)                  │
│  ┌──────────────┐  ┌─────────────┐  ┌──────────────┐  ┌───────────┐  │
│  │  POV ingest  │  │  Map memory │  │  VLM POIs    │  │ YOLO      │  │
│  │  + JPEG enc. │  │  (BEV + IO) │  │  (OpenAI)    │  │ promotion │  │
│  └──────────────┘  └─────────────┘  └──────────────┘  └───────────┘  │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  Operator chat agent — OpenAI tool-calls into the above + MCP │    │
│  └──────────────────────────────────────────────────────────────┘    │
└────────────────────────────┬─────────────────────────────────────────┘
                             │ MCP
                             ▼
┌──────────────────────────────────────────────────────────────────────┐
│       dimos robot stack — `unitree-go2-slamass-mcp` daemon           │
│             (sim or real Unitree Go2 @ 192.168.123.161)              │
└──────────────────────────────────────────────────────────────────────┘

• slamass/ — Python sidecar service (~250 LOC of glue around dimos plus map memory, storage, YOLO protocol, chat agent).
• slamass-app/ — React + Vite + Tailwind frontend (~570 LOC of src/).
• docs/ — quickstart, full demo spec, and the operator-agent tool reference.

---

Quick start

Full step-by-step in docs/quickstart.md. Short version (assumes the dimos runtime is already installed):

# Build the frontend once
cd slamass-app
npm install && npm run build

# Shell 1: start the robot stack
source .venv/bin/activate
dimos --simulation --viewer none run unitree-go2-slamass-mcp --daemon

# Shell 2: start the SLAMASS sidecar
source .venv/bin/activate
export OPENAI_API_KEY=sk-...
dimos-slamass

# Then open http://localhost:7780

To run against the real robot replace --simulation with --robot-ip 192.168.123.161 and connect to the Go2's Wi-Fi.

The full operator-agent tool catalogue is in docs/agent_tools.md; the running product spec in docs/demo.md.

---

Team

Built in 36 hours at HACK26, the Dimensional Inc. robotics hackathon in Zurich, March 2026, by a six-person team mixing ETH Zurich master's students and Dimensional engineers.

This repo is Federico Cominelli's snapshot of the team's feat/polaris-merged branch, packaged for portfolio display.

• 🛠️ Original team repo (Sherif Nafie's account, full upstream history): https://github.com/sherifnafie/dimos-slamass
• 🤖 Built on top of Dimensional Inc.'s dimos robotics framework (Apache 2.0).

---

License

This snapshot retains the upstream Apache License 2.0 — see LICENSE. Copyright remains with Dimensional Inc. and the original contributors. The hackathon slamass additions are released under the same Apache 2.0 license.