roboserve

roboserve is a Python-first robot serving and control stack that pairs a frontier embodied-reasoning model (Gemini Robotics-ER) with a self-hosted VLA (vision-language-action) action layer (OpenVLA-7B). It runs end-to-end against MuJoCo-simulated tasks today and ships with the full plumbing for scaled GPU inference on Google Cloud Vertex AI.

The project is simulation-only by design. Real-robot or hardware integration is intentionally out of scope; every contract here is built and validated against MuJoCo and a deterministic grasp surrogate. See docs/design.md for the full status.

What's in the box

• MuJoCo Franka environment with three canonical tasks (pick_place, block_stack, colour_sort), Cartesian end-effector control, native camera rendering with a deterministic synthetic fallback, and a simple grasp/release surrogate so scripted and VLA policies can move blocks without IK.
• Gemini Robotics-ER reasoning with strict-JSON Pydantic schemas, a recorded-fixture client for offline/CI use, and a live GeminiReasoningClient adapter behind an optional dependency.
• Two policy implementations: a hand-written PrimitivePolicy for scripted baselines and a VLAPolicy that talks to the serving layer.
• FastAPI VLA serving with /v1/act, /healthz, request/response validation, Prometheus metrics, a synchronous batcher facade, and four pluggable runtimes (stub, openvla, openvla-7b-int8, smolvla).
• Real OpenVLA-7B integration (BF16 and bitsandbytes INT8) backed by a CUDA Docker image and a Vertex AI CustomJob submitter.
• C++/pybind11 perception hot path (roboserve_cpp_preproc) with a pure Python fallback and bit-exact parity tests.
• Evaluation runner that produces metrics.json, an aggregated docs/results/main-table.md, optional PNG frame sequences, rollout.gif, and rollout.mp4 artifacts.
• Benchmarks: latency, throughput, and BF16-vs-INT8 quantisation comparison, all writing back to docs/results/serving-benchmarks.md.
• Mock ROS2 bridge (Ros2BridgeEnv) for interface smoke tests; it does not talk to ROS2 or hardware.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,sim]"

# Sanity: tests + a primitives rollout against the recorded reasoning fixture
pytest
roboserve-run-task --task pick_place --policy primitives \
  --fixture-dir tests/fixtures/er16_responses/pick_place

# Same rollout but capture PNG frames, a GIF, and an MP4
roboserve-run-task --task pick_place --policy primitives \
  --fixture-dir tests/fixtures/er16_responses/pick_place \
  --save-frames --save-gif --save-mp4

# All three canonical tasks through the in-process stub VLA serving runtime
roboserve-run-task --task all --policy vla --max-episode-length 2

# Local serving benchmarks (writes docs/results/serving-benchmarks.md)
roboserve-run-benchmark all --latency-requests 5 --throughput-requests 10 \
  --write-report

Artifacts land under predictions/<task>/<policy>/<seed>/ (gitignored), and the aggregated summary table at docs/results/main-table.md is regenerated by roboserve-aggregate-results.

Installation

Python ≥ 3.11 is required. The recommended setup uses a venv plus the dev,sim extras.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,sim]"

Optional extras (declared in pyproject.toml)

Extra	Purpose	Notable packages
dev	Tooling	pytest, ruff, mypy, types-PyYAML
sim	MuJoCo simulator + frame/GIF/MP4 artifacts	mujoco, pillow, imageio, imageio-ffmpeg
reasoning	Live Gemini Robotics-ER calls	google-genai, python-dotenv
vla	OpenVLA runtime (CUDA-only in practice)	torch>=2.2,<2.3, transformers==4.40.1, bitsandbytes==0.43.3, accelerate==0.26.0, timm==0.9.10, tokenizers==0.19.1, numpy>=1.26,<2, pillow
cloud	GCP / Vertex AI submitter and GCS sync	google-cloud-aiplatform, google-cloud-storage, click, python-dotenv

C++ extension (optional)

pip install -e . will attempt to build the roboserve_cpp_preproc pybind11 extension. The build is wrapped by OptionalBuildExt in setup.py; if a C++17 toolchain isn't available the install still succeeds and roboserve falls back to the Python implementation in roboserve.perception.preprocessor. Bit-exact parity is enforced by tests/contracts/test_perception_contract.py whenever the extension is present.

Data setup

roboserve does not require any model weight or dataset download for the default development loop:

• MuJoCo scenes live under tasks/<task>/scene.xml and ship in the repo. They are loaded by roboserve.tasks.load_task which also reads eval_config.yaml (e.g. max_episode_length: 25) and task.py (the natural-language instruction and an object-position success_fn).

• Reasoning fixtures are checked into tests/fixtures/er16_responses/<task>/{plan,success,points}.json. These were recorded from the live Gemini API and are replayed deterministically by RecordedReasoningClient. Tests and offline rollouts use them directly.

• Recording new fixtures (optional) requires the reasoning extra and a valid Gemini API key. Copy .env.example to .env, fill in GEMINI_API_KEY (and optionally ER16_MODEL), then run:

  pip install -e ".[reasoning]"
  cp .env.example .env  # then edit
  python -m roboserve.scripts.record_reasoning_fixtures --task all \
    --delay-between-tasks-s 40

  The script throttles between tasks to stay inside the public Gemini rate limits.

• OpenVLA weights are loaded directly from Hugging Face on first use of the openvla / openvla-7b-int8 runtimes (default model openvla/openvla-7b, pinned revision 47a0ec7fc4ec123775a391911046cf33cf9ed83f). The Cartesian un-norm key defaults to bridge_orig. All three values are overridable via ROBOSERVE_OPENVLA_MODEL_ID, ROBOSERVE_OPENVLA_MODEL_REVISION, and ROBOSERVE_OPENVLA_UNNORM_KEY. A 7B model in BF16 needs ~16 GB VRAM; INT8 fits in ~10 GB. CPU-only runs are not practical.

Repo structure

roboserve/
├── benchmarks/                Local benchmark scripts (latency, throughput, quantization, suite)
├── cloud/
│   ├── docker/                CUDA-enabled VLA image, Cloud Build config, container entrypoint
│   ├── gcs/sync.py            Two-way GCS ↔ local artifact sync
│   ├── openvla_smoke.py       Minimal step-by-step OpenVLA load/inference probe
│   ├── scripts/               build_and_push.sh, grant_iam.sh
│   └── vertex/submit.py       Vertex AI CustomJob submitter (click CLI)
├── configs/                   Sample YAML configs (env, policy, serving, experiments)
├── cpp/preproc/               C++17 image-ops + pybind11 bindings (roboserve_cpp_preproc)
├── docker/                    Local stub-only Dockerfiles + docker-compose
├── docs/
│   ├── architecture.md        Layer breakdown + Mermaid sequence diagram
│   ├── design.md              In-repo implementation status & milestones
│   └── results/               Auto-generated benchmark/rollout summaries
├── scripts/                   Top-level shims for the console scripts
├── src/roboserve/
│   ├── evaluation/runner.py   run_rollout / run_task_suite + artifact writers
│   ├── perception/            Preprocessor protocol + Python and C++ implementations
│   ├── policy/                PrimitivePolicy and VLAPolicy
│   ├── reasoning/             Recorded + live Gemini clients, prompts, strict JSON parsers
│   ├── robot/                 RobotEnv protocol, MuJoCoFrankaEnv, Ros2BridgeEnv
│   ├── scripts/               Console-script entry points
│   ├── sdk/                   RoboserveClient public facade
│   ├── serving/               FastAPI app, batcher, runtimes (stub, OpenVLA, SmolVLA)
│   ├── tasks.py               Canonical task registry + loader
│   └── types.py               Pydantic data contracts shared across layers
├── tasks/<task>/              MJCF scene + eval_config.yaml + task.py
├── tests/
│   ├── contracts/             Layer-contract tests (perception, reasoning, robot)
│   ├── fixtures/er16_responses/  Frozen Gemini Robotics-ER plan/success/point fixtures
│   ├── integration/           Rollout, ROS2 smoke, SDK smoke
│   └── unit/                  Policy, server, batcher, registry, aggregator unit tests
├── Makefile                   install/test/lint/typecheck/smoke + cloud targets
├── pyproject.toml             Project metadata, extras, console scripts, mypy/ruff config
├── setup.py                   Optional pybind11 build (OptionalBuildExt)
├── docker-compose.yml         Forwards to docker/docker-compose.yml
├── roboserve-design-v0.md     Original scope document
└── README.md                  This file

Architecture in one paragraph

Every layer is a typing.Protocol (RobotEnv, Preprocessor, ReasoningClient, Policy, ServingTransport, VLARuntime) so concrete implementations stay swappable. The default flow is: the rollout runner resets MuJoCoFrankaEnv, asks RoboserveClient.plan(...) for a strict Plan (recorded fixture or live Gemini), and then for each step in the plan iterates policy.act(step, obs) → env.step(action) while task.success_fn watches the observation state for an object-position success. VLAPolicy encodes each observation as a base64 PNG and calls the in-process or HTTP ServingTransport. The serving layer wraps a Batcher around one of the runtimes and exposes POST /v1/act. See docs/architecture.md for a Mermaid diagram of the exact data flow.

Running locally

Tasks and rollouts

roboserve-run-task is the main entry point. Useful flags (the underlying function is roboserve.evaluation.run_rollout / run_task_suite):

Flag	Meaning
--task pick_place|block_stack|colour_sort|all	Single task or full suite
--policy primitives|vla	Scripted policy or VLA-via-serving
--seed N, --seeds 0,1,2	Single seed or comma-separated list
--fixture-dir PATH	Per-task recorded reasoning fixture
--fixture-root PATH	Root for --task all (default tests/fixtures/er16_responses)
--serving-runtime stub|openvla|openvla-7b-int8|smolvla	Runtime selector for --policy vla
--max-episode-length N	Override the per-task step cap
--save-frames / --save-gif / --save-mp4	Optional artifact generation
--output-dir PATH	Where metrics.json and frame artifacts land (default predictions/)

After one or more rollouts, regenerate the summary table:

roboserve-aggregate-results        # writes docs/results/main-table.md

Local serving

make serve   # uvicorn roboserve.serving.server:app --host 0.0.0.0 --port 8000

The app global uses the deterministic stub runtime so import time stays cheap. To pick a real runtime, use --factory and let create_app read ROBOSERVE_VLA_RUNTIME:

ROBOSERVE_VLA_RUNTIME=openvla \
  uvicorn roboserve.serving.server:create_app --factory --host 0.0.0.0 --port 8000

/v1/act accepts the body defined by roboserve.serving.client.ActRequest (base64 PNG or raw tensor bytes, plus instruction and request_id). /healthz echoes the model version and current batcher config.

Benchmarks

roboserve-run-benchmark latency      --requests 25
roboserve-run-benchmark throughput   --requests 100 --concurrency 8
roboserve-run-benchmark quantization --bf16-latency-ms ... --int8-latency-ms ...
roboserve-run-benchmark all          --write-report

--write-report updates docs/results/serving-benchmarks.md. The quantisation script does not run inference itself; it accepts numbers measured by separate BF16/INT8 runs (typically on Vertex GPUs) and renders them into the report.

ROS2 mock smoke

make ros2-smoke
# or
python -m roboserve.scripts.ros2_smoke --task pick_place

Ros2BridgeEnv inherits from MuJoCoFrankaEnv. The smoke test exists to prove the interface a real ROS2 adapter would expose; nothing here talks to ROS2 or to hardware.

GCP / Vertex AI workflow

Defaults are wired for the project this was originally validated against (nigerian-accent-generator); override the env vars in cloud/vertex/submit.py for your own project. The full runbook lives in cloud/README.md.

# One-time / idempotent
./cloud/scripts/grant_iam.sh

# Build and push the CUDA VLA image via Cloud Build
./cloud/scripts/build_and_push.sh vla-gpu --cloud
# or:
make cloud-build

# Submit an INT8 benchmark or rollout job to Vertex AI A100 SPOT
make cloud-benchmark-int8
make cloud-rollout-int8

# Pull the artifacts back into the local working tree
python cloud/gcs/sync.py \
  --bucket nigerian-accent-generator-roboserve-runs-us1 \
  --job <ROBOSERVE_JOB_ID> --download

The container entrypoint (cloud/docker/entrypoint.sh) accepts benchmark, rollout, serve, and openvla-smoke as subcommands. It forwards ROBOSERVE_* environment variables (set by cloud/vertex/submit.py) into the actual scripts and uploads everything under predictions/ and docs/results/ to gs://<bucket>/<ROBOSERVE_JOB_ID>/ when the job finishes.

The previously measured numbers on Vertex A100 are kept in docs/results/serving-benchmarks.md; BF16 OpenVLA is currently faster than INT8 for this single-request, small-batch serving path.

Development

make install        # pip install -e ".[dev,sim]"
make test           # pytest
make lint           # ruff check .
make typecheck      # mypy src/roboserve
make check          # lint + typecheck + test
make smoke          # primitives + VLA-stub rollout
make benchmark-report
make results        # roboserve-aggregate-results
make clean

Continuous integration runs the same lint/type/test gates plus a smoke rollout and a tiny benchmark suite on every PR (.github/workflows/ci.yml).

Configuration & secrets

• .env.example documents the recognised environment variables. Copy to .env (gitignored) and fill in real values when needed.
  • GEMINI_API_KEY — required only for the live GeminiReasoningClient.
  • ER16_MODEL — Gemini Robotics-ER model id; defaults to gemini-robotics-er-1.6-preview.
• ROBOSERVE_VLA_RUNTIME selects the serving runtime (stub, openvla, openvla-7b-int8, smolvla).
• ROBOSERVE_MAX_BATCH_SIZE, ROBOSERVE_MAX_WAIT_MS — batcher knobs read by the FastAPI app at startup.
• ROBOSERVE_MUJOCO_RENDERER (default true) — set to false in headless containers to skip the native MuJoCo Renderer and use the deterministic synthetic image fallback. The cloud GPU image sets this for you.
• All ROBOSERVE_* and HF_TOKEN / HUGGING_FACE_HUB_TOKEN variables are forwarded into Vertex CustomJobs by cloud/vertex/submit.py (see FORWARDED_ENV_VARS).

Never check secrets into the repo. .env, .env.* (except .env.example), predictions/, *.so, and the usual Python caches are already in .gitignore.

Documentation

• docs/design.md — implementation status, milestones, and out-of-scope notes.
• docs/architecture.md — layer breakdown and a Mermaid sequence diagram of the rollout path.
• cloud/README.md — full GCP runbook and defaults.
• docs/results/main-table.md — auto-generated rollout summary (success rate, episode length, latency).
• docs/results/serving-benchmarks.md — auto-generated latency/throughput/quantisation report.
• roboserve-design-v0.md — original scope document the implementation has been tracking against.

License & disclaimers

• This project is for research and engineering exploration; no warranty.
• Simulation only. The mock ROS2 bridge and any "robot"-shaped contracts exist purely for interface testing — do not connect them to physical hardware.