README.md

MASBench · Game

Local observations. Resource collection. Coordinated delivery.

Main README · Reasoning · Scheduling

Task and Maps

Agents explore a grid world, collect resources, and deliver them to warehouses with resource-specific quotas. Each agent observes a local window and can carry only one resource type at a time, subject to capacity limits. Messages help agents coordinate across their separate views.

This task evaluates memory, routing, and protocol. The release includes 100 maps in custom_map/: 50 simple 15×15 maps and 50 complex 18×18 maps. Each map has a text authoring format and a JSON runtime format.

Original map data and source code, including map conversion helpers, are licensed under MIT. Copyright (c) 2026 Qizhi Chu.

Setup

Activate the shared environment from the main installation guide. From the repository root:

cd Game
cp .env.example .env

Edit .env:

QWEN_OPENAI_API_KEY=your_api_key
QWEN_OPENAI_BASE=https://your-openai-compatible-endpoint/v1

--provider QWEN selects this variable prefix; model IDs are passed to the endpoint unchanged. Environment variables override .env. All commands below run inside Game/.

Run One Map

CUSTOM_MAP_PATH=custom_map/15x15_v1.json MECHANISM_PRESET=base \
python eval.py \
  --provider QWEN \
  --model qwen-plus \
  --max_round 2

This runs two environment rounds. Each round can make several API calls for agent decisions and communication. A two-round smoke test can finish with a score of zero; the purpose is to verify execution and logging.

Results are written to .preset_logs/<timestamp>_<map>_<preset>_<model>_viewjson/. Inspect the agent_log_*.json, game_log_*.json, and meta_log.json files for model responses, trajectory, configuration, and scores.

For the default 50-round budget, omit --max_round 2. Set CUSTOM_MAP_PATH to choose another map; invalid paths raise an error.

Compare Mechanisms

Select one of the registered presets through MECHANISM_PRESET:

Setting	Preset
Base	base
Memory only	stable_fair_memory_only
Memory + random routing	stable_fair_all_random
Memory + gated routing	stable_fair_gate_llm
Memory + protocol + random routing	stable_fair_protocol_all_random
Memory + protocol + gated routing	stable_fair_protocol_gate_llm

For example:

CUSTOM_MAP_PATH=custom_map/15x15_v1.json \
MECHANISM_PRESET=stable_fair_protocol_gate_llm \
python eval.py --provider QWEN --model qwen-plus

Batch runs across settings, models, and maps

Run all six settings on one map:

PYTHON_BIN="$(command -v python)" \
bash run_evaluation_stable_fair.sh custom_map/15x15_v1.json QWEN qwen-plus

Run selected settings:

PYTHON_BIN="$(command -v python)" SETTINGS=base,protocol_gate_llm \
bash run_evaluation_stable_fair.sh custom_map/15x15_v1.json QWEN qwen-plus

Run multiple maps with explicitly chosen models:

PYTHON_BIN="$(command -v python)" MODEL_PROVIDER_PAIRS="QWEN:qwen-plus" \
bash run_all_stable_fair.sh \
  custom_map/15x15_v1.json custom_map/18x18_v1.json

Add provider/model pairs separated by spaces, configuring credentials for each provider prefix. Batch runs execute immediately by default; USE_AT=1 uses the system at scheduler. Set PYTHON_BIN to the active environment's absolute Python path.

Summarize Results

python summarize_results.py \
  --scan .preset_logs \
  --mid-csv analysis/summary_by_map_model_with_tokens.csv \
  --final-out analysis/model_base_mode_table_with_tokens.csv

The summarizer excludes runs with a round budget below 50 by default. Add --min-max-round 0 to inspect smoke-test results; omit that flag for full benchmark aggregation. Add --split-map-size for separate map-size summaries. --scan takes the parent directory containing runs, not an individual run directory.

Replay a Run

Replay dependencies are included in the shared requirements. Replace <run_dir> with an actual run directory:

python replay_interactive_latex.py .preset_logs/<run_dir>

Use the left/right arrow keys to navigate rounds. Add --show-views for per-agent observations. To export video:

python replay_interactive_latex.py .preset_logs/<run_dir> \
  --export-video --video-file results/replays/example.mp4 --fps 2

Example 15×15 map from the manuscript

Add a Mechanism

Implement MemoryMechanism, RoutingMechanism, and/or ProtocolMechanism from masbench/mechanisms/base.py, then register the combination in masbench/mechanisms/registry.py. The reference implementation is masbench/mechanisms/stable_fair.py.

register_preset(
    name="my_method",
    memory_cls=MyMemoryMechanism,
    routing_cls=MyRoutingMechanism,
    protocol_cls=MyProtocolMechanism,
    config_factory=my_method_config_factory,
)

Run it with MECHANISM_PRESET=my_method. To author new maps, use the conversion helpers in custom_map/; see their --help options.

The shared installation supports OpenAI-compatible endpoints and includes google-genai for native Gemini calls. Native Gemini (--provider GEMINI) uses GEMINI_API_KEY; Gemini models served by a compatible endpoint use that endpoint's normal provider prefix.