Ask about the startup parameters for the test run of the official ranking list on the official website

[jibuzixin]

Hello, I'd like to ask which startup parameters are used when testing official models. This includes parameters such as --runs and --timeout-multiplier, specifically under what circumstances each of them would be adopted and how to set their values appropriately.

Additionally, I have a follow-up question regarding the openclaw.json config file:
Are there any hard constraints for model configuration parameters such as max output tokens and context window size defined in this file? Do these test parameters need to be configured differently for distinct models respectively?

[ScuttleBot]

👋 Hi @jibuzixin! I'm @olearycrew's OpenClaw bot, helping triage this repo.

Great questions! Here's what the official benchmark runs use:

Official Run Parameters

Parameter	Official Value	Notes
--runs	1	Single run per task for the leaderboard. Multiple runs (--runs 3-5) are recommended for community users who want variance data.
--timeout-multiplier	1 (default)	Official runs use the default multiplier. Each task defines its own timeout_seconds in the frontmatter.
--thinking	off (default)	Unless specifically testing reasoning models, thinking is off.

openclaw.json Configuration

The official runs use a minimal OpenClaw configuration:

• No hard constraints on max output tokens or context window — the model's natural limits apply via the provider (OpenRouter)
• No per-model config differences — every model gets the same benchmark environment and timeouts
• The key config values: gateway port, model routing via OpenRouter, and API keys

The philosophy is: same conditions for every model. The benchmark doesn't handicap slower models with longer timeouts or give faster models shorter windows. If a model can't complete a task in the default timeout, that's a valid signal about its speed/capability tradeoff.

Practical Guidance

• For community runs with slower local models (e.g., via sglang/vllm), --timeout-multiplier 2 or 3 is reasonable
• For hosted API models through OpenRouter, the default multiplier should be fine
• The openclaw.json model config (max tokens, context window) only matters if you're using a local/custom endpoint that needs explicit limits

Hope that helps! Feel free to ask follow-ups.