What we know
The server's instructions field reaches the model verbatim and governs how an agent answers once it calls a tool — verbatim citation blocks, verbatim refusal, staleness passed on. That held across Claude, Gemini/Antigravity and Codex.
It does not govern whether the tool is called. One cold Codex session (docs/CodexQandA.md, session 2) asked How should I chunk documents for retrieval? with no scoping phrase, answered from its own training, and never touched the server. One sentence of scoping flipped it, and every later question in that session used the tool.
Every unscoped question that reached the tool followed a scoped question in the same session. That is one trial each way. It is an anecdote, not a rate.
What to measure
Fresh session per trial, one question, nothing else in context. Record whether any tool call happened (rollout log) and whether an event landed in var/telemetry.ndjson.
Two arms, because the plain reading and the alternative explanation make different predictions:
- Arm A — the question names something the instructions enumerate ("model ids, context windows, endpoint paths, API versions, prices"). e.g.
What is the context window of claude-opus-5?
- Arm B — the question is exploratory and names none of them. e.g.
How should I chunk documents for retrieval?
If tool use tracks the enumeration, the instructions are doing the work and the fix is to widen them. If both arms fail cold, session context was doing the work and no wording of instructions will bind tool choice — it has to move to the harness.
Enough trials per arm to distinguish a rate from a coin flip, across at least two runtimes so the result is not one vendor's harness.
Why it matters
A grounded context layer an agent can silently decline to consult is a governance gap. It is visible in telemetry as a question that produced no event — which is exactly how the Codex case was caught.
The likely conclusion is that tool selection belongs in the harness: system prompt, a required tool-choice setting, or a policy layer above the model. Worth stating with a measurement behind it rather than as an opinion.
Done when
- A table of trials per arm per runtime, committed alongside the transcripts.
- The README's "Out of scope" bullet either keeps its qualified wording or gets replaced by the measured claim.
What we know
The server's
instructionsfield reaches the model verbatim and governs how an agent answers once it calls a tool — verbatim citation blocks, verbatim refusal, staleness passed on. That held across Claude, Gemini/Antigravity and Codex.It does not govern whether the tool is called. One cold Codex session (
docs/CodexQandA.md, session 2) askedHow should I chunk documents for retrieval?with no scoping phrase, answered from its own training, and never touched the server. One sentence of scoping flipped it, and every later question in that session used the tool.Every unscoped question that reached the tool followed a scoped question in the same session. That is one trial each way. It is an anecdote, not a rate.
What to measure
Fresh session per trial, one question, nothing else in context. Record whether any tool call happened (rollout log) and whether an event landed in
var/telemetry.ndjson.Two arms, because the plain reading and the alternative explanation make different predictions:
What is the context window of claude-opus-5?How should I chunk documents for retrieval?If tool use tracks the enumeration, the instructions are doing the work and the fix is to widen them. If both arms fail cold, session context was doing the work and no wording of
instructionswill bind tool choice — it has to move to the harness.Enough trials per arm to distinguish a rate from a coin flip, across at least two runtimes so the result is not one vendor's harness.
Why it matters
A grounded context layer an agent can silently decline to consult is a governance gap. It is visible in telemetry as a question that produced no event — which is exactly how the Codex case was caught.
The likely conclusion is that tool selection belongs in the harness: system prompt, a required tool-choice setting, or a policy layer above the model. Worth stating with a measurement behind it rather than as an opinion.
Done when