Keep the system prompt stable across turns so prompt caching survives - #971
Merged
Conversation
16francej
force-pushed
the
prompt-cache-stability
branch
2 times, most recently
from
September 8, 2026 04:57
604c16c to
7ba9be4
Compare
Prompt caches are prefix-based: any change to the system prompt invalidates every cached message block behind it. The per-turn blocks (local time at minute resolution, memory recall, onboarding) were appended to the end of the system prompt, so in production no session got a cache read at turn start and deep sessions re-wrote 150-300k tokens of history on every turn. Those blocks now ride the per-turn <environment> note on the user message; per-scope state and policy (logins, connected apps, roster, obligations, automation directives) stay in the system prompt, where they change rarely and keep system priority. The note is persisted on the user entry, replayed when history is reconstructed, and counted by the token estimator, so reconstructed history matches what was sent and compaction sees the real history size. With the system prompt stable, the split-at-boundary machinery is gone. The same option now asks pi-ai for long cache retention (1h ttl on Anthropic system, tools and last message; 24h retention on OpenAI; pi-ai's per-provider compat flags decide where it is safe), so turns 10-70 minutes apart still hit. 1h writes cost 2x base input instead of 1.25x, paid only on the new suffix. Cap the compaction budget at 150k tokens. Half of a 1M window let sessions grow to 300k+ tokens per call without ever compacting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
16francej
force-pushed
the
prompt-cache-stability
branch
from
September 8, 2026 05:00
7ba9be4 to
5738021
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Prompt caches are prefix-based: any change to the system prompt invalidates every cached message block behind it. The per-turn blocks (local time at minute resolution, memory recall, onboarding) were appended to the end of the system prompt, so no session got a cache read at turn start and deep sessions re-wrote their whole history on every turn. Replayed history also never byte-matched the live prompt, because the user entry stored only the input text, not the environment note that was sent with it.
Separately, the compaction budget for 1M-window models was 436k tokens, so long DM sessions sat at 250k+ tokens per call and never compacted.
What
<environment>note on the user message. Per-scope state and policy (logins, connected apps, roster, obligations, automation directives) stay in the system prompt, where they change rarely and keep system priority. The note is persisted on the user entry, replayed when history is reconstructed, and counted by the token estimator.PI_SYSTEM_CACHE_SPLIT) now asks pi-ai for long cache retention (1h ttl on Anthropic system, tools and last message; 24hprompt_cache_retentionon OpenAI; pi-ai's per-provider compat flags decide where it is safe), so turns tens of minutes apart still hit.contextTokenBudgetForModelis capped at 150k tokens.Verified
Live against the API with the same 23k-token conversation, second turn one minute later:
New tests: the system prompt is byte-identical across turns a minute apart; replay reproduces the environment note; the estimator counts it; the budget cap. Typecheck, lint and the orchestrator, memory, prompt, compaction, tape and replay suites pass.
Trade-offs: user entries grow by the environment note (a few k tokens per turn, itself cached); 1h cache writes cost 2× base input instead of 1.25×, paid only on the new suffix.