Repository navigation
Historical totals silently drift when Claude Code rewrites session files in place (repro + numbers) #994
Description
Summary
Claude Code rewrites session JSONL files in place on resume/compact: between
two scans of the same file on a 34-day corpus (~26k messages), one transcript
lost 5 assistant messages and the file mtime bumped. tokscale recomputes
totals from the live files on every run, so those messages' tokens silently
leave history — and since tokscale submit feeds the leaderboard, submitted
numbers drift the same way.
To be clear about what is already right: your per-message accounting is
exact. On a synthetic corpus with a known manifest (540 distinct assistant
messages, streaming multi-line records with repeated message.id), a cold
run of tokscale@4.7.0 matches ground truth to the digit on output,
cacheRead, cacheWrite and message count, including transcripts under
subagents/**. The dedup key, the per-field max merge for streaming
partials, and the deep discovery walk all check out. This report is about
persistence only.
The measurement
Rewrite one transcript in place, dropping its last 6 assistant messages (the
observable effect of a real resume/compact), then run again:
field before after drifted prediction
output 1,115,321 1,104,872 1,104,872
cacheRead 24,447,923 24,192,633 24,192,633
cacheWrite 8,130,148 8,052,730 8,052,730
messages 540 534 534
Every field lands on the drifted prediction exactly, which pins the cause to
recompute-from-live-files rather than anything in parsing. The
source-message cache behaves as designed — message_cache.rs documents that
the cache must invalidate when the file changes, and
load_or_parse_source_with_fingerprint_and_policy (lib.rs:745) re-parses
and replaces on any fingerprint change — so the design tracks live file
content, and live file content is not stable.
Repro
Self-contained, ~1 minute, no real ~/.claude data involved:
git clone https://github.com/lizhuojunx86/traceguard
cd traceguard/usage-tracker-audit/tokscale-drift-check
./run_check.shIt installs the published npm package (pinned), generates the corpus with an
exact manifest under an isolated fake $HOME, runs three passes
(cold / warm / post-rewrite), and compares. Exit 0 = totals survive the
rewrite, 1 = drift — so the same script doubles as a regression test after a
fix.
Fix shape that has shipped elsewhere
Two trackers in this space already answered this with a keyed
freeze-at-first-sight store: splitrail added an append-only SQLite history
keyed on hash(requestId, messageId) in 3.6.0
(Piebald-AI/splitrail#200; I contributed the e2e fixture pinning that
behavior, Piebald-AI/splitrail#208), and codeburn's daily cache freezes
finalized days and carries them forward when source files vanish. A message
once observed survives any later rewrite; your existing
messageId:requestId key and max-merge already handle the streaming-partial
side, so the missing piece is only that the keyed layer never deletes.
Happy to open a PR wiring the check in as a regression test, or to rerun it
against a branch if you take the store approach.
Unrelated, spotted while reading
input includes a ~4-chars/token estimate of tool_result content on top
of the API-reported input_tokens (+6 tokens per tool-use message on this
corpus, +16% input overall). Tool results are already part of the next
turn's prompt and counted in its input_tokens, so this double counts a
small amount. Separate issue if useful; the numbers above assert only on
fields free of it.
Activity
fix(cache): scope Claude retention to keys that survive a fork
feat(usage-tracker-audit): let tokscale-drift-check verify any binary
lizhuojunx86 commentedon Aug 2, 2026
Verified against merged main (
9ceae64), built from source, using the sameharness that produced the report — it was written so that a fixed version
exits 0.
Rewrite of one transcript, 6 assistant messages / 18 lines removed:
Frozen on every field. Cold-start accuracy is unchanged — still token-exact
against the manifest on all four fields and the message count — so the
retention did not cost correctness on the path it does not touch. The
released 4.8.1 still drifts exactly as 4.7.0 did, which is the case for
cutting a release rather than leaving it on main.
I also checked the invariant you chose over the append-only shape, since
that trade is the interesting part of the fix. Removing a transcript
entirely from the tree after a compaction had already retained turns from
it: totals fell by exactly that file's contents — 40 messages, 78,770 output
tokens, nothing left behind. Retention is bounded by file existence as
documented, and a deleted session does not come back. Your reasoning for
rejecting my suggested shape holds, and the narrower rule is the better one.
Not covered by any of this: the concurrent-scan path you flagged as
untested, and cross-file ordering where a retained copy is less complete
than a live one. My harness is single-process and single-corpus, so it says
nothing about either.
The harness now takes
--bin, so any branch or unreleased main can bechecked the same way:
usage-tracker-audit/tokscale-drift-check.Happy to rerun it on anything.
I'll file the
input/tool_resultdouble count separately, as yousuggested.
junhoyeo commentedon Aug 3, 2026
Fixed in
v4.9.0via #1008.The measurement in this report is what made it actionable — the before/after table landing exactly on the drifted prediction pinned it to persistence rather than parsing, and saved us looking in the wrong place. It was reproduced in-tree before anything changed: three assistant turns, rewrite the transcript down to one, rescan, totals report one.
Observed messages are now retained from the source cache across an in-place rewrite.
One deliberate departure from your suggested shape: retention is not an append-only store that never deletes. That would resurrect sessions a user intentionally deleted, which is the production data loss
d9df8c9cexists to prevent. Retention is instead restricted to dedup keys that are stable across files, so a forked transcript replaying a tool result under a new session id cannot double-count it — that specific hazard was caught in review.Everything you verified as already correct — per-message accounting, the dedup key, the per-field max merge, the deep discovery walk — is untouched.
fix(claude): stop char-estimating tool_result input tokens (#1037)