Skip to content

e2e-native-pi lane flaky red since 2026-11-10: fake upstream's directive-queue key contains volatile ACP render-tag bytes, so a scripted fold replays the queue ("compress must be CALLED exactly once ... got 2") #6160

Closed

Description

@ranxianglei

What this is

The CI lane "real pi package-native lane vs deterministic fake upstream (zero tokens)" (.github/workflows/ci-e2e-native.yml, runs tests/e2e/e2e-native-pi.test.ts) has been failing intermittently on master since ~2026-11-10 19:30Z, with the two #2791 cross-protocol tests:

  • pi cross-protocol switch keeps the session, conv, and fold (#2791) — compress must be CALLED exactly once across the protocol switch (got 2) (tests/e2e/e2e-native-pi.test.ts:832)
  • pi openai→responses switch keeps the session, conv, and fold (#2791 responses column) — the fold must be REUSED, not re-executed (compressCalls=2) (:916)

Not introduced by any in-flight PR: clean master (58a8796) reproduces the exact same assertion locally (node 25.9 + pi-stable 0.83.6, deterministic), and master's scheduled CI runs (11-10 19:38Z run 42887264198; 11-11 00:35Z run 42932505675) fail the same way. The same SHA goes red and green across CI runs (e.g. 42848363612 green 15:49Z vs 42892982144 red 20:12Z; snapshot-foundation's e17f18c red at 20:34Z, green at 00:27Z), and ~20 other PRs in the same window are green — a timing race, not a tree regression.

Root cause (wire-verified)

The fake upstream (tests/e2e/fake-upstream-chat.mjs) keys its scripted directive queue by convKey|sourceText.slice(0,64) where sourceText is the LAST user message text including its ACP render tag. That tag is volatile BY DESIGN:

  1. The scripted fold (run 2, compress on m00001) succeeds → stream.ts sets pendingLedgerAlign → the next scheduleSave runs maybeRestampToLedger (src/persist.ts) → restampRefsToLedger (After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 layer 3, "exactly once per state").
  2. # Cache hit collapses to 1–12% for a whole generation, then the very next request returns to 100% — repeated 15× in one day #5608-D (owner ruling 2026-11-06) defers that alignment to the session's NEXT SUCCESSFUL FOLD so the renumbering rides the fold's prefix break free — so it lands exactly one request after the scripted compress.
  3. The alignment (a) renumbers old refs onto store-row seats (the filler's tag went m00001 → m00015, refs-never-reused) and (b) RAISES timestampsHighWaterRefSeq to the aligned max (kernel/src/refs.ts, # Cache hit collapses to 1–12% for a whole generation, then the very next request returns to 100% — repeated 15× in one day #5608-C: "so none of them suddenly gains a time attribute"), so the stamp node (kernel/src/compress.ts:1229, seq <= hwm → no ts) drops time= from every pre-fold message's tag.
  4. Wire evidence (local repro, fake reqlog): request [6] sends <acp tokens="17" type="text" time="2026-11-11 09:06:49">m00011</acp>\n请调用probe_tool;请调用compress;请调用acp_status as the last user message; request [7] — same conversation, same message — sends <acp tokens="17" type="text">m00011</acp>\n... (time= gone). The 64-char queue-key slice contains the tag → new key → the fake re-parses the directives from the same prompt → fresh queue [probe_tool, compress, acp_status] → request [8] emits a SECOND compress call (no-op: the re-anchored range matches the existing block) → got 2.

The green runs are the race side: when the unified store has no visible rows for the session at the post-fold scheduleSave (persistence timing on loaded runners), maybeRestampToLedger no-ops, no renumber/hwm raise happens, the tag stays byte-stable and the queue survives.

The product behavior (one-time tag rewrite at the first fold) is the accepted cost documented by #5608-C/#5608-D — no product change requested here.

Fix

The harness must not key a conversation turn on volatile tag bytes. In answerFor() the queue key becomes convKey|<source message ref>|<tag-stripped body>.slice(0,64):

  • the ref names the message OCCURRENCE — stable across the turn's requests (a mid-turn remint voids the script anyway: the message's bytes are in a fold block), and keeps distinct prompts distinct even when their bodies are byte-identical (the switch/back runs both send 请调用probe_tool);
  • stripAcps (the helper already present for title-call detection) removes the volatile tag wrapper from the keyed bytes.

One change, tests/e2e/fake-upstream-chat.mjs — covers all six suites that share the fake (native pi/dsh/opencode, delegate-notify, subagent-sessions).

Verification

  • Both dsh anthropic-messages lane bypasses native-intercept fetch patch: anonymous pfa-* session, fold ledger orphaned, silent ~182s re-fold on cross-protocol switch #2791 tests pass locally (previously deterministic fail): the fold run's queue now continues through the fold — probe (qIdx 0) → compress (qIdx 1, real fold) → acp_status (qIdx 2) → done, exactly one compress call; switch/back runs keep their own per-run queues (distinct refs).
  • Full local regression pass over all fake-consuming suites: e2e-native-pi 7/8 (the /acp stderr-assertion failure is a pre-existing local-environment artifact — identical on master, unrelated), e2e-native-dsh 2/2, e2e-native-opencode 4/4, e2e-delegate-notify 2/2, e2e-subagent-sessions 1 failure reproduced identically with the UNMODIFIED fake (pre-existing local artifact, not this change).

Fix lands as a follow-up commit on the PR #6150 branch (the lane must be green for that PR to merge); tracked here for the record.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions