perf(server): bound orchestration replay and slow-client live buffers

[leoisadev1]

Problem

WebSocket catch-up replay loaded a global event range and filtered per chat afterwards. A stale cursor, or a handful of large tool payloads, could decode gigabytes and OOM the server. Slow clients could also retain live shell and thread events without a memory bound, including the batch waiting for an RPC ACK. Recursive event-store page concatenation kept consumed pages alive.

Adaptation

Thread resume now measures and reads only that chat's rows through a captured head, with a 1,000-event cap and an 8 MiB serialized payload budget. A recreated chat ID falls back to a fresh snapshot. Shell resume still uses the global range, but it measures payload bytes in SQL before decoding and snapshots when the gap or budget is unsafe.

Live shell and thread subscriptions share a per-subscription budget that includes queued, coalescing, and unacknowledged work. Overflow detaches the producer even without another pull and returns OrchestrationGetSnapshotError so the client can resume from its last sequence. Event-store pages are released as they are consumed.

Akeru bot, group, channel, and people fields on the shell snapshot stay in place. Missing-chat errors still say Chat, not Thread. Clients that omit afterSequence still get a snapshot, including pre-pagination clients.

Upstream credits

Reviewed adaptations of T3 Code:

• #8992 bound replay payloads
• #9521 start live thread consumers immediately
• #9715 slow-client live budgets
• #9726 thread-scoped replay
• #10777 release consumed replay pages

Local #16 bootstrap replay, #143 coalescing, and #198 raw-observer policy are preserved. Provider adapters are unchanged.

Scope

Server wire/replay only. Client stream lifetime, config bootstrap, reducer reference stability, and static HTTP caching are separate PRs.

Verification

Focused tests:

• LiveStreamBudget, ThreadLiveEventCoalescer, OrchestrationEventStore, ProjectionSnapshotQuery replay stats: 17 passed
• subscribeThread/subscribeShell overflow, ACK retention, thread-scoped catch-up, recreated-ID snapshot, deleted-chat shell removal: 18 passed
• Mock-updated unit tests: 212 passed
• vp run --filter akeru-bot typecheck passed
• Targeted lint on changed files passed (one existing unused helper warning in server.test.ts)

Isolated web client, home /tmp/akeru-replay-verify.VTW8Fb, populated fixture, worktree ports 6892/14932:

• Paired into live BotThreadLanding at /bots/ui-scout. Scout chat, roster, reply/react/read-aloud, and composer rendered.
• Reload on the same chat kept /bots/ui-scout and the same messages, so reconnect resumed the subscription instead of an empty load.
• 390x844 viewport still showed the Scout chat and composer.
• Second pairing token opened a second browser session to the same environment and landed on the same Scout chat while the first session stayed live.
• Create → New group opened the live group dialog. The populated fixture has one bot and no groups, so Create group stayed disabled. GroupThreadLanding was not a loaded chat in this fixture.

These stills are the screens after those actions. They do not replace the pair, reload, second-client, and viewport steps above.

Limitations

This PR does not change provider adapters. Effect stays at beta.103; Stream.paginate is available locally. Native mobile was not required for this server change.

Grok 4.6 High in Grok Build via Orca.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
akeru-bot-landing	Skipped		Sep 10, 2026 5:42pm UTC

[greptile-apps]

Greptile Summary

This change bounds orchestration catch-up and live subscription memory, adds thread-scoped replay statistics and reads, coalesces live thread updates under a shared retention budget, and releases consumed event-store pages. The outstanding shell replay path can still include an event newer than its captured replay head when persisted sequence numbers have gaps, then deliver that event again from the already-attached live stream.

Confidence Score: 4/5

Not safe to merge until the outstanding shell replay defect is fixed.

The prior shell replay finding remains outstanding. The replay call uses the difference between the captured head and client cursor as an event count rather than an upper sequence boundary. When sequence values are sparse, replay can include a newer event outside the captured range; because live delivery is attached first, that same event can then be delivered again from the buffered live stream.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for the posted P2 finding.
• T-Rex validated the current revision with a focused WebSocket RPC test, observing head 5, replaying with limit 5, and verifying that persisted replay produced sequences 1 and 6 with 6 published to the live stream and the shell output showing [6, 6, "synchronized"], while the code around line 2014 matched the prior commit, the diff to HEAD for apps/server/src/ws.ts was empty, and blame attributes the relevant lines to db4b2d8; a historical runtime execution attempt was blocked by the detached worktree's unresolved @effect/platform-node/NodeHttpServer dependency.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. apps/server/src/ws.ts, line 2014 (link)

   Unbounded sequence replay

   • Bug
     • Focused server/API-contract execution confirmed that readEvents(0, 5) can return sequence 6 when sequences 2 through 5 are absent, and that the pre-attached live stream then emits the same sequence 6 a second time.
   • Cause
     • readEvents forwards its second argument to the event store as a maximum event count, while the WebSocket catch-up path treats headSequence - afterSequence as though it were an inclusive sequence bound.
   • Fix
     • Use an event-store replay API that accepts toSequenceInclusive: headSequence, or filter replayed events to sequence <= headSequence before shell coalescing.

   _{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into leoisadev1/t3-p..." | Re-trigger Greptile}

[leoisadev1]

This is Leo's agent. Holding refreshed head 1d2a9ab despite green Repository checks: the actual refreshed Greptile assessment is 4/5 with an outstanding shell replay boundary finding. Independent source tracing corroborates the mismatch: ws.ts passes headSequence - afterSequence to readEvents; OrchestrationEngine forwards that as the row limit; the event-store SQL only applies sequence > cursor and LIMIT, without sequence <= captured head. The reviewer reports a real WebSocket sparse-sequence reproduction; I have not independently executed that reproduction. Required scoped repair: bound shell catch-up to captured head before coalescing, or expose a bounded sequence-range read, and add a committed deterministic WebSocket test with sequence gaps and a newer event already queued live. Assert that the newer event is delivered only once from the live tail, with no replay past the captured boundary. Preserve snapshot fallback and memory budgets; rerun focused tests and fresh PRGL. Existing 155 passing tests and browser synchronization checks do not cover this case. This path is inherited rather than introduced by the branch refresh, but it remains an unresolved acceptance finding. Orca is unavailable, so this comment is not verified worker delivery or executed repair.