Repository navigation
RFC: is roll-triggered Ambiguous/Stale reclassification overly broad? (follow-up to #790) #793
Description
Context
Follow-up from #790. That issue's proximate trigger (a keepalive interval far too aggressive for reattach — fixed in #792) is resolved. But @pedrosakuma's original pushback deserves its own issue, independent of the timer bug:
continuo vendo o roll como um critério que não faz sentido. ficar desconectado x tempo nao implica que devemos fazer o roll [...] o fixp supostamente ja deve garantir que as mensagens seriam recebidas apos reconexao. caso isso nao se concretize por algum motivo ai sim é cenario para roll e reconciliação via adm.
[...] mas peço que abra uma issue sobre o motivo do roll. sinceramente vejo como um fluxo que é mais danoso do que qualquer coisa.
The concern
Today, any confirmed session roll (venue rejects Establish-reuse and issues a fresh SessionVerId, ReconnectKind.Renegotiated) triggers PendingNewReapingConnectRollReactor (backend/src/B3.Trading.Application/ConnectSessionRollReactor.cs), which unconditionally:
- Reclassifies every in-flight
PendingNewoutbound mutation for that firm asAmbiguous— resolvable only via manual operator action throughOutboundReconciliationService(backend/src/B3.Trading.Application/Outbound/OutboundReconciliationService.cs). There is no automatic retry, gap-fill probe, or venue query attempted first. - Flags every surviving
Working/PartiallyFilledorder for that firm STALE.
This fires on every roll, regardless of why the venue rejected the reuse — a genuine multi-minute outage where the venue plausibly discarded state, a brief blip where the venue's own idle-timeout was simply too tight (#790/#792), a venue-side restart/failover, or any other transient cause. The blast radius (every in-flight mutation + every working order for the whole firm, requiring a human in the loop) is the same regardless of severity.
@pedrosakuma's view: this is backwards. FIXP's own reconnect/retransmission machinery (Establish-reuse, sequence retransmit window) exists precisely so a disconnection of some bounded duration recovers cleanly (Reattached) without discarding anything. A session roll should be rare and load-bearing evidence that the venue truly lost state — not something we should routinely expect or design our resilience story around. Given how disruptive the current roll handling is (blanket Ambiguous/Stale, no auto-remediation, needs an operator), we may be over-triggering it, or over-reacting to it once it does trigger, or both.
What to investigate (RFC, no fix implied yet)
This repo's convention (AGENTS.md, "RFC-first for ambiguous source-of-truth changes") is to investigate before designing, given this touches order-state source-of-truth semantics. Questions to resolve here, informed by the actual FIXP spec / B3EntryPointClient SDK / B3MatchingPlatform behavior (not assumption):
- What venue-side conditions can legitimately produce a confirmed roll (
UNNEGOTIATEDreuse-reject) in production, beyond the keepalive-idle-timeout case just fixed? (e.g. venue process restart/failover, explicit admin session reset, genuine multi-region failover with state loss.) Are all of these truly "venue lost state," or are some of them also just venue-side conservatism that could be relaxed/negotiated away? - Is
Ambiguous/Stale-on-every-roll the right blast radius, or should the reactor distinguish "venue confirmed it has no record of this session at all" (justifies the current blanket treatment) from "venue rejected reuse for an unrelated/policy reason but plausibly still has recent history" (might allow a narrower reconciliation, e.g. a targeted status/gap query per order before giving up and marking Ambiguous)? - Is there a lighter-weight recovery path available before falling back to full manual reconciliation — e.g. a per-order status query /
OrderStatusRequest-equivalent against the venue immediately after a roll, so orders the venue actually confirms are resolved automatically and only genuinely indeterminate ones becomeAmbiguous? - Should the roll-detection criteria itself be tightened (e.g. distinguish "reuse rejected because keepalive/idle lapsed under normal negotiated parameters" — which, post-fix(#790): raise default FIXP keep-alive interval so reattach can survive a real disconnection #792, should now be rare/require a genuinely long outage — from other reuse-reject reasons) so operators can reason about why a roll happened, not just that one did?
- Cross-reference the upstream evidence gate (AGENTS.md): what does the merged
B3EntryPointClientSDK andB3MatchingPlatformofficially guarantee about message delivery across a reconnect within the negotiated retransmit window, and what, precisely, is the contract once that window is exceeded? This determines how much of items 1-4 above is actually addressable in this repo vs. inherent to the protocol.
Non-goals for this issue
- Not proposing to remove
Ambiguous/manual reconciliation as a concept — a case where the venue genuinely lost state legitimately needs an operator in the loop; WAL is the source of truth, not venue session identity (perConnectSessionRollReactor.cs's own doc comments). - Not re-litigating ci: "Chaos drill" network-partition scenario fails 100% of the time — post-recovery orders never reach Filled #790/fix(#790): raise default FIXP keep-alive interval so reattach can survive a real disconnection #792's keepalive-interval fix, which is merged/validated separately.
Suggested next step
Dispatch a research pass over the B3EntryPointClient SDK + B3MatchingPlatform sources (both available locally) to answer question 1 and 5 above with actual evidence (commits, session-state-machine code, any documented reconnect/retransmit guarantees), then come back here with findings before proposing any design change to ConnectSessionRollReactor/OutboundReconciliationService.
cc @pedrosakuma
Activity
pedrosakuma commentedon Sep 4, 2026
Research pass: B3EntryPointClient + B3MatchingPlatform evidence for Q1/Q5
Per the issue's suggested next step, dispatched a research pass over both local sibling repos (
B3EntryPointClient,B3MatchingPlatform) plus the reactor/reconciliation code here. Findings below (file:line cited throughout in the full report).Q1 — What venue-side conditions produce a confirmed roll?
Traced every codepath in
FixpStateMachine/EstablishValidator/FixpSession.Lifecyclethat rejects Establish-reuse and forces a freshSessionVerId:SuspendedTimeout(TCP drop, 5 min reaper)KeepaliveLapsed(peer silent ~3×keepAlive, ~45s w/ #792 defaults)DailyReset(scheduled/operator)Reattached, no roll, reactor never firesINVALID_SESSIONVERID(stale verId after takeover)Key finding: no codepath was found where a verId bump fires while the venue retains matching-session state in a reachable form. The one case with attached uncertainty is default-config process restart (
retransmitPersistenceDir: ""in the Helm chart,deploy/charts/b3-matching/values.yaml:132) — the FIXP session is genuinely gone, but the underlying order book WAL may still be intact. Our own runbook already flags this as advisory (docs/operations/runbook-failover-recovery.md:445-450), andConnectSessionRollReactor.cs's own doc comments already accept this conservatism ("session identity is transport evidence, not authoritative proof").Q5 — What does FIXP retransmission actually guarantee across reconnect, and what happens once exceeded?
Reattachedreconnect, the venue guarantees recovery of all outbound frames still inside the retransmit window: an in-memory ring (default 1024 frames) plus, if persistence is wired, a cold journal (default 256 MiB / 24h retention).RetransmitRequest(FIXP §4.5.6) serves up to 1000 frames per request; falling outside the window returnsOUT_OF_RANGEand the client's own gap detector/OrderStaleningVenueReactorhandles it independently of the roll reactor.Renegotiated(rolled) reconnect, the SDK resets all inbound/outbound sequence tracking (EntryPointClient.cs:1897-1902) andRetransmitWindowReady = false— there is no in-band way to recover the old session's messages, because the new session starts at seq 1 under a newSessionVerId. This is the concrete, spec-level reason the Ambiguous/Stale reactor fires only on roll: FIXP's own retransmit machinery is architecturally incapable of bridging aSessionVerIdchange, so "was this order acknowledged" becomes genuinely undecidable in-band once a roll happens.Conclusion so far: the evidence does not support "the roll criteria is over-triggering" — every traced trigger is a real forfeiture of the FIXP retransmit window, not venue-side over-conservatism we control. The one nuance (default-config restart) is already handled as an advisory/clearable stale flag, not a hard-blocking one.
What would still narrow the blast radius (per Q3/Q4, not yet designed)
A post-roll per-order status reconciliation probe (an
OrderStatusRequest-equivalent against the venue, immediately afterRenegotiated) is the one lever with real evidence supporting it — it wouldn't change whether the reactor fires, but could auto-resolve individualAmbiguousmutations without waiting on an operator, for any order the venue can positively confirm post-roll. Whether B3MatchingPlatform's FIXP session layer exposes anything equivalent (vs. only ER/admin flows) was not confirmed and would need its own targeted look before scoping a follow-up implementation issue.Evidence gaps (open questions, need confirmation before any design change)
retransmitPersistenceDir(vs. the disabled Helm default) — determines how often restarts hit the ephemeral-vs-persisted path in practice.ExchangeHost.cs) uses aCloseKind.HostShutdown-equivalent that preserves state/journal — would mean rolling restarts don't need to roll at all.NotApplied/OutboundReconciliationRequiredwedge reproduced in commite5fcb06(FIRM01 sandbox incident) — unclear if this is the specific trigger for this RFC or a separate concern.TerminateAllSessions.Full report with file:line citations and commit hashes available on request — kept this comment to the actionable summary. No code change proposed here per this issue's RFC-first scope; next step would be a targeted follow-up on gap #2 (graceful shutdown persistence) since that's the cheapest lever to reduce roll frequency without touching
ConnectSessionRollReactorsemantics at all.pedrosakuma commentedon Sep 5, 2026
Follow-up issues opened from this research pass: