[RUM] Direct-to-instance :9925 connectivity failures at 3.4× baseline (19 sessions, 11 orgs) — and the whole class is invisible in Error Tracking by design

[dawsontoth]

Summary

Direct-to-instance operations calls (POST https://<instance-host>:9925/) are failing at the
network level (status_code:0, i.e. the request never completed) at 1.93% of all :9925
calls in the last 24h — 218 failures across 19 sessions and 16 distinct instance hosts in 11
organizations. The trailing-7d rate is 0.57%, so this is ~3.4× baseline.

69% of those failures (151 of 218) burn the full 60s client timeout before the UI learns
anything, on queries whose poll interval is 5–10s.

Separately, and independent of whether the rate above is a Studio problem at all: none of
these failures produce an Error Tracking issue. shouldKeepEvent drops connectivity-class
failures unconditionally, so a fleet-wide reachability degradation is invisible to Error
Tracking and to any alert built on it. The only trace is @resource.status_code:0, which
nothing currently monitors. That is why a 3.4× elevation across 11 orgs surfaced only via a
manual review.

Surfaced by the automated daily RUM review (app f590deee-…).

Datadog findings

@type:resource @resource.url:*:9925* @resource.status_code:0

Window	:9925 calls	status-0	rate	sessions
24h → now	11,284	218	1.93%	19
7d → 24h ago	27,616	158	0.57%	—
14d → 7d ago	44,888	485	1.08%	—

Breadth in the 24h window: 16 distinct instance hosts, 11 organizations, 19 of 118 sessions
(16%). 214 of the 218 are POST /; the other 4 are GET /health.

Duration distribution of the 218 failures — the spike at 60s is the axios client timeout
(src/config/getInstanceClient.ts:68, timeout: 60000):

Duration	Count
~60s (timeout)	151
8–30s	22
<2s	44
other	1

What the per-session signature does and does not show

Worst single session: 77 failures over ~23 minutes against one host. Timeline (host and
session identifiers withheld):

16:40:31 → 16:40:53   23 × POST /  → 200, 25–533ms   (p50 253ms)
16:41:03              host goes dark
16:41:35 …            bursts of 6–7 concurrent POSTs, each hanging 60s,
16:49:42              repeating every ~61s
16:51:33               7 × fail fast (13.7s) — host now actively refusing
17:03:55              13 × 60s hang

I want to be explicit that this session does not demonstrate a Studio retry bug, because
that was my first read and the data refuted it:

• The 23 successes all precede 16:41:03; they are not interleaved with the hangs. So the
  host genuinely became unreachable and stayed unreachable — the "instance is down,
  restarting, or unreachable" case that shouldKeepEvent and pollUnlessForbidden both
  explicitly treat as an expected state, not a bug.
• The observed cadence is self-limiting: because React Query will not refetch a key while
  a request for it is in flight, and each request occupies the full 60s timeout, the 5–10s
  poll timers degrade to ~1 request per key per 60s. ~3.3 requests/min across ~7 polling
  queries is not runaway amplification.
• I also checked, and disproved, a connection-pool-starvation theory: successful requests
  to the same host in the same session completed in 253ms p50 / 533ms max, so the stalled
  requests were not blocking siblings.

So the elevated rate is most likely an instance/infrastructure reachability signal rather
than a Studio code defect. It is filed here because Studio's RUM is currently the only place
it is visible at all.

The observability gap (this part is a Studio-side defect)

src/integrations/datadog/shouldKeepEvent.ts:122-129:

// A request timeout is a connectivity-class failure, never a Studio bug: the
// instance, cluster, or backend was too slow to answer in time. …
// Backend latency is tracked server-side, not from the browser.
if (/timeout of \d+ms exceeded/i.test(message)) {
	return false;
}

Confirmed empirically: @type:error @error.message:*timeout* returns 0 events in 24h
against 218 resource-level failures.

The per-event rationale is sound — these did flood Error Tracking (#1371). The gap is that
dropping them left no aggregate signal in its place. "Backend latency is tracked
server-side" does not cover client-observed reachability: whether a browser can reach a
customer instance on :9925 is only observable from the browser.

Open questions for a human

1. Is a 60s timeout right for a 5–10s poll? getInstanceClient is shared by polls and by
   genuinely slow operations (large queries, deploys), so a blanket reduction would risk
   breaking the latter. A per-call or per-query-type timeout is the obvious shape, but this
   touches core shared behavior and I did not want to change it unilaterally. 69% of failures
   currently cost a user 60s of indeterminate loading.
2. Should there be a monitor on @resource.status_code:0 for first-party hosts? That is
   the signal that would have caught this without a manual review, and it does not reintroduce
   the [Monitoring] AxiosError: API request timeouts (15–60s) affecting multiple views (regression since v2.111.3) #1371 error flood because it is a resource-rate monitor, not an Error Tracking issue.
3. Is the underlying reachability drop known to infra? 16 hosts across 11 orgs in 24h
   suggests something broader than individual customers stopping instances, but I cannot
   distinguish those two from RUM alone — the operation name lives in the POST body, which RUM
   does not capture.

Related, not duplicate

Same endpoint, different failure modes: #1602 (400s on :9925), #1527 (500s on proxied
/operation), #1630 (404s on Cluster/{id}/operation), #1603 (poll fan-out volume). None of
those cover status_code:0 / timeouts, and all of them are visible in Error Tracking, which
this class is not.