Repository navigation
Commit b705853
authored
Render dropped rerank failure detail; count DegradeFlags to catch silent degradation (#11)
* Interpolate caught error detail into rerank/search log messages
RerankHttpError already builds a fully diagnostic message (status, URL,
body snippet), and search.ts already passed it through as a structured
`error` context field on log.warn/log.error calls — but the configured
log sink doesn't render context fields for these call sites, so the
detail never reached a human. This is what let CL-4600's 100% rerank
failure rate read as "retrieval is just a bit weak" for as long as it
was deployed.
Fix: interpolate the caught error's message directly into the log
template at every affected call site, via a new formatCaughtError()
helper (src/log.ts), while keeping the structured context field for
sinks that do render it. Audited and fixed all catch sites in the
search path with the same pattern: dense-retrieval fallback, all three
rerank catch branches (RerankQueryTooLongError / RerankConfigError /
generic), and the outer route-level catch in routes/search.ts.
No behavior change — soft-degrade to fused ranking is preserved.
Claude-Session: https://claude.ai/code/session_017GTgGzn5xAwvkU2GAPAHpF
* Emit a counter per DegradeFlag so silent degradation is visible
Reranking degraded on 100% of queries for as long as CL-4600 was
deployed and nobody noticed, because DegradeFlag values were logged
per-request but never aggregated — a flag nobody counts is
indistinguishable from no flag at all.
There is no metrics facility anywhere in this repo (no statsd,
prometheus, or otel dependency; @intx/log exports no counter), and this
is a mountable library, not a standalone service that owns its own
/metrics port — so this adds none.
New src/core/degrade-metrics.ts: an in-process counter per DegradeFlag
plus a totalSearches counter (so a rate, not just a raw count, is
computable), exported as getDegradeMetricsSnapshot() for a host with
its own metrics backend to poll and forward, and re-exported from the
package entrypoint. Rather than a setInterval, a structured summary log
line fires every 100th call inside recordDegrade() itself, and escalates
from log.info to log.error when a single flag has saturated the entire
window — the exact shape of the incident that motivated this ticket.
recordDegrade() is wired into hybridSearch once per invocation,
immediately before return, covering every degrade path (dense down,
rerank down, rerank query-too-long) and the fully healthy case.
Claude-Session: https://claude.ai/code/session_017GTgGzn5xAwvkU2GAPAHpF
* Rework degrade-metrics to detect near-neighbour incident shapes
Replace the binary "every call in a fixed 100-call bucket" saturation
check with a per-tenant, rate-based rolling window (200 calls) with
hysteresis (20%/10% watermarks), so 99% failure, single-tenant outages
among many tenants, and mid-window onset all escalate instead of only
the exact 100%-of-100 shape from CL-4600. Escalation is checked on
every call rather than only at the periodic-summary boundary.
Key recordDegrade/getDegradeMetricsSnapshot by tenant with an LRU
eviction over tracked tenants to bound memory. Add
getAllDegradeMetricsSnapshots for hosts that poll every tenant. The
snapshot now exposes a windowed rate alongside the lifetime cumulative
counts, since cumulative-only numbers dilute a live incident into an
unremarkable average.
Interpolate every log message's numbers into the message string
itself rather than relying on the context object, since some
@intx/log sinks drop context — the same bug this PR's earlier commits
fixed at routes/search.ts is also present at routes/capture.ts,
embed-model-registry.ts, services/capture.ts, services/transform.ts,
and a bare catch in knowledge.ts that discarded the error entirely.
Fix all of them.
Make ALL_DEGRADE_FLAGS exhaustive at compile time via a `satisfies
Record<DegradeFlag, true>` object, so a new flag added to
hybrid-search.ts without updating this module is a compile error.
Decouple SUMMARY_INTERVAL (reporting cadence) from the detection
window, and expose both plus the watermarks via
configureDegradeMetrics so a host can tune them.
Adopt the reviewer's scenario tests (mid-run onset, 99% failure,
cumulative dilution, multi-tenant isolation) into the real suite as
assertions against the fixed behavior.
Claude-Session: https://claude.ai/code/session_017GTgGzn5xAwvkU2GAPAHpF
* Guard degrade-metrics escalation against cold-start noise and fix exports
A rate/hysteresis redesign closed the 99%/multi-tenant/mid-onset gaps
but had no minimum-sample floor, so a cold or freshly-evicted tenant
could escalate off one or two calls at small-sample noise. Hold
escalation evaluation until a tenant's window has at least
minSamplesFor(highWatermark) samples (32 at the default 20%
watermark), derived from the n*p*(1-p) >= 5 rule of thumb for when a
binomial proportion's normal approximation is trustworthy.
Export configureDegradeMetrics and getAllDegradeMetricsSnapshots from
the package entrypoint — the PR body promised a tunable operator
surface, but only getDegradeMetricsSnapshot was reachable from outside
core/degrade-metrics.ts.
Validate every configureDegradeMetrics field, not just
highWatermark > lowWatermark: reject non-integer or non-positive
windowSize/summaryInterval/maxTrackedTenants (windowSize=0 previously
made the window grow unbounded via slice(-0); summaryInterval=0
divided by zero and silenced summaries permanently) and
out-of-[0,1)/[0,1] watermarks.
Stop treating LRU eviction as tenant-blind: never evict a tenant that
is currently escalated (falling back to evicting it anyway, loudly,
only once every tracked tenant is escalated), and log every eviction
so a tenant disappearing from getAllDegradeMetricsSnapshots is visible
rather than silent.
Replace the per-call O(windowSize) rate re-scan with a fixed-size
circular buffer plus running per-flag counts, since the rate is
computed up to 3 flags x (escalate-check + summary) times per search.
Add tests for cold-start/low-sample escalation, an idle tenant's
escalated state surviving in its snapshot, eviction during a live
incident, and configureDegradeMetrics validation.
Claude-Session: https://claude.ai/code/session_017GTgGzn5xAwvkU2GAPAHpF
* Close configuration paths that could disable degrade escalation
Round 2's minimum-sample floor stopped cold-start false positives but
introduced three config-dependent ways to silence the alarm entirely:
highWatermark=1 makes the floor's denominator p*(1-p) zero (Infinity
samples needed, so escalation never evaluates); a windowSize smaller
than the derived floor for a given watermark is unreachable the same
way; and shrinking windowSize on a live tenant left its buffer length
and windowFilled counter out of sync, producing a windowedDegradeRate
above 1.0.
Add one invariant, enforced in configureDegradeMetrics/validateConfig:
no config the validator accepts may make escalation unreachable. The
derived minimum-sample floor for highWatermark must be finite and must
not exceed windowSize, or the config is rejected outright. This
subsumes highWatermark=1 (non-finite floor) and any windowSize/
highWatermark pairing where the floor can't fit in the window, rather
than special-casing either symptom.
Resize every tracked tenant's rolling window in place when windowSize
is reconfigured, rather than leaving existing tenants on their
creation-time buffer length: cumulative totals survive, the window
resets to empty and re-accumulates under the same minimum-sample floor
as a new tenant.
Adopt the reviewer's probes (watermark=1, window-smaller-than-floor,
shrink-produces-rate>1) as permanent tests, plus new coverage that
specifically reconfigures a watermark or windowSize and asserts
escalation still fires afterward — the gap that let all three survive
review. tmp/critique-tests/ removed.
Claude-Session: https://claude.ai/code/session_017GTgGzn5xAwvkU2GAPAHpF1 parent b3ab858 commit b705853
12 files changed
Lines changed: 1015 additions & 57 deletions
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| 105 | + | |
105 | 106 | | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
| 107 | + | |
| 108 | + | |
111 | 109 | | |
112 | 110 | | |
113 | 111 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
33 | 43 | | |
34 | 44 | | |
35 | 45 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
| 90 | + | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
94 | | - | |
95 | | - | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
96 | 98 | | |
97 | 99 | | |
98 | 100 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
| 64 | + | |
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
517 | 517 | | |
518 | 518 | | |
519 | 519 | | |
520 | | - | |
521 | | - | |
522 | | - | |
523 | | - | |
524 | | - | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
525 | 524 | | |
526 | 525 | | |
527 | 526 | | |
528 | | - | |
529 | | - | |
530 | | - | |
531 | | - | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
532 | 531 | | |
533 | 532 | | |
534 | 533 | | |
535 | 534 | | |
536 | | - | |
537 | | - | |
538 | | - | |
539 | | - | |
540 | | - | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
541 | 540 | | |
542 | 541 | | |
543 | 542 | | |
| |||
0 commit comments