Skip to content

fix(retain): bound entity-resolution candidate scoring (#3211) - #3213

Merged
nicoloboschi merged 1 commit into
mainfrom
rios/3211-bound-entity-candidate-scoring
Aug 7, 2026
Merged

fix(retain): bound entity-resolution candidate scoring (#3211)#3213
nicoloboschi merged 1 commit into
mainfrom
rios/3211-bound-entity-candidate-scoring

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #3211.

The symptom

Every fuzzy candidate is scored with difflib.SequenceMatcher in a synchronous loop on the event-loop thread, and candidate volume per query text was bounded only by what the trigram probe returned. On a bank whose index is polluted by many near-identical names, one resolution batch becomes minutes of uninterrupted CPU: /health cannot answer, the orchestrator kills the worker mid-op, and the requeued op wedges the next one.

Reproduction

A 100-mention batch driven through _resolve_from_candidates with mock candidates, alongside a 10 ms heartbeat task standing in for the health endpoint:

candidate rows before after
200k 2.13 s @ 99% CPU, 0 heartbeats, one 2.12 s stall 0.27 s, max stall 11 ms
1M 10.61 s @ 99% CPU, 0 heartbeats, one 10.6 s stall 0.38 s, max stall 14 ms

Cost scales linearly with candidates, which is how a real polluted bank reaches minutes.

The fix

  1. Cap candidates per query text in SQL, ranked by the similarity score the probe already computes — CROSS JOIN LATERAL (… ORDER BY similarity(…) DESC LIMIT $3) on PG, ROW_NUMBER() OVER (PARTITION BY q.query_text ORDER BY JARO_WINKLER …) on Oracle. New HINDSIGHT_API_RETAIN_ENTITY_RESOLUTION_MAX_CANDIDATES, default 200.
  2. Yield to the event loop every 256 scored candidates, so responsiveness does not depend on the cap being configured sanely.
  3. Backstop truncation in _resolve_from_candidates with an O(1)-per-candidate ordering key, for candidate sets built without a DB score (the full strategy's Python substring matching).

The new PG query was verified against a real PostgreSQL + pg_trgm on a throwaway instance: the cap is honoured per query text and the best trigram match ranks first.

Behaviour change beyond the cap

The PG rewrite drops SELECT DISTINCT ON (e.id), which deduplicated across query texts: an entity matching two mentions in the same batch was silently dropped as a candidate for one of them. On a 2-text / 2001-entity fixture the old query returned 1004 + 997 rows instead of 2001 each. The LATERAL form dedups per (query text, entity), which is what the grouping downstream expects.

Resolution results are otherwise unchanged for any mention with ≤ 200 candidates. Above that, candidates ranked below the top 200 by trigram similarity are no longer scored — they could previously win on the co-occurrence/temporal boost, but a match that far down the similarity ranking is noise.

Tests

hindsight-api-slim/tests/test_entity_resolver_candidate_cap.py — scoring is capped at the configured maximum, truncation keeps the exact match, a concurrent task keeps getting scheduled during a wide batch (the regression the issue asks for), and both dialects' SQL carries the cap.

@nicoloboschi
nicoloboschi force-pushed the rios/3211-bound-entity-candidate-scoring branch from 0bb2ea9 to 8934161 Compare August 6, 2026 16:11
Every fuzzy candidate was scored with difflib.SequenceMatcher in a
synchronous loop on the event-loop thread, and candidate volume per query
text was bounded only by what the trigram probe returned. On a bank whose
index is polluted by many near-identical names, one resolution batch
became minutes of uninterrupted CPU: /health could not answer, the
orchestrator killed the worker mid-op, and the requeued op wedged the
next one.

Measured on a 100-mention batch (mock candidates, 10ms heartbeat task):
1M candidate rows took 10.6s at 99% CPU with the heartbeat getting zero
turns; it now takes 0.38s with a max loop stall of 14ms.

- Cap candidates per query text in SQL, ranked by the similarity score
  the probe already computes: LATERAL ... ORDER BY similarity() LIMIT on
  PG, ROW_NUMBER() OVER (PARTITION BY query_text) on Oracle. New
  HINDSIGHT_API_RETAIN_ENTITY_RESOLUTION_MAX_CANDIDATES (default 200).
- Yield to the event loop every 256 scored candidates, so responsiveness
  does not depend on the cap being configured sanely.
- Backstop truncation in _resolve_from_candidates with an O(1)-per-
  candidate ordering key, for sets built without a DB score (the "full"
  strategy's Python substring matching).

The PG rewrite also drops DISTINCT ON (e.id), which deduplicated across
query texts: an entity matching two mentions in the same batch was
silently dropped as a candidate for one of them (on a 2-text fixture the
old query returned 1004 + 997 of 2001 matches instead of 2001 each).
@nicoloboschi
nicoloboschi force-pushed the rios/3211-bound-entity-candidate-scoring branch from 8934161 to a7224b2 Compare August 7, 2026 00:15
@nicoloboschi
nicoloboschi merged commit b5aec64 into main Aug 7, 2026
205 of 208 checks passed
@nicoloboschi
nicoloboschi deleted the rios/3211-bound-entity-candidate-scoring branch August 7, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolver: synchronous candidate-scoring loop can block the worker event loop for minutes on large candidate sets

1 participant