Skip to content

import: bank import never classifies label entities, silently disabling the #3208 partial trigram index #3236

Description

@nicoloboschi

Problem

import_bank_async resolves the target bank's config before the archive's bank row is restored, so entity_labels is always empty for the whole import and every label entity is stored with entity_kind='regular'.

# memory_engine.py — import_bank_async
resolved_config = await self._config_resolver.resolve_full_config(bank_id, request_context)
return await import_bank(..., config=resolved_config, ...)

The bank cannot exist at that point: import_bank explicitly refuses to write into an existing bank ("import-bank restores into a fresh bank (it is not a merge)"). So the resolve can only ever see global + tenant config — the bank's own entity_labels, which the archive carries in banks.json, arrives later, when import_bank restores the bank row and then replays the documents through retain Phase 1 with the stale config object.

This contradicts what #3214 assumed: "Bank import needs no changes: transfer archives treat entities as derived data and re-resolve them through the standard retain Phase 1, which now stamps the kind." Phase 1 does stamp the kind — but from getattr(config, "entity_labels", None), which is empty here.

Impact

  1. entities: exclude exact-match-only label entities from the trigram fuzzy-match index #3208/perf(entity-resolution): exclude label entities from the trigram fuzzy-match index #3214 is silently inactive on any imported bank. Label rows keep entity_kind='regular', so the partial trigram index ... WHERE entity_kind != 'label' excludes nothing and every fuzzy probe pays the recheck-discard cost the fix was meant to remove.
  2. Label values are fuzzy-merged during the import itself — the exact-match-only path (perf(entity-resolution): skip fuzzy probing for exact-match-only label entities #3187) is keyed off the same classification, so distinct label values can collapse into each other, which is what Memory Tags added when Entities are not during retain #1558 forbids.
  3. The migration backfill does not save you: it runs once, at migration time. A bank imported afterwards is misclassified with nothing to correct it.

Evidence

Importing a real 862-document / 38k-fact production export (bank has a free-text brief_bio label group) into a fresh instance on main @ f9fb3e9:

entities_total 12300 | by_kind {'label': 500, 'regular': 11800}

5,374 of those entities are label-shaped, and the only 500 marked label were created by a post-import ingest burst, not by the import. Applying the same classification the migration backfill uses (is_label_entity against the bank's config) marks 5,355 rows as labels.

Measured cost on that bank (identical population, both sides REINDEXed, all other banks dropped so the shared trigram index is not polluted):

as imported after correcting the kinds
entity-resolution p50 (25 names) 263 ms 37 ms
single fuzzy probe (EXPLAIN ANALYZE) 8.28 ms 0.38 ms
candidates pulled → kept 1,830 → 128 252 → 56

So the defect forfeits roughly 7× of the win #3214 delivers, on the exact bank the feature was built for.

Fix

Re-resolve the config after the bank row is restored, and replay documents with that. import_bank takes a resolve_config callback that import_bank_async supplies; the pre-restore config stays as the value used for everything up to the restore.

Regression test: test_bank_import_classifies_label_entities — configure entity_labels, export, delete, import, assert the label entity comes back as entity_kind='label' and a regular one as 'regular'. It fails on main with 'regular' != 'label'.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions