Openconceptlab/ocl_issues#2505 | Fix US/British spelling mismatch in semantic concept search

[snyaggarwal]

Refs OpenConceptLab/ocl_issues#2505 (Problem B only — Problem A still open)
Refs OpenConceptLab/ocl_issues#2511 (Lexical Variant Dictionary infrastructure)

Heads up — approach changed mid-flight

Sunny — apologies for the redirect. After deeper review of #2505 with @paynejd, we landed on a substantially different architecture and felt it was faster to push the new approach directly than to iterate via comments. Your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept — only the variant source changed from inline regex to a dictionary-driven lookup. Happy to walk through the diff on a call if useful.

Three reasons for the redirect:

1. False positives in the original regex list. hem/haem matches themselves, anthem, hemisphere, hemp, hemlock, remember — hard to fix defensively in regex without enumerating exclusions for every English word.
2. Reusability. The fix should be available beyond ConceptFuzzySearch. Standard /concepts/?q=… has the same gap, and upcoming abbreviation-expansion work needs the same primitive. A dictionary-as-Source approach makes that reuse trivial.
3. Eat our own dog food. Storing dictionaries as OCL Sources gives versioning, release management, locale support, editability through the OCL UI, and discoverability for free — and aligns with how UMLS / SNOMED / OBO model lexical variants.

Tradeoff worth flagging

This approach puts pressure on oclapi2 to do caching well. The MVP scale is trivial (~40 entries, in-memory dict), but as dictionary types and sizes grow — especially abbreviation dictionaries (UMLS LRABR is ~250k rows) and once variants are enabled by default on high-QPS endpoints — the per-token DB lookup will need to graduate beyond the MVP's module-level cache. Likely Redis-backed token-indexed cache for >1k-entry dictionaries; ES-backed lookup for very large ones. Plan to revisit when (a) a dictionary exceeds ~5k entries, or (b) variants are enabled by default on a high-QPS endpoint. Captured in #2511.

What this PR does

• New helper: core/common/lexical_variants.py — get_lexical_variants(text, source_uri) and get_variant_terms(text, source_uri). Tokenizes input (lowercase + whitespace + ASCII punctuation strip), looks each token up against the dictionary's Names, returns sibling Names from the matching Concept. Module-level cache keyed on (source_uri, version). Returns [] and never raises if the dictionary is missing — graceful degradation.
• Seed data: core/common/data/lexical-variants-en.json — OCL bulk import file creating OCL/lexical-variants-en (Source + 43 vetted Concepts + Source Version v1.0). Each Concept has en-US and en-GB Names with name_type=Fully Specified. False-positive-prone substring entries dropped.
• Wired into $match: MetadataToConceptsListView reads variants from request body, passes the resolved URI (or None) to ConceptFuzzySearch.search(..., variants_repo=...). The kNN sub-query construction and rescore expansion are unchanged in shape; only the variant source changed.
• Empty-synonyms rescore crash fix: preserved from your initial commit.
• Removed: get_spelling_variant() from core/common/utils.py.

API: variants request body field

Lexical variant expansion is OFF by default. Clients opt in. Same shape will apply to standard concept search when that wiring lands as a follow-up.

Value	Behavior
absent / null / false / "false" / "0"	disabled (default)
true / "true" / "1"	enabled, default dictionary (OCL/lexical-variants-en)
URI string (e.g. "/orgs/myorg/sources/my-dictionary/")	enabled, that dictionary

Tests

core/common/tests.py LexicalVariantsTest covers:

• Tokenization
• Multi-token expansion
• Cache lifecycle (per-source-version) + invalidation
• Missing-source graceful degradation
• Regression: themselves, anthem, hemisphere, hemp, hemlock, remember return no variants

Deployment

The dictionary Source has already been seeded on prod (OCL/lexical-variants-en v1.0, 43 concepts, public read) so the wiring will work as soon as this PR merges and deploys. For dev / qa / staging environments, run:

ocl --server ocl-dev import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-qa import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-staging import file --wait core/common/data/lexical-variants-en.json

Verify with: ocl repo get OCL lexical-variants-en (should show 43 concepts, source_type: "Lexical Variants", extras.dictionary_kind: "lexical_variant", version v1.0 released).

The helper degrades gracefully (returns [], never raises) if the Source doesn't exist yet — so a deploy gap on any environment won't break search.

Out of scope (deferred follow-ups)

• ?variants=... on standard concept search (ConceptListView) — touches CustomESSearch / get_raw_search_string layer, smaller blast radius as a separate PR
• $lexical-variants HTTP operation
• Abbreviation dictionary type (Mapping-based)
• Multi-dictionary composition
• Problem A from #2505 (CIEL bridge exact-match scoring via apply_score() highlight) — entirely separate work; this PR no longer auto-closes that part of the ticket

Architecture / design references

• Full design proposal: Lexical Variant Dictionaries — UMLS / SNOMED / OBO precedents, dictionary-type modeling decisions, tokenization-first reasoning, multi-dictionary roadmap
• Infrastructure tracking ticket: #2511 — Phase 2/3 work (standard search wiring, $lexical-variants operation, abbreviation dictionary type, caching strategy, etc.)

[paynejd]

@snyaggarwal — heads up, I pushed two commits on top of yours that redirect this PR to a dictionary-driven approach (full rationale in the updated description above + the proposal doc).

TL;DR — your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept. The variant source changed from inline regex to a dictionary lookup against ocl/lexical-variants-en (a new OCL Source seeded by bulk import). Variant expansion is now off by default; clients opt in via variants: true (or a custom dictionary URI) in the request body. We'll need follow up work to utilize this in Mapper (expose in project config?).

Apologies for the in-flight redirect — the false-positive cases in the regex list (themselves, anthem, hemisphere, hemp, etc. all matching hem/haem) plus the realization that the same primitive will be needed for upcoming synonym/abbreviation/keyword-expansion evaluation that we're starting pushed me to the dictionary-as-Source approach. Phase 2/3 work tracked in #2511.

Happy to walk through the diff or rework anything that doesn't sit right.