feat(query-analyzer): configurable dateparser locale detection

[yufanw03]

Summary

DateparserQueryAnalyzer calls search_dates() without a languages argument (query_analyzer.py:253), so dateparser runs auto-detection across 200+ locales on every query. Two consequences: latency, as temporal analysis becomes the dominant CPU cost on the recall critical path; and locale misdetection, where English queries are identified as another language and yield a non-null but incorrect temporal constraint.

This adds an optional language restriction, defaulting to None (current auto-detection).

1. Latency

The call sits on the recall critical path via extract_temporal_constraint (retrieval.py:743) inside retrieve_all_fact_types_parallel, as synchronous CPU work. The class docstring states ~10-50ms per query (query_analyzer.py:189). Measured over 300 locomo10.json questions:

	mean (ms)	P50	P95	P99
auto-detect	68.19	62.09	130.76	164.95
languages=["en"]	0.25	0.17	0.62	1.68

2. Locale misdetection

from dateparser.search.search import DateSearchWithDetection

DateSearchWithDetection().search_dates(
    "What kind of professional experience did Jon get accepted for on May 23, 2023?"
)
# {'Language': 'bas',
#  'Dates': [('Jon', datetime.datetime(2026, 8, 1, 0, 0)),
#            ('May 23, 2023', datetime.datetime(2023, 11, 23, 0, 0))]}

FullTextLanguageDetector identifies this English query as bas (Basaa, a Bantu language of Cameroon), whose month table maps May to November. The first match, 'Jon', is correctly rejected by the scoring from #2772 — that mechanism works as intended; the problem is the second match.

Output is byte-identical on dateparser 1.2.2 (the pinned version) and 1.4.1 (shipped in the 0.8.4 image), and stable across 5 consecutive runs. Testing the same query against each of 44 locales individually, none produces November: en/es/tr/tl give the correct 2023-05-23, and the remaining 40 degrade to the fragment '23, 2023' → 2023-07-23.

Three failure shapes:

	Sample	auto-detect	["en"]
Month read under the wrong locale	…on May 23, 2023?	'May 23, 2023' → 2023-11-23	2023-05-23
Only a fragment returned; full expression not among candidates	…on 31 October, 2022?	'on 31' → 2026-07-31	2022-10-31
A non-date number read as a date	…got her at age 10?	'10' → 2025-10-31	None

In the second row, 'on 31' carries no month or year, so RELATIVE_BASE fills both in from the current date — landing nearly four years off the date written in the query.

These constraints are non-null but wrong, the case #2768 identified as worse than no constraint at all. This is complementary to the scoring in #2772 rather than overlapping with it: scoring ranks candidates by the date signal they carry, but the problem here is the candidate set itself. 'May 23, 2023' scores 150, the highest of its candidates, and 'on 31' scores 100 on the digit alone; both clear any reasonable threshold.

Why the default stays None

Restricting languages degrades explicit dates in unlisted locales, and degrades them to a wrong date rather than to None:

Locale	Input	auto-detect	["en"]
es	el 3 de junio de 2023	2023-06-03 ✓	2023-07-31 ✗
fr	le 3 juin 2023	2023-06-03 ✓	2023-07-31 ✗
it	il 3 giugno 2023	2023-06-03 ✓	2023-07-31 ✗
pt	em 3 de junho de 2023	2023-06-03 ✓	2023-07-31 ✗

extract_period covers fixed phrases in these locales (ayer, hier, letzte woche) but not explicit dates, so those necessarily reach search_dates. German and Chinese are unchanged. Chinese is handled by extract_chinese_period: of 32 Chinese temporal expressions tested, 23 resolve on that path — including Chinese-numeral dates such as 五月三日 and 二零二三年六月三日 — without reaching search_dates. ["en"] and ["en","zh"] produce identical results across all 300 English queries.

Locale coverage is limited to the 8 languages tabulated above; among non-Latin scripts, only Chinese, Japanese, and Russian were checked.

Changes

• query_analyzer.py: __init__ takes languages: list[str] | None = None. Adds _search_kwargs() so analyze() and the warm-up call in load() use the same locale set — otherwise the warm-up is wasted and part of dateparser's lazy-load cost lands on the first real query. With languages=None it returns an empty dict, byte-identical to the previous behavior.
• config.py: adds HINDSIGHT_API_QUERY_ANALYZER_LANGUAGES and the query_analyzer_languages field, comma-separated. Named following the scoping convention from feat(bm25): configurable native language + opt-in pgroonga backend #1538 (HINDSIGHT_API_BM25_LANGUAGE → ..._TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE). No value validation: this never reaches SQL, and an invalid locale code surfaces as a dateparser error. Happy to add startup validation if you'd prefer to fail early.
• memory_engine.py: passes languages=config.query_analyzer_languages at construction.

Testing

tests/test_query_analyzer.py adds 4 cases: the default passes no languages, an explicit list reaches search_dates unchanged, load()'s warm-up uses the same locale set, and a regression guard that on 31 October, 2022 resolves to 2022-10-31. The file is at 426 passed on dateparser 1.2.2.

Env var parsing: unset / "" / ",," → None; "en" → ["en"]; "en,zh" → ["en","zh"]; " EN , ZH " → ["en","zh"].

The latency benchmark runs 300 locomo10.json questions (random.seed(42), reference_date pinned to 2026-07-31), taking the minimum of 3 runs per query to filter scheduling jitter, fully offline with no service or network involved. Environment: Apple Silicon MacBook Pro / Python 3.11.15 / dateparser 1.2.2.

Arm order was swapped and re-measured to rule out warm-up bias; mean holds: auto first gives 68.19 / 0.25 ms, en first gives 0.25 / 68.33 ms. In this sample extract_period short-circuits 16 queries and the other 284 reach search_dates.

Correctness was enumerated over all 1,986 questions rather than sampled, giving 17 differences: 13 where auto-detect is wrong (1 locale month misread, 9 fragments, 3 non-date numbers), 3 where the restriction adds a correct constraint (next month, next year ×2), and 1 where it adds a questionable one (half an hour → 'an hour' → 2026-07-31, where auto-detect returns None; this is a behavior change introduced here and I have not evaluated it further). The 9 fragment cases:

Query (tail)	auto-detect	["en"]
…on 31 October, 2022? ×2	'on 31' → 2026-07-31	→ 2022-10-31
…community on 7 July, 2023?	'on 7' → 2026-07-31	→ 2023-07-07
…help with on 13 March, 2023?	'on 13' → 2026-07-13	→ 2023-03-13
…Caroline on October 13, 2023?	'13, 2023' → 2023-07-13	→ 2023-10-13
…Nate on February 7, 2022?	'7, 2022' → 2022-07-31	→ 2022-02-07
…goodbye to on 3 June, 2023?	'2023' → 2023-07-31	→ 2023-06-03
…evening of 7 July, 2023?	'2023' → 2023-07-31	→ 2023-07-07
…subjects on 9 February, 2023?	'2023' → 2023-07-31	→ 2023-02-09

Related

#3140 (open) also reduces the cost of this function, but by bounding query length — orthogonal to this change. Prior changes to this file (#893, #2772, #2791, #2767) do not touch locale selection.

[nicoloboschi]

please ensure the doc page configuration.md contains this new config, thanks

[yufanw03]

Added the entry to hindsight-docs/docs/developer/configuration.md and skills/hindsight-docs/references/developer/configuration.md (following #3043), under ### Retrieval next to the other recall-path knobs. The description also states the tradeoff, since it matters before enabling: explicit dates in an unlisted locale misparse rather than yield no constraint. Thanks for the quick review.

[nicoloboschi]

@yufanw03 please rebase

[yufanw03]

Rebased onto latest main.