fix(consolidation): keep observations in their source facts' language - #3181
Merged
Conversation
Consolidation's prompt is entirely English and only carried a language rule when HINDSIGHT_API_LLM_OUTPUT_LANGUAGE was set, so with it unset multilingual models drifted: Chinese source facts produced English observations (#3166). Retain already defaults to preserving the input language; consolidation now does the same, and the rule settles the three ambiguous cases — language is picked per observation from its own source facts, an update rewrites the whole observation in the new facts' language (so drifted banks self-heal), and proper nouns/identifiers are never translated. An explicit output language still wins: the default rule is dropped in that case rather than left to contradict "translate everything into X". Reproduced and verified against gpt-oss-120b: before, the issue's Chinese facts yielded "The user often walks their pet in the park on weekends."; after, they yield 用户周末经常带宠物去公园散步。 Fixes #3166
The rule rides in the system prefix of every consolidation call, so on providers without prompt caching its size is paid per batch. Four sentences carry the same four constraints the bullet list did, at 69 tokens instead of 223. Re-verified against gpt-oss-120b: identical output on all four cases (Chinese creates, English observation updated by a Chinese fact, explicit English override, English facts left alone).
CI showed Gemini merging a Chinese fact into an existing English observation by editing the English sentence in place, keeping it English — the same result with the verbose rule and the compressed one, so wording length was not the problem. Name the failure mode instead: don't edit the old text, compose the merged observation from scratch in the new facts' language. Also stop the test asserting the merge routing. Whether the model updates the existing observation or records a sibling is its call (gpt-oss-120b does both across runs); asserting UPDATE made this a flaky test of merge behaviour rather than of language. It now checks every emitted text, create or update.
nicoloboschi
force-pushed
the
fix/consolidation-source-language
branch
from
August 5, 2026 12:31
99a0a7d to
82b42b3
Compare
All three tests now go through one helper that retries up to three times while the output language is wrong, so a single stray response doesn't fail the suite — the same shape test_multilingual.py uses. The update test is additionally xfail(strict=False): Gemini keeps an existing observation's English wording when a Chinese fact updates it, editing in place rather than recomposing, and did so identically across three CI runs and three prompt wordings. The OpenAI-compatible models the issue reports against comply, so it xpasses there. The creates test stays a hard gate — that is the reported bug, and every model tried passes it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3166.
Problem
build_consolidation_system_prompt()only carried a language rule whenHINDSIGHT_API_LLM_OUTPUT_LANGUAGEwas set —output_language_directive(None)returns"". Everything else in the consolidation prompt is English, so with no configured output language multilingual models drift and a bank of Chinese facts receives English observations. Retain has defaulted to preserving the input language since #181/#184; consolidation never did.Reproduced against
openai/gpt-oss-120bwith the exact facts from the issue:And on the update path — an existing English observation plus a new Chinese fact:
HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=Englishstill forces English output from those same Chinese facts, and English facts still stay English (no over-correction).Change
A
## LANGUAGEsection is added to the consolidation system prompt only when no output language is configured. Beyond "keep the source language", it settles the three cases the issue asked to pin down:source_fact_ids. A mixed batch yields mixed observations; within one observation the majority language of its facts wins.When an output language is configured the new section is dropped rather than left to contradict "translate any source content into X".
Scope
This implements proposal 1 from the issue (prompt rule) and not proposals 2/3 (language detection + corrective retry + metrics), matching how retain solved the same class of drift. Two things worth knowing about that choice: the guarantee is prompt-level, which the docs now say explicitly; and
_consolidate_batch_with_llmsilently skips a batch once attempts are exhausted, so a "fail the batch" policy there would drop observations rather than surface an error. Happy to follow up with detection + retry if the prompt rule proves insufficient in the field.Tests
tests/test_multilingual_bm25.py— deterministic prompt-assembly coverage for both branches (rule present when unset, replaced by the directive when set).tests/test_consolidation_output_language.py(new,hs_llm_core) — real-LLM tests driving_consolidate_batch_with_llmand asserting via the LLM judge: Chinese creates, English observation updated by a Chinese fact, and the explicit-override case. Picked up automatically by the core-LLM CI job, which selects by marker.Docs:
hindsight-docs/docs/developer/multilingual.mddocuments the unset-language default, including the mixed-batch and update policies.