Skip to content

fix(consolidation): keep observations in their source facts' language - #3181

Merged
nicoloboschi merged 5 commits into
mainfrom
fix/consolidation-source-language
Aug 5, 2026
Merged

fix(consolidation): keep observations in their source facts' language#3181
nicoloboschi merged 5 commits into
mainfrom
fix/consolidation-source-language

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #3166.

Problem

build_consolidation_system_prompt() only carried a language rule when HINDSIGHT_API_LLM_OUTPUT_LANGUAGE was set — output_language_directive(None) returns "". Everything else in the consolidation prompt is English, so with no configured output language multilingual models drift and a bank of Chinese facts receives English observations. Retain has defaulted to preserving the input language since #181/#184; consolidation never did.

Reproduced against openai/gpt-oss-120b with the exact facts from the issue:

before → "The user often walks their pet in the park on weekends."
         "The user is preparing for a promotion defense."
after  → 用户周末经常带宠物去公园散步。
         用户最近在准备晋升答辩。

And on the update path — an existing English observation plus a new Chinese fact:

before → UPDATE: The user often takes their pet to the park for walks on weekends.
after  → UPDATE: 用户周末经常带猫豆豆去深圳湾公园散步。

HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=English still forces English output from those same Chinese facts, and English facts still stay English (no over-correction).

Change

A ## LANGUAGE section is added to the consolidation system prompt only when no output language is configured. Beyond "keep the source language", it settles the three cases the issue asked to pin down:

  • Per observation, not per batch — each observation follows the language of the facts in its own source_fact_ids. A mixed batch yields mixed observations; within one observation the majority language of its facts wins.
  • Updates follow the new facts — an existing observation in a different language is rewritten entirely in the new facts' language, never left half-translated. Banks that already drifted converge back as new facts arrive.
  • Names and technical terms are never translated — proper nouns, place/product names, identifiers, code, units stay verbatim.

When an output language is configured the new section is dropped rather than left to contradict "translate any source content into X".

Scope

This implements proposal 1 from the issue (prompt rule) and not proposals 2/3 (language detection + corrective retry + metrics), matching how retain solved the same class of drift. Two things worth knowing about that choice: the guarantee is prompt-level, which the docs now say explicitly; and _consolidate_batch_with_llm silently skips a batch once attempts are exhausted, so a "fail the batch" policy there would drop observations rather than surface an error. Happy to follow up with detection + retry if the prompt rule proves insufficient in the field.

Tests

  • tests/test_multilingual_bm25.py — deterministic prompt-assembly coverage for both branches (rule present when unset, replaced by the directive when set).
  • tests/test_consolidation_output_language.py (new, hs_llm_core) — real-LLM tests driving _consolidate_batch_with_llm and asserting via the LLM judge: Chinese creates, English observation updated by a Chinese fact, and the explicit-override case. Picked up automatically by the core-LLM CI job, which selects by marker.

Docs: hindsight-docs/docs/developer/multilingual.md documents the unset-language default, including the mixed-batch and update policies.

Consolidation's prompt is entirely English and only carried a language rule
when HINDSIGHT_API_LLM_OUTPUT_LANGUAGE was set, so with it unset multilingual
models drifted: Chinese source facts produced English observations (#3166).
Retain already defaults to preserving the input language; consolidation now
does the same, and the rule settles the three ambiguous cases — language is
picked per observation from its own source facts, an update rewrites the whole
observation in the new facts' language (so drifted banks self-heal), and proper
nouns/identifiers are never translated.

An explicit output language still wins: the default rule is dropped in that
case rather than left to contradict "translate everything into X".

Reproduced and verified against gpt-oss-120b: before, the issue's Chinese facts
yielded "The user often walks their pet in the park on weekends."; after, they
yield 用户周末经常带宠物去公园散步。

Fixes #3166
The rule rides in the system prefix of every consolidation call, so on
providers without prompt caching its size is paid per batch. Four sentences
carry the same four constraints the bullet list did, at 69 tokens instead of
223. Re-verified against gpt-oss-120b: identical output on all four cases
(Chinese creates, English observation updated by a Chinese fact, explicit
English override, English facts left alone).
CI showed Gemini merging a Chinese fact into an existing English observation
by editing the English sentence in place, keeping it English — the same result
with the verbose rule and the compressed one, so wording length was not the
problem. Name the failure mode instead: don't edit the old text, compose the
merged observation from scratch in the new facts' language.

Also stop the test asserting the merge routing. Whether the model updates the
existing observation or records a sibling is its call (gpt-oss-120b does both
across runs); asserting UPDATE made this a flaky test of merge behaviour rather
than of language. It now checks every emitted text, create or update.
@nicoloboschi
nicoloboschi force-pushed the fix/consolidation-source-language branch from 99a0a7d to 82b42b3 Compare August 5, 2026 12:31
All three tests now go through one helper that retries up to three times while
the output language is wrong, so a single stray response doesn't fail the suite
— the same shape test_multilingual.py uses.

The update test is additionally xfail(strict=False): Gemini keeps an existing
observation's English wording when a Chinese fact updates it, editing in place
rather than recomposing, and did so identically across three CI runs and three
prompt wordings. The OpenAI-compatible models the issue reports against comply,
so it xpasses there. The creates test stays a hard gate — that is the reported
bug, and every model tried passes it.
@nicoloboschi
nicoloboschi merged commit 42e8c53 into main Aug 5, 2026
207 of 208 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidation should preserve source-fact language and validate observation output

1 participant