Skip to content

feat: localize the chatbot's grounding & source metadata by thread language - #61

Merged
rdahis merged 3 commits into
stagingfrom
feat/thread-language-locale
Aug 5, 2026
Merged

feat: localize the chatbot's grounding & source metadata by thread language#61
rdahis merged 3 commits into
stagingfrom
feat/thread-language-locale

Conversation

@rdahis

@rdahis rdahis commented Aug 5, 2026

Copy link
Copy Markdown
Member

Localizes the chatbot's data metadata — both what the user sees and what the agent grounds on — to the thread's language, with a pt fallback everywhere. Complements #58, which already made the bot answer in the site language and persists Thread.language; this PR closes the metadata side, which was still Portuguese-only.

What was wrong

The agent read and displayed dataset/table metadata in Portuguese regardless of the thread language:

  • The "Fontes dos Dados" source names resolved via a GraphQL lookup that fetched only the pt name.
  • The retrieval tools (search_datasets, get_dataset_details, get_table_details) read pt-only name/description, so the model grounded on — and often echoed — Portuguese names, descriptions, themes, tags, organizations, and usage guides.

What changed

Display — source names (app/api/streaming/data_sources.py)

  • Resolve namePt/nameEn/nameEs and pick by the thread's language (pt fallback where a translation is missing); cache keyed by (language, table_id).

Grounding — retrieval tools (app/agent/tools/)

  • search_datasets passes locale to the /search/ endpoint, which is already locale-aware (it matches the localized text field and returns dataset_name_{locale}/dataset_description_{locale} plus localized themes/tags/organizations, all with server-side pt fallback).
  • get_dataset_details fetches the explicit namePt/nameEn/nameEs + descriptionPt/... columns and picks by language for the dataset and its tables; themes, tags, and organizations are localized too, and the usage guide is fetched from userGuide/{locale}/ with a pt fallback (only pt guides exist today).
  • get_table_details localizes the table name/description and column descriptions (column names stay — they are real BigQuery identifiers).

Plumbing

  • New shared helper app.i18n.localized_field(node, field, language) (used by both the resolver and the tools).
  • Language reaches the tools via injected RunnableConfig (config["configurable"]["language"], set alongside thread_id when the run is dispatched) — the same pattern execute_bigquery_sql already uses, so the model never sees the config argument.

Deliberately left Portuguese

  • Column identifiers — real BigQuery column names (municipio, ano, …).
  • The Brazil-specific few-shot examples in app/agent/prompts.py (not user-visible; steer the model).

Coverage / fallback

Every path falls back to pt when a localized value is empty. On a 100-table sample: ~91% have nameEn, ~89% nameEs; descriptions and en/es usage guides are lower and fall back to pt.

Verification

  • python -m py_compile clean on all changed files.
  • Verified live that TableNode/DatasetNode/ColumnNode/OrganizationNode/ThemeNode/TagNode expose the nameEn/Es + descriptionEn/Es columns, and that the /search/ endpoint localizes results (locale=en"American Time Use Survey (ATUS)" / theme "Economy" vs pt "Pesquisa Americana de Uso do Tempo (ATUS)" / "Economia").
  • Could not run the full app locally (no env) — CI is the build + tool-schema gate.

🤖 Generated with Claude Code

rdahis added 2 commits August 5, 2026 17:54
…anguage

Fontes dos Dados showed pt names because the GraphQL lookup only fetched the
default name. Fetch nameEn/nameEs, pick by the thread's language with a pt
fallback where a translation is missing, and cache per (language, table_id).
Route dataset/table/column metadata through the thread's language so the
agent grounds on en/es content (pt fallback) instead of pt-only:

- search_datasets passes locale to the (already locale-aware) /search/
  endpoint, so names, descriptions, themes, tags and organizations return
  localized.
- get_dataset_details / get_table_details fetch the explicit
  namePt/nameEn/nameEs and descriptionPt/... columns and pick by language;
  themes/tags/organizations and the usage guide (userGuide/{locale}/, pt
  fallback) are localized too. Column identifiers stay pt.
- Language reaches the tools via injected RunnableConfig, the same pattern
  execute_bigquery_sql already uses; the model never sees config.
- Add a shared app.i18n.localized_field helper; use explicit namePt for the
  data-source name resolver instead of the ambiguous name accessor.
@rdahis rdahis changed the title feat(data-sources): localize resolved dataset/table names by thread language feat: localize the chatbot's grounding & source metadata by thread language Aug 5, 2026
@rdahis rdahis self-assigned this Aug 5, 2026
Match tests to the new API: GraphQL mocks return the explicit
namePt/descriptionPt columns; _resolve_table_name and resolve_data_source_names
take the thread language; the search mock keeps name/description since the
/search/ endpoint localizes server-side.
@rdahis
rdahis merged commit 5e7a2d9 into staging Aug 5, 2026
1 check passed
@vrtornisiello
vrtornisiello deleted the feat/thread-language-locale branch August 5, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant