Skip to content

fix(translate): download every released language, and list them in the docs - #1405

Merged
jakeaturner merged 1 commit into
fix/translate-forward-redirectsfrom
fix/translate-model-fetch
Oct 5, 2026
Merged

jakeaturner merged 1 commit into
fix/translate-forward-redirectsfrom
fix/translate-model-fetch

Conversation

@chriscrosstalk

Copy link
Copy Markdown
Collaborator

Stacked on #1379. Both should merge before project-nomad-translate:0.1.1 is built, so one image carries both fixes (#1379 already bumps the seeder to 0.1.1).

Problem. fetch_models.py sorts model versions with float(). Firefox Remote Settings now publishes Nightly pre-releases (1.0a1) next to each release, so the fetch crashes with ValueError for 22 of the 34 named languages. Adding sv to TRANSLATE_LANGS downloads nothing and no button appears. Separately, translated pages inherited Kiwix's max-age=3600, so a newly added language's button didn't show for up to an hour, and a hard refresh doesn't reach the article frame.

Fix.

  • Only take records desktop Firefox uses on the release channel, and pick the newest complete version per direction instead of mixing files across versions.
  • Support pairs that publish separate source and target vocabs (Japanese).
  • Isolate failures per language so one bad language doesn't stop the rest.
  • Send Cache-Control: no-cache on translated HTML, and include the installed language set in the ETag.
  • Docs: add-a-language steps and the full code table; sizes corrected to 45-140 MB.

Testing.

  • All 50 languages downloaded, loaded and translated on bergamot 0.4.5 from the shipped 0.1.0 image (NOMAD3).
  • End to end on a stock 1.35.0 box (NOMAD1): added sv, which downloaded and translated, and the button appears without a cache clear.
  • New test_fetch_models.py, modelled on the registry's real records. It passes, and test_blocks.py still passes.

🤖 Generated with Claude Code

…e docs

fetch_models.py sorted versions with float(), and Firefox Remote Settings
now publishes Nightly pre-releases ("1.0a1") alongside each release. The
first one crashed the whole fetch, so adding Swedish, or 21 of the 34 named
languages, silently did nothing and the button never appeared.

Only take records desktop Firefox would use on the release channel, pick
the newest complete version per direction instead of mixing files across
versions, support pairs that publish a separate source and target vocab
(Japanese), and keep one failing language from stopping the rest.

That opens 50 languages, up from the ~40 the docs claimed, including
Arabic, Japanese, Korean and Thai, which the docs said were unavailable.
All 50 downloaded, loaded and translated on the bergamot 0.4.5 engine in
the shipped image (NOMAD3). Chinese stays out: it is published as
zh-Hans/zh-Hant, which the proxy's two-letter codes cannot carry yet.

Translated pages now send Cache-Control: no-cache and put the installed
language set in the ETag. Kiwix's max-age=3600 kept the old language bar
for up to an hour after a language was added, and a normal refresh does
not reach the article frame.

Docs: step-by-step instructions and the full code table in the Supply
Depot section; sizes corrected to 45-140 MB per language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jakeaturner
jakeaturner merged commit 602d929 into fix/translate-forward-redirects Oct 5, 2026
1 check passed
@jakeaturner
jakeaturner deleted the fix/translate-model-fetch branch October 5, 2026 21:12
caweis pushed a commit to caweis/project-nomad that referenced this pull request Oct 8, 2026
…other language

Adds upstream's Translated Library as a Supply Depot app. It shows the
Information Library in another language, translated on the Mac by Bergamot
(the engine behind Firefox's page translation) and offline once the models
have downloaded. It is a reverse proxy in front of kiwix-serve on port 8460;
the library at its usual address is not touched. French, Spanish and German
are fetched on first start. The engine has models for 50 languages.

What is in it
- install/nomad-translate/: upstream's directory at v1.35.1, byte for byte
  (the eight files' blobs match crosstalk/main). That includes the two fixes
  released after the first version: redirects are forwarded to the browser
  instead of followed (upstream Crosstalk-Solutions#1379, eef5b6c), and every released language
  downloads instead of the fetch stopping at the first pre-release model
  (upstream Crosstalk-Solutions#1405, ee203df). FORK_NOTES.md beside them is this fork's: what
  differs, and what has not been seen working yet.
- A seeder entry (display order 13, port 8460, depends on the Information
  Library) that pins upstream's published image,
  ghcr.io/crosstalk-solutions/project-nomad-translate:0.1.1, and names the
  platform linux/amd64 so the previous commit's plumbing asks Docker for it.
- SERVICE_NAMES.TRANSLATE, its docs anchor, and a place in the Knowledge &
  maps deck. home_decks.standalone.ts gains a check that it sits beside the
  library.
- A Translated Library section in the Supply Depot docs.
- .gitattributes (*.sh stay LF, as upstream has it) and Python bytecode in
  .gitignore.

Adapted
- Upstream's image is linux/amd64 only, because Bergamot's intgemm backend is
  x86 specific. On Apple Silicon it runs under the container engine's x86
  emulation. No arm64 build is attempted here; #54 tracks
  one.
- Not ported: upstream's workflow that builds and publishes the image
  (85da928). It pushes to their registry, and only the users on their
  deployment list can run it.
- The docs say the language list is fixed for now. Upstream's text sends
  people to Manage > Edit, and this fork shows Edit on custom apps only.

Tests: the two Python test files pass (block planner, model selection). The
backend typechecks. The frontend has the same 12 errors as before and no new
ones. 68 standalone files pass (829 checks).
Not verified: the image starting under emulation, intgemm on that path, the
first-start model download, a pull and create with a platform against a real
Docker engine, or the seeder against a real database.
#56 lists the on-device checks.

Ports upstream Crosstalk-Solutions#1292 (1b67a87), Crosstalk-Solutions#1379 (eef5b6c) and Crosstalk-Solutions#1405 (ee203df),
adapted. Thanks to chriscrosstalk for the upstream work.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZGqfCQBTdU6Mf4GnNnwF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants