Skip to content

fix(scripts): transliterate accents in slugify across all four copies (#766) - #767

Merged
tractorjuice merged 2 commits into
mainfrom
fix/slugify-transliterate-766
Aug 13, 2026
Merged

tractorjuice merged 2 commits into
mainfrom
fix/slugify-transliterate-766

Conversation

@tractorjuice

Copy link
Copy Markdown
Owner

Addresses #766. Does not close it, see "What remains" below.

The bug

slugify is implemented four times: bash and Python, each in a root copy and a plugin copy. f8544b4a (#204, "preserve accented characters in slugify") landed in scripts/bash/common.sh only. The other three kept the original [^a-z0-9] class:

$ slugify "Café Modernisation"
scripts/bash/common.sh                        -> café-modernisation
plugins/arckit-claude/scripts/bash/common.sh  -> caf-modernisation
scripts/python/common.py                      -> caf-modernisation
plugins/arckit-claude/scripts/python/common.py-> caf-modernisation

The character is dropped mid-word. Marketplace users never touch scripts/bash/, so the single copy that carried the fix was the one they never ran, and #204 has been effectively unfixed for them since March.

Why the fix was not just "port #204"

[:alnum:] follows LC_CTYPE, so the root copy's output depended on the caller's environment:

LC_ALL=C.UTF-8   -> cafe-zurich-ecole   (accents kept)
LC_ALL=C         -> caf-z-rich-cole     (the fix silently reverts)

So "preserve accents" was not a stable behaviour to converge on.

The fix

All four copies transliterate to ASCII from a shared table (Latin-1 Supplement plus the Latin Extended-A characters used across European official languages). Characters outside the table are dropped, consistently in both languages.

"Café Zürich ÉCOLE"  ->  cafe-zurich-ecole
"Straße"             ->  strasse
"Łódź Rail"          ->  lodz-rail
"日本 Project"        ->  project

Transliteration over preservation because these names become filesystem paths, git entries and published URLs, and because macOS stores accents decomposed (NFD) where Linux composes them (NFC), so a literal accented directory name can mismatch across a clone.

Two portability details, both load-bearing:

  • The bash implementation pins LC_ALL=C for the byte-oriented steps and spells out its character classes rather than using \+ or \|, which are GNU extensions and not portable to BSD sed. The previous implementation used both.
  • Python lowercasing is restricted to A-Z. str.lower() maps the Turkish dotted capital İ to i plus a combining dot, which would have diverged from bash on that one input.

Tests

New tests/plugin/test_slugify.py, written before the fix (confirmed RED at 57 failed / 45 passed, the 45 being the plain-ASCII corpus that already worked; GREEN at 102 passed):

  • each of the four copies against a shared corpus
  • all four agree on every corpus entry, since the table is duplicated four ways and nothing else stops it drifting
  • bash output identical under LC_ALL=C and LC_ALL=C.UTF-8
  • no copy can emit non-ASCII

End to end, create-project.sh --name "Café Zürich Modernisation" now creates 001-cafe-zurich-modernisation.

Full suite 1399 passed / 225 skipped, 285 node tests, sync-shared-assets.py --check clean, markdownlint clean. Extensions regenerated.

What remains on #766

  • No general parity guard for the two common.sh copies. This PR adds one for slugify specifically.
  • The find_repo_root divergence (.arckit/ in the root copy, projects/ in the plugin copy) is deliberate, and stated in the Python docstrings, but still undocumented in the bash copies.
  • get_templates_dir sits in a different position in each bash copy, which is cosmetic but will trip a naive diff-based guard.

Relationship to #764

Independent. Both touch common.sh but in different functions, and this branch is cut from main, so they merge in either order.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VKbHCujejpxkgvh47BndDE

…#766)

slugify is implemented four times: bash and Python, each in a root copy
and a plugin copy. f8544b4 (#204, "preserve accented characters in
slugify") landed in scripts/bash/common.sh only. The other three kept the
original [^a-z0-9] class, so "Café Modernisation" became
"caf-modernisation" with the character dropped mid-word. Marketplace
users never touch scripts/bash/, so the one copy carrying the fix was the
one they never ran.

#204's approach would not have held anyway: [:alnum:] follows LC_CTYPE,
so the root copy returned cafe-zurich-ecole under a UTF-8 locale and
caf-z-rich-cole under LC_ALL=C.

All four copies now transliterate to ASCII, identical in every locale and
in both languages. Transliteration was chosen over preserving the accent
because these names become filesystem paths, git entries and published
URLs, and because macOS stores accents decomposed (NFD) where Linux
composes them (NFC), so a literal accented directory name can mismatch
across a clone.

The bash implementation pins LC_ALL=C for the byte-oriented steps and
spells out its character classes rather than using \+ or \|, which are
GNU extensions absent from BSD sed. Python lowercasing is restricted to
A-Z because str.lower() maps the Turkish dotted capital I to "i" plus a
combining dot, which would diverge from bash.

tests/plugin/test_slugify.py holds all four copies to one corpus, asserts
the bash output is identical under LC_ALL=C and C.UTF-8, and asserts no
copy can emit non-ASCII.

Does not close #766: the general common.sh parity guard and the
deliberate find_repo_root divergence are still outstanding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VKbHCujejpxkgvh47BndDE
@tractorjuice
tractorjuice merged commit 13c8861 into main Aug 13, 2026
3 checks passed
@tractorjuice
tractorjuice deleted the fix/slugify-transliterate-766 branch August 13, 2026 18:57
@tractorjuice tractorjuice added the bug Something isn't working label Aug 31, 2026
@tractorjuice tractorjuice self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant