Repository navigation
fix(scripts): transliterate accents in slugify across all four copies (#766) - #767
Merged
Merged
Conversation
…#766) slugify is implemented four times: bash and Python, each in a root copy and a plugin copy. f8544b4 (#204, "preserve accented characters in slugify") landed in scripts/bash/common.sh only. The other three kept the original [^a-z0-9] class, so "Café Modernisation" became "caf-modernisation" with the character dropped mid-word. Marketplace users never touch scripts/bash/, so the one copy carrying the fix was the one they never ran. #204's approach would not have held anyway: [:alnum:] follows LC_CTYPE, so the root copy returned cafe-zurich-ecole under a UTF-8 locale and caf-z-rich-cole under LC_ALL=C. All four copies now transliterate to ASCII, identical in every locale and in both languages. Transliteration was chosen over preserving the accent because these names become filesystem paths, git entries and published URLs, and because macOS stores accents decomposed (NFD) where Linux composes them (NFC), so a literal accented directory name can mismatch across a clone. The bash implementation pins LC_ALL=C for the byte-oriented steps and spells out its character classes rather than using \+ or \|, which are GNU extensions absent from BSD sed. Python lowercasing is restricted to A-Z because str.lower() maps the Turkish dotted capital I to "i" plus a combining dot, which would diverge from bash. tests/plugin/test_slugify.py holds all four copies to one corpus, asserts the bash output is identical under LC_ALL=C and C.UTF-8, and asserts no copy can emit non-ASCII. Does not close #766: the general common.sh parity guard and the deliberate find_repo_root divergence are still outstanding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VKbHCujejpxkgvh47BndDE
…rate-766 # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #766. Does not close it, see "What remains" below.
The bug
slugifyis implemented four times: bash and Python, each in a root copy and a plugin copy.f8544b4a(#204, "preserve accented characters in slugify") landed inscripts/bash/common.shonly. The other three kept the original[^a-z0-9]class:The character is dropped mid-word. Marketplace users never touch
scripts/bash/, so the single copy that carried the fix was the one they never ran, and #204 has been effectively unfixed for them since March.Why the fix was not just "port #204"
[:alnum:]followsLC_CTYPE, so the root copy's output depended on the caller's environment:So "preserve accents" was not a stable behaviour to converge on.
The fix
All four copies transliterate to ASCII from a shared table (Latin-1 Supplement plus the Latin Extended-A characters used across European official languages). Characters outside the table are dropped, consistently in both languages.
Transliteration over preservation because these names become filesystem paths, git entries and published URLs, and because macOS stores accents decomposed (NFD) where Linux composes them (NFC), so a literal accented directory name can mismatch across a clone.
Two portability details, both load-bearing:
LC_ALL=Cfor the byte-oriented steps and spells out its character classes rather than using\+or\|, which are GNU extensions and not portable to BSD sed. The previous implementation used both.A-Z.str.lower()maps the Turkish dotted capitalİtoiplus a combining dot, which would have diverged from bash on that one input.Tests
New
tests/plugin/test_slugify.py, written before the fix (confirmed RED at 57 failed / 45 passed, the 45 being the plain-ASCII corpus that already worked; GREEN at 102 passed):LC_ALL=CandLC_ALL=C.UTF-8End to end,
create-project.sh --name "Café Zürich Modernisation"now creates001-cafe-zurich-modernisation.Full suite 1399 passed / 225 skipped, 285 node tests,
sync-shared-assets.py --checkclean, markdownlint clean. Extensions regenerated.What remains on #766
common.shcopies. This PR adds one forslugifyspecifically.find_repo_rootdivergence (.arckit/in the root copy,projects/in the plugin copy) is deliberate, and stated in the Python docstrings, but still undocumented in the bash copies.get_templates_dirsits in a different position in each bash copy, which is cosmetic but will trip a naive diff-based guard.Relationship to #764
Independent. Both touch
common.shbut in different functions, and this branch is cut frommain, so they merge in either order.🤖 Generated with Claude Code
https://claude.ai/code/session_01VKbHCujejpxkgvh47BndDE