Repository navigation
Update onnxruntime-extensions to include the STL regex pre-tokenizer UTF-8 fix - #2875
Open
Wang, Zhenze (zz002) wants to merge 1 commit into
Open
Wang, Zhenze (zz002) wants to merge 1 commit into
Wang, Zhenze (zz002) wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
2 of 7 tasks
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The dependency pin change leaves the required generated cgmanifest stale, so the checked-in inventory no longer matches the onnxruntime-extensions revision fetched by CMake.
1 open issue
What changed in this PR
Updates ONNX Runtime GenAI’s onnxruntime-extensions dependency to include the upstream UTF-8 pre-tokenizer fix, addressing encoding failures when multi-byte characters precede a regex match.
Changes:
- Advances the pinned commit from
e2e119c0to62f2c894to incorporate the fix used by Laguna-XS 2.1’s tokenizer.
| File | Description |
|---|---|
cmake/deps.txt |
Updates the onnxruntime-extensions commit pin. |
Review effort: Balanced
| microsoft_wil;https://github.com/microsoft/wil/archive/refs/tags/v1.0.230629.1.zip;e4a542a323c070376f7c2d1973d0f7ddbc1d2fa5 | ||
| directx_headers;https://github.com/microsoft/DirectX-Headers/archive/refs/tags/v1.613.1.zip;47653509a3371eabb156360f42faf582f314bf2e | ||
| onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;e2e119c08b4379d038bcb5c75d9111328a9b86c1 | ||
| onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;62f2c894df545f71071ec13c7aeefc390ca58b45 |
Wang, Zhenze (zz002)
added a commit
to ROCm/hip-ep
that referenced
this pull request
Nov 6, 2026
## Summary Build OGA v0.19.0 with one PR diff instead of v0.17.0 with six. The remaining patch, microsoft/onnxruntime-genai#2875, updates the onnxruntime-extensions pin to include microsoft/onnxruntime-extensions#1136. ## Related issue or design microsoft/onnxruntime-genai#2875, microsoft/onnxruntime-extensions#1136 ## Why v0.19.0 contains microsoft/onnxruntime-genai#2685, #2691 and #2712, plus #2672 (which #2705 was extracted from) and #2714 (the runtime #2801 ported onto v0.17.0). Its onnxruntime-extensions pin `e2e119c` already has `Qwen3_5Tokenizer` (microsoft/onnxruntime-extensions#1142) but not #1136, and #2792's diff targets the v0.17.0 pin line, so it conflicts on v0.19.0. ## What - `OGA_VERSION: "0.19.0"` and `OGA_PR_PATCHES: "2875"` in `windows-deps.yml`, `linux-build.yml`, `windows-rmt.yml` and `linux-rmt.yml`; `linux-rmt.yml` now gets the extensions pin too. - `docs/quick_start.md` and `docs/quick_start_linux.md` follow the new tag and PR list. - ONNX Runtime stays at 1.31.0. ## Test plan - [ ] build-windows / build-linux green: v0.19.0 + #2875 applies and builds - [ ] gpu_test OGA benchmark: per-model TTFT / TPS / peak working set compared with main - [ ] OGA wheel smoke green ## Notes for reviewers v0.19.0's `model_benchmark` counts `OgaConfig::Create` in model creation time; its other output is unchanged. ## Checklist - [x] The change is focused, or links a design/series explaining its scope. - [ ] Relevant tests were added or updated and the results are documented. - [x] User-facing or design documentation was updated when needed. - [ ] Substantial AI assistance is disclosed, and I reviewed and understand the result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The pinned onnxruntime-extensions commit, microsoft/onnxruntime-extensions@e2e119c0, fails to encode text with
invalid string_view positionwhen a pre-tokenizerSplitpattern takes thestd::regexfallback and a multi-byte character precedes a match. Laguna-XS 2.1's tokenizer hits this on any prompt with a non-ASCII character before a newline.Solution
Updates onnxruntime-extensions to microsoft/onnxruntime-extensions@62f2c894, which adds only microsoft/onnxruntime-extensions#1136 on top of the current pin.