[kunal-vaishnavi]

Description

This PR keeps trailing slashes with punctuation across CR/LF boundaries for the GPT-4o regex, while preserving the CR/LF-only matcher. It also adds native regressions for LF, CRLF, mixed suffixes, and the non-slash control.

Motivation and Context

This PR fixes an issue with the GPT-4o tokenizer family.

[Copilot]

🟢 Approval recommended

No active review issues remain for the GPT-4o punctuation-suffix change.

No open issues 🎉

What changed in this PR

Fixes GPT-4o pre-tokenization so punctuation retains trailing slashes across carriage-return and newline boundaries, while preserving the existing CR/LF-only matcher.

Changes:

• Shares punctuation-matching logic with an option to include slashes in the suffix.
• Adds native regressions for LF, CRLF, mixed suffixes, and the CR/LF-only control.

File	Description
test/​pp_api_test/​test_tokenizer_impl.cc	Adds regression tests for slash-inclusive and CR/LF-only suffix matching.
operators/​tokenizer/​bpe_utils.hpp	Routes the GPT-4o punctuation pattern to slash-aware suffix matching.

_{Review effort: Balanced}

[Sayan Shaw (sayanshaw24)]

Reviewed the matcher split and regressions. The slash-aware path now implements the exact [\r\n/]* suffix while the existing CR/LF-only path preserves [\r\n]*; the tests cover LF, CRLF, mixed slash/newline suffixes, and the non-slash control. I found no correctness issues in the change.

[Sayan Shaw (sayanshaw24)]

I also checked the failing matrix jobs. The representative Linux, macOS, and Windows failures are all the unrelated test_fast_tokenizer.py::TestAutoTokenizer::test_mistral dependency error: the installed sentencepiece_model_pb2.py imports google.protobuf.runtime_version, but the environment has an older protobuf package. The new native punctuation-suffix tests are not implicated.