Skip to content

Fix quadratic BPE merging for long pieces - #148

Open
JayYarlagadda wants to merge 1 commit into
openai:mainfrom
JayYarlagadda:fix/issue-79-bpe-performance
Open

Fix quadratic BPE merging for long pieces#148
JayYarlagadda wants to merge 1 commit into
openai:mainfrom
JayYarlagadda:fix/issue-79-bpe-performance

Conversation

@JayYarlagadda

Copy link
Copy Markdown

Summary

  • keep the existing cache-friendly vector merge for pieces shorter than 400 bytes
  • use a binary heap and linked boundary arrays for larger pieces, reducing merge work from O(mn) to O(m log n)
  • preserve the existing leftmost-lowest-rank tie-breaking behavior
  • add synthetic, randomized, real-vocabulary, multilingual, threshold-boundary, and long-dash parity coverage
  • add a deterministic operation-count regression test for quadratic versus quasilinear scaling

Validation

  • maturin develop --release
  • pytest -q (61 passed)

Fixes #79

@JayYarlagadda
JayYarlagadda marked this pull request as ready for review August 7, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extremely slow (quadratic) tokenization speed for a sequence of dashes

1 participant