Custom tokenizer - #730

Closed

PaulKoudelka wants to merge 9 commits into

MindWorkAI:mainfrom

PaulKoudelka:custom-tokenizer

Commits on Apr 10, 2026

added functionality to save/move tokenizer file
PaulKoudelka
committed
0854deb
added functionality to add tokenizer to LLM and embedding models
PaulKoudelka
committed
e07ca37

Commits on Apr 14, 2026

token count now adapts to chosen provider tokenizer
PaulKoudelka
committed
4157340

Commits on Apr 15, 2026

cleanup to ensure uniform style of code
PaulKoudelka
committed
4a8ab90
removed print statements
PaulKoudelka
committed
95fb213

Commits on Apr 23, 2026

added possibility to add tokenizer via plugin and refactored earlier changes
PaulKoudelka
committed
aa554d4

Commits on Apr 24, 2026