AI glossary · Basic · Language models
Tokenizer
A tokenizer is a converter that turns text into token pieces and numbers a model can read, and turns the numbers the model outputs back into text. Each model has its own fixed tokenizer.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Language models
- Added
- 2026-10-12
How it works
A tokenizer builds a vocabulary by collecting groups of characters that often appear together in training data. When text comes in, it cuts the text into pieces according to this vocabulary and assigns a number to each piece. Common words become a single piece, while rare words are split into several pieces. Special tokens that mark the start and end of a conversation are also included in this vocabulary.
How text is split into tokens
Where you see it
Files such as tokenizer.json and tokenizer_config.json in Hugging Face model repositories are the tokenizer. The vocabulary size (vocab size) in the config file is the number of pieces in the vocabulary. Token counts on API price lists and context window lengths are all counted with this tokenizer. You can check how many tokens your text uses with the token calculators that AI companies provide.
Things to know
The same sentence has a different token count on each model, so account for this difference when comparing context windows or prices across models. Languages less represented in training data, such as Korean, sometimes take more tokens for the same content. Using a tokenizer different from the model's causes the numbers to mismatch and produces nonsensical output. The way numbers and spaces are split into small pieces can also cause mistakes in tasks such as arithmetic or counting characters.
Example
A developer estimating API costs put the same Korean document into two models' token calculators and found quite different token counts. They learned that choosing on unit price alone is not enough. Tokenizers are also an important variable in research. A study covered on this site pointed out that when doing On-policy Distillation between a Teacher model and a Student model with different tokenizers, how reliable the supervision signal is matters more than how tokens are aligned.