Basics
Tokenizer
The component that splits text into tokens before a model processes it, and turns tokens back into text.
Each model family has its own tokenizer, so the same text can cost a different number of tokens on different models. Anthropic notes, for example, that its newer tokenizer produces about 30% more tokens for the same text.
Frequently asked questions
What does Tokenizer mean?
The component that splits text into tokens before a model processes it, and turns tokens back into text. Each model family has its own tokenizer, so the same text can cost a different number of tokens on different models. Anthropic notes, for example, that its newer tokenizer produces about 30% more tokens for the same text.