Tokenizer
Last verified
A tokenizer converts text into the token IDs a model consumes. Most modern LLMs use a byte-pair encoding variant, which merges frequent character sequences into single tokens.
Practical implications:
- Counts differ per provider. The same paragraph is a different number of tokens on GPT, Claude and Gemini, so a price comparison per token carries a margin of error.
- English is the efficient case. Code, JSON, and non-Latin scripts tokenize less efficiently — sometimes much less — so the same content costs more in those forms.
- Formatting has a price. Verbose JSON with long key names costs more than compact output.
Counting tokens locally before you send is the cheapest way to estimate cost, and it needs no credentials.