Token
Last verified
A token is the unit of text an LLM processes and the unit you are billed in. Tokenizers split text into sub-word pieces, so a token is roughly 3–4 characters of English — common words are one token, rare words split into several, and whitespace and punctuation count.
Two consequences worth internalising:
- Token counts are not word counts. As a rough planning figure, 1,000 English words is somewhere around 1,300–1,500 tokens. Code, JSON and non-English text tokenize less efficiently.
- Every provider tokenizes differently. The same text is a different number of tokens on GPT, Claude and Gemini, so comparing prices per token between providers is only approximate.
Input and output tokens are priced separately — see output token.