BPE (byte-pair encoding)

Byte-pair encoding is a compression technique adapted for tokenization. Starting from individual bytes, it repeatedly merges the most frequent adjacent pair into a new symbol, building a vocabulary where common sequences are single tokens and rare ones are several.

This is why token counts behave the way they do: the is one token, an unusual proper noun might be four, and a base64 blob is close to one token per few characters.

It also explains the non-English penalty. Vocabularies are trained on corpora dominated by English, so other scripts fragment into more tokens for the same meaning — and cost more.