Type the word “unbelievable” into a language model, and the model never sees those twelve characters as a unit. Depending on the tokenizer, it might receive two or three fragments: “un”, “believ”, “able”. The model then processes each fragment as an integer drawn from its vocabulary.
What counts as one token
Common short English words like “the” and “run” are each a single token. Longer or rarer words get split: “tokenization” might become two or three tokens, and a technical term with little training precedent can fragment further still. For English, the number of tokens in a passage typically exceeds its word count.
The ratio is less favourable for languages with rich morphology or non-Latin scripts. Arabic or Korean text can require far more tokens to express the same meaning as English, which makes language choice a real cost variable when building on LLM APIs.
How tokenizers decide where to split
The dominant approach is Byte-Pair Encoding (BPE), adapted for neural machine translation by Sennrich, Haddow, and Birch in 2016 (arXiv:1508.07909). BPE starts with individual characters and repeatedly merges the most frequent adjacent pair, building up a vocabulary of common subword units until a target vocabulary size is reached. WordPiece (used in BERT) and SentencePiece (used in LLaMA) operate on similar principles, differing in how they score pairs to merge.
Each model ships with its own fixed vocabulary, produced by running BPE or a variant on that model’s training corpus. The same sentence can produce different token counts across GPT and Claude. There is no universal token.
Context windows are counted in tokens
A language model’s context window is measured in tokens. The same budget fills faster with code or mathematical notation than with plain prose, and faster still with languages that fragment into more tokens per word.
The limit covers everything in the request: the system prompt, all prior conversation turns, any documents passed in, and the model’s output. When the combined token count exceeds the limit, earlier content must be truncated or compressed before the model can continue.
Tokens are the billing unit
Virtually every commercial LLM API prices by token, with separate rates for input and output. Output tokens typically cost more than input, because generating each one requires a forward pass through the model, whereas reading input is cheaper. A prompt that sends a large document and asks for a short summary is therefore cheaper than the reverse.
Questions, answered
How many tokens is a word?
For English, token count typically exceeds word count. Common short words are often a single token; longer or rarer words split into two or more fragments. Non-English text and code generally produce more tokens per equivalent word count, with some scripts fragmenting considerably more.
What is tokenization in AI?
Tokenization is the process of breaking raw text into discrete units (tokens) and converting each to a number the model can process. Most modern LLMs use Byte-Pair Encoding or a close variant, which learns common subword sequences from a large training corpus.
Do different AI models use the same tokens?
No. Each model ships with its own tokenizer vocabulary, so the same sentence can produce a different token count in GPT than in Claude. Token counts are not directly comparable across providers.
Why do tokens matter for cost?
LLM APIs price by token, with separate rates for input and output. Output tokens are typically more expensive because generating each one requires a full forward pass through the model, whereas processing input is cheaper.
Brief is a team of AI associates you direct in plain language. Opening to a small group at a time.
Request access