Tokens

A token is a chunk of text

A model does not see letters and it does not see words. Before anything happens, your text is chopped into pieces called tokens, each of which the model has learned as a single unit. A token is usually a common word, a fragment of a longer word, a punctuation mark, or a space plus the word that follows it.

Roughly, unbelievable might become three pieces along the lines of un / believ / able, while the is a single piece because it appears constantly. Common text compresses into few tokens; unusual text sprawls into many.

Why not just use words?

Because words are a bad unit for a general system. Tokenisation solves four problems at once:

• Unknown words. A word-based system hits something it has never seen and has no representation for it at all. A token-based one spells it out from familiar fragments, which is how these models handle new product names, surnames and jargon.
• Typos. teh is not a word, but it is a perfectly reasonable sequence of pieces.
• Other languages. Many languages don't delimit words with spaces at all, so "split on spaces" isn't available.
• Code and structured text. Brackets, indentation and symbols carry meaning and need to survive the process.

The rule of thumb everyone repeats is attached to a tokeniser

You will find, in a great many places, the claim that one token is about four characters, or about three quarters of a word. That number entered circulation years ago, attached to a particular generation of tokeniser, and it has been copied forward ever since without being rechecked.

It is wrong for the newest models and still right for others, which is the more interesting fact. Anthropic describes the one-million-token context window on its current flagship models as approximately 555,000 words and approximately 2.5 million characters. Work backwards from the vendor's own figures and you get:

Derived by division from Anthropic's stated approximations (1M tokens ≈ 555k words ≈ 2.5M characters) on its models overview, fetched 12 August 2026. These are approximations of approximations: treat them as an order of magnitude, not a calculator.

That is not a rounding difference. If you budget a job using the old ratio, you can be out by something like forty per cent on token count, and therefore on cost. Anthropic states directly that its newer tokeniser produces roughly 30% more tokens for the same text than the generation before it, and dates the change precisely: models from Claude Opus 4.7 onward use it, everything earlier does not.

Which is why the folklore is not simply dead. On the same page, the older one-million-token models are described as approximately 750,000 words and 3.4 million characters, and Claude Haiku 4.5 — a current model, on this site's own Model facts table — as approximately 150,000 words and 680,000 characters for its 200,000-token window. Both work out to 0.75 words per token — the memorised figure exactly — and about 3.4 characters, nearer the folklore's four than the 2.5 above. The old ratio is not dead; it is accurate on models people are using today. One vendor's documentation supports both sets of numbers, in different rows of the same table, and it is right to.

Why you should care

It is the unit you are billed in

Pricing is quoted per million tokens, and quoted separately for input and output. Output costs several times more than input on every current model, which is worth internalising: asking for a long answer is far more expensive than supplying a long question. Current rates are on Model facts.

It is the unit the context window is measured in

The window is a token budget, not a page count, and it is shared between what you send and what comes back. This is why the two concepts have to be learned together. → Context windows

It makes some content far more expensive than it looks

Because compression depends on how ordinary the text is, cost per page varies more than people expect.

• Cheap per page: plain English prose, common vocabulary.
• More expensive: code, especially with deep indentation and long identifiers. JSON and XML, where structural punctuation is a large fraction of the content. Tables. Long unbroken strings such as IDs, hashes and base64.
• Considerably more expensive: languages that were less represented in the tokeniser's training. The same meaning can cost noticeably more tokens in some languages than in English, which is a real and under-discussed fairness problem in how these systems are priced.

Don't try to count them yourself

Two specific traps.

Don't estimate from a character count. If the number matters for budgeting, ask the model's own API for the count. Anthropic, for instance, exposes a dedicated token-counting endpoint. Every guess you make is worse than an exact answer that is free to obtain.

Don't use another vendor's tokeniser. The popular tiktoken library is OpenAI's, and it is simply the wrong tool for Claude. Anthropic's published guidance for its own developers puts the error at 15–20% undercounting on typical text, and worse on code or non-English input. This is a common enough mistake to be worth naming: cross-vendor token estimates are not approximately right, they are systematically wrong in one direction.

Sources

Word, character and token approximations come from Anthropic's models overview, where they appear as tooltips on each model's context window. The 30% tokeniser increase, and the fact that it begins with Claude Opus 4.7, come from its token counting documentation. Both fetched 12 August 2026.

The tiktoken undercounting range is Anthropic's figure but is not on either page above, which is where this page cited it until 12 August 2026. It is published in Anthropic's token counting guidance for developers: "It undercounts Claude tokens by ~15–20% on typical text, and by much more on code or non-English input." A citation you cannot follow to the claim is worse than no citation at all, so the correction is logged on the changes page rather than quietly fixed.

Ratios in the comparison table above are arithmetic on the published figures rather than independent measurements, and are labelled as such. Equivalent figures for other vendors have not been verified and are deliberately not stated here.

Next → Context windows · All → Concepts