Tokenizer Profiler

Questions

Tokens, in plain terms

• What is a token, and how many tokens is 1,000 words?

  A token is the unit a model actually reads: usually a common word, a fragment of a rarer one, or a piece of punctuation. For ordinary English prose, 1,000 words lands around 1,300–1,500 tokens — but that ratio is a rule of thumb, not a rule, and it falls apart on code, tables and non-English text. The point of this tool is that you can stop estimating and see the real split for your own text.

• Why do Russian text and source code cost more tokens than English?

  Because the vocabularies were built mostly on English. A common English word is often a single token; the same word in Russian is frequently cut into three or four pieces, and source code splits on nearly every symbol. In practice Cyrillic and code run about 2–3× the token count of the same amount of English prose. That is a real line on the bill, not a curiosity — paste both and the difference shows up immediately.

• Why is the Claude count marked approximate instead of exact?

  Anthropic and Google do not publish their tokenizers, so nothing running on your device can reproduce their split. When the inference server is reachable, this tool asks the provider's own API for the real number and marks it Verified. When it is not, it falls back to a nearby open vocabulary and marks the result Approximate. That is deliberate: a number presented as exact when it is not is worse than no number at all.

• What are hidden reasoning tokens, and why can't this tool count them?

  Several models generate an internal chain of thought before answering. You are billed for it as output, and you never see it, so the real cost of a call can be several times the visible reply. This tool cannot count those tokens for one honest reason: the number does not exist until the model actually generates an answer — it is not derivable from the text you typed. So the models that do this carry a warning instead of an invented figure.