What is a token in AI and how is it counted?
A token is the smallest piece of text a language model works with. Pricing, speed and how much it can read at once are measured in tokens.
What is a token?
A token is the smallest unit a language model uses when it processes text. The model does not read letter by letter or word by word. It first splits the text into tokens, turns each token into a number and does all of its math on those numbers. A token can be a whole word, part of a word or a single punctuation mark.
A short, common word like "house" is usually one token. A long or rare word gets split into several pieces. The splitting rules are fixed when the model is built, so the same sentence can come out as a different number of tokens in two different models.
How is text split into tokens?
The component that does this is called a tokenizer. Common methods keep frequent character sequences from the training data as single pieces; the best known is BPE (byte pair encoding). Frequent sequences become one token, rare ones are broken into smaller parts.
OpenAI's rule of thumb for English is that one token is about four characters, or roughly three quarters of a word. Languages that build long words from many suffixes, such as Turkish, usually need more tokens for the same content, because most tokenizers are built mainly on English text.
What does the token count decide?
Most models used through an API are priced by the number of tokens sent in and generated. Generated tokens usually cost more than input tokens, so a long instruction and a long answer both show up on the bill.
The amount of text a model can consider at once is also limited in tokens. That limit is the context window. Speed depends on tokens too: the model writes its answer one token at a time, so a long answer takes longer to appear.
Spending fewer tokens
Getting the same job done with fewer tokens lowers both cost and waiting time. Writing a short, clear prompt, sending only the relevant part of a long document and setting an explicit length for the answer are the simplest ways to do it.
Systems that work with company documents handle this with RAG: instead of sending the whole archive with every question, they pick the few passages that matter and give only those to the model.