Definition
Tokenization converts an input into a sequence of discrete units and maps those units to identifiers from a fixed vocabulary. For text models, a token may represent a whole word, part of a word, punctuation, whitespace, or a byte-level pattern. The model processes the resulting identifiers rather than raw text.
The tokenizer and model form a contract. Vocabulary entries, splitting rules, normalization, and special tokens must match what the model was trained to interpret.
Simple example
The text unavailable might become one token for one model and several pieces such as un, avail, and able for another. A code sample, emoji, or German compound word may split differently again. Each piece maps to a numeric identifier before its representation enters the model.
When the model generates token IDs, a decoder reconstructs the output text from the sequence.
Why it matters
APIs commonly measure context and billing in tokens, not characters or words. Tokenization affects whether an input fits the context window, token-based request costs, and the computation required to process the sequence. It can also affect model behavior when important strings split into rare or awkward pieces.
Applications that truncate, batch, or budget input should count tokens using the tokenizer and input format for the selected model or serving API.
One important nuance
Tokens are not words. Token counts vary with the tokenizer and the exact input text, including its language and formatting. Character length is only a rough proxy. If you store token counts or token-level offsets, record which tokenizer produced them. A tokenizer change can invalidate those values.