Definition
A language model learns statistical patterns in sequences of tokens. Depending on its training objective, it may predict the next token, recover a hidden token, or score a piece of text. Language models can work with natural language, source code, and other symbolic sequences.
Modern neural language models typically process token identifiers rather than raw text. A tokenizer converts the input into those identifiers. Older statistical models can work directly with units such as words or characters, so token identifiers are not part of the general definition.
Simple example
Given the prefix The build failed because the, an autoregressive language model might give tests a higher probability than banana as the next token. The model produces the distribution. An inference runtime selects or samples from it, appends the chosen token to the prefix, and runs the model again to continue the response.
This does not require billions of parameters. A small model trained on a narrow collection of text is still a language model, even if its predictions are limited.
Why it matters
A language model’s token probabilities depend on its learned parameters and the current context. They are not probabilities that the resulting statements are true. A false continuation can still rank highly. The result can sound plausible and still be wrong.
Prompts and decoding controls can influence the output, but they do not change this boundary. When correctness depends on external facts, the system may need to supply or verify them through context, retrieval, or tools. If a rule must be enforced reliably, the system should not rely on unconstrained model output alone. It can reject output that fails validation. Constrained generation applies only when the rule can be expressed as an output constraint, such as a grammar or schema. Other rules may require deterministic logic.
One important nuance
“Language model” and “large language model” are not synonyms. An LLM is a language model considered large in scale, although what counts as “large” is relative. Many modern LLMs handle a broad range of tasks, but that breadth is common rather than definitional. There is no universal parameter count at which a language model becomes an LLM. The broader term also includes older statistical models and smaller neural models.