Definition
A masked language model is trained to predict the original tokens at selected positions, typically after those tokens have been hidden or otherwise corrupted. The model uses the surrounding context for each prediction.
Unlike standard left-to-right autoregressive language modeling, masked language modeling can use tokens before and after the selected position. In that left-to-right objective, each prediction depends only on the preceding tokens. The masked objective trains representations informed by the visible context on both sides. It does not directly train standard left-to-right generation.
Simple example
Consider this training input:
The deployment failed because the [MASK] expired.
The model can use both deployment failed and expired when predicting the hidden token. It might assign a high probability to certificate or token and a low probability to an unrelated word such as database.
In standard left-to-right autoregressive training, the prediction for the next token after The deployment failed because the can depend only on that prefix. A later token such as expired cannot influence the prediction, even if the model processes the full training sequence in one pass.
Why it matters
That difference affects downstream use. Masked language models can be adapted for tasks that use representations of a complete input, such as text classification, token labeling, or ranking. The adapted model can process the input in one pass and return labels or contextual representations.
The same objective does not directly train the model to produce an open-ended response one token at a time. For chat and other free-form generation, autoregressive models are usually a more direct fit.
One important nuance
The term “masked language model” refers to a model trained with a masked language modeling objective. It does not name a particular neural-network architecture. Many well-known masked language models use encoder-only transformers, but those terms are not interchangeable.
A deployed masked language model also does not need to receive a literal mask token. Fine-tuning can adapt its learned representations to tasks such as classification, where the application supplies ordinary text and reads a task-specific output.