Definition
Pretraining is an initial training phase in which a model learns parameter values from data before task- or behavior-specific adaptation. An optimization process updates those parameters to reduce loss on a training objective. For modern foundation models, this work usually runs at large scale. Language-model objectives often involve predicting the next token or recovering tokens hidden from the input.
The result is a pretrained model whose parameters encode patterns learned from the training data. It can be used directly through prompting or become the starting point for later training. Pretraining itself does not imply that the model follows instructions or behaves like a finished assistant.
Simple example
Consider an autoregressive language model trained on a large collection of text and code. It receives a sequence of tokens and predicts the token that comes next. The training process compares those predictions with the actual tokens, calculates a loss, and updates the model parameters. Repeating that process across many batches gradually produces a base model.
A later supervised fine-tuning run might use reviewed conversations to teach that base model a question-and-answer format. That later run adapts the model. It is not part of pretraining.
Why it matters
Pretraining establishes much of what a model can do before an application team uses it. Later adaptation can change its behavior, but it may not erase weak language coverage or biases inherited from pretraining.
Most application engineers do not run pretraining themselves, but the distinction still matters when comparing models. A base model and an instruction-tuned version may share an architecture yet behave very differently. The later training changes model parameters using different data, a different objective, or both.
One important nuance
Pretraining is one part of model development. Preparing the data is distinct from updating the model parameters, although data selection and preprocessing strongly affect the result.
In common LLM terminology, post-training does not include every training run after the initial pretraining phase. Continued pretraining can apply a pretraining-style objective to new or domain-specific data. Post-training methods such as supervised fine-tuning or preference optimization use different data or objectives to adapt the pretrained model toward desired behavior.