Definition
Model distillation trains one model, the student, to reproduce selected behavior of another, the teacher. The student is often smaller, but it can also use a different architecture.
The training signal might be the teacher’s probabilities over possible answers, its generated responses, or selected internal representations. Training updates the student’s parameters. Distillation does not require copying the teacher’s weights, although some methods initialize the student from selected teacher weights.
Simple example
A team uses a large classifier to sort support tickets into access, billing, and bug. For a ticket about a failed password reset, the teacher assigns 90% to access, 8% to billing, and 2% to bug. A smaller classifier trains on those probabilities across many tickets, then handles new tickets without calling the teacher each time.
The team checks its predictions against reviewed tickets kept out of training. Matching the teacher is useful, but the teacher can be wrong too.
Why it matters
A smaller student may use less memory and compute for a task that runs often. That can reduce request latency or serving cost, depending on the deployment. Generating teacher outputs and training the student have an upfront cost.
Before replacing the teacher, compare both models on held-out examples, including cases where mistakes matter. Measure serving costs under the intended workload too.
One important nuance
The student learns from the teacher signals and examples it receives, not from everything the teacher can do. Sparse or unrepresentative training inputs can leave gaps. Wrong teacher outputs can teach the wrong behavior. Distillation also differs from quantization: quantization changes how model values are represented, while distillation trains another model. The two techniques can be combined.