Definition
Benchmark contamination occurs when benchmark test cases, or close variants of them, appear in data used to train or fine-tune the model being evaluated. The model may then perform well partly because it has seen the questions, answers, or solutions before. Its score can overstate how well it handles new cases.
The benchmark is supposed to test the model on unseen cases. For any case that also appears in training, it cannot cleanly measure how the model handles an unfamiliar input.
Simple example
A team fine-tunes a support model on reviewed question-and-answer pairs. Later, it evaluates the model on 200 support questions intended to be held out. During data preparation, 20 of those questions and their approved answers were copied into the fine-tuning set.
The model answers most of those 20 questions correctly. Those results are weak evidence of how it will handle new support requests, since the questions and answers also appeared in fine-tuning. The team reports the affected cases separately and runs a fresh evaluation on questions kept out of training.
Why it matters
An inflated score can lead a team to select a model that disappoints on real requests. The overlap is easy to miss when training data comes from a large web collection or a reused internal dataset. It can also happen when someone generates training examples from an existing test set.
Where you control the training data, check it against the evaluation set for exact matches and near-duplicates before training or fine-tuning. Keep the evaluation set separate and record where examples came from. For a model whose training data you cannot inspect, use fresh or private cases to reduce the chance of overlap.
One important nuance
Text overlap alone does not prove that a score was inflated. A model might have seen a source passage without seeing the benchmark question or its answer. Conversely, a paraphrased test question may escape an exact-match check. Inspect what actually overlapped and compare results on clean cases before drawing a conclusion.