Definition
An evaluation harness is the infrastructure that runs evaluation cases against an AI system in a repeatable way. It sends inputs to the system under test and records outputs, measurements, and run metadata. Depending on the evaluation, it may also load references or expected properties and run automated checks.
The system under test may include more than a model call. For a RAG application, the harness can exercise retrieval, context assembly, generation, citations, and post-processing as one pipeline.
Simple example
A team maintains 100 reviewed support questions with expected source documents and required answer properties. Before a release, the harness runs every question against the candidate configuration. It records retrieved document IDs, prompts, model outputs, latency, token use, faithfulness scores, and deterministic citation checks, then compares the results with the previous baseline.
Each result stays associated with the recorded model identifier and the prompt, retrieval-index, and scorer versions for that run.
Why it matters
Without a harness, evaluation often becomes a collection of manual prompts and remembered impressions. Repeatable execution makes measured changes visible and lets engineers compare prompt, model, retrieval, or configuration changes against the same cases.
Capturing intermediate artifacts also helps separate retrieval failures from generation failures instead of reducing the entire system to one final score.
One important nuance
A repeatable harness does not make an unrepresentative dataset useful. Test cases, rubrics, and scorers can encode blind spots or reward the wrong behavior. Keep the dataset versioned, include important failure modes, review score distributions rather than only averages, and inspect examples where automated and human judgments disagree. Use production observations to add cases, but protect sensitive user content.