Definition
Reference-based evaluation assesses an output by comparing it with designated reference data. The reference might be an expected answer, a set of required facts, a source passage, a structured record, or one or more reviewed examples of acceptable output.
The comparison method depends on the task. It may use exact matching, semantic similarity, deterministic checks, human review, or a model judge with access to the reference.
Simple example
An extraction system reads invoices and returns supplier, currency, and total fields. The evaluation dataset contains reviewed values for each invoice. A harness compares normalized fields, reports exact matches, and records field-level errors.
For a support answer, the reference may instead list facts that must be present and evidence that must not be contradicted. Multiple wordings can then pass without matching one canonical sentence.
Why it matters
A fixed reference gives evaluation a stable target for comparing prompts, models, and releases. It is especially useful for tasks with verifiable answers or required properties.
Granular scorers, often paired with structured references, can help locate failures. Engineers can see whether an answer missed a required fact, introduced a contradiction, or merely used different wording.
One important nuance
A reference is not automatically complete ground truth. Open-ended tasks can have several valid answers, and a single reference response may encode one style or approach rather than every acceptable result. References can also contain labeling mistakes or become stale. Design scorers around the task’s real invariants, allow legitimate alternatives, version the reference set, and review failures before treating every mismatch as a model error.