Definition
Comparative evaluation assesses two or more outputs, models, or system configurations relative to one another. An evaluator may select a preferred response, rank several candidates, or state that they are equivalent according to defined criteria.
Human reviewers, a model judge, or an automated rule can compare candidate results directly. In absolute evaluation, each candidate is judged independently against a criterion or scale. Comparing those separate judgments later does not make the underlying evaluation comparative.
Simple example
A team evaluates two prompt versions on the same 100 support questions. For each question, a reviewer sees both answers in randomized order and chooses which better satisfies correctness, completeness, and policy constraints. The analysis reports wins, losses, and ties rather than comparing two averages produced by separate scoring sessions.
The team keeps the question, retrieved evidence, and model configuration fixed so the prompt change is the main controlled difference.
Why it matters
Reviewers often find it easier to identify the better of two concrete outputs than to assign a consistent score on an abstract scale. Comparative evaluation is useful for model selection, prompt experiments, regression checks, and preference-data collection.
It can reveal a meaningful difference even when independent scores cluster closely, but it should still cover representative cases and practical failure modes.
One important nuance
Pairwise judgments can be biased by presentation order, verbosity, formatting, or the evaluator’s preference for a familiar style. Randomize or swap candidate positions, allow ties, hide irrelevant identity information, and use a clear rubric. A relative winner can still be unacceptable in absolute terms, so preserve minimum quality and safety gates alongside the comparison.