Definition
Pairwise evaluation presents two candidate outputs or system results for the same case and asks an evaluator to judge them against the same criteria. The result can be a preference for A or B, a tie, or a relative score. The evaluator can be a person, a model judge, or a task-specific rule.
The judgment is relative to that pair. Pairwise evaluation is one form of comparative evaluation, which also includes ranking several candidates. Pointwise evaluation scores each candidate independently, even when those scores are compared later.
Simple example
A team tests two prompt versions against the same question about failed imports. The runbook says a failed import requires a manual retry. Both versions receive the same question and runbook excerpt.
Answer A says the system retries automatically. Answer B says an operator must retry the import, but adds an unsupported claim that failed jobs are deleted after 24 hours. The review criteria put the correct retry instruction first, so the reviewer prefers B despite its unsupported deletion claim. That preference is one pairwise judgment. It does not give B a passing accuracy score.
Why it matters
Choosing between two concrete answers can be easier than assigning each one a number on a fixed scale. Across a representative set of cases, a team can count wins, losses, and ties to estimate how often one version is preferred over another under the chosen criteria. Keep the input, evidence, and judging criteria the same for both candidates so the comparison tests the intended change.
One important nuance
Presentation can affect the result. A reviewer or model judge may favor the first answer or mistake a longer answer for a better one. Hide candidate identities when possible, vary their order, and allow ties. Inspect cases where the preferred answer still violates a requirement. A relative win does not mean the answer is ready to use.