AI Concepts category

Evaluation

Concepts for measuring AI-system behavior and output quality.

AI-as-a-Judge uses a model to evaluate outputs against defined criteria, references, or competing responses.

Benchmark contamination occurs when evaluation examples enter model training or tuning data, making test scores less trustworthy.

Comparative evaluation judges outputs or systems directly against one another rather than assessing each independently.

Drift detection checks whether an AI system's inputs, outputs, quality, or runtime behavior have changed over time.

An evaluation criterion names one property of an AI output or behavior that an evaluator should assess.

Written instructions for applying evaluation criteria consistently to AI outputs or behavior.

An evaluation harness runs cases against an AI system in a repeatable way and records outputs, measurements, and run configuration.

Evaluation-driven development uses repeatable tests to decide how an AI system should change and whether each change helped.

Exact-match evaluation checks whether a system output equals an expected value under a defined comparison rule.

Factual consistency assesses whether an output's claims are compatible with the reference information being evaluated.

Faithfulness measures whether an answer's claims are supported by the evidence supplied to the model.

Functional correctness describes whether an output satisfies executable requirements, regardless of how closely it resembles a reference answer.

Ground truth is target data treated as correct for a specific evaluation case.

Pairwise evaluation compares two candidates on the same task and records a preference, tie, or relative score.

Pointwise evaluation scores one AI output against defined criteria without directly comparing it with another output.

Reference-based evaluation compares a system output with known answers, evidence, or other designated reference data.

Reference-free evaluation assesses an output against criteria without requiring a canonical answer.

A scoring rubric defines criteria and score levels that guide evaluation of an output.