Definition
AI-as-a-Judge uses a model to evaluate one or more generated outputs. The judge might score a response against a rubric, compare candidates, identify defects, or explain a decision. Depending on the task, it may receive the original input, a reference answer, retrieved evidence, or other candidate outputs.
The generator and judge can use the same underlying model or models from different families. Using a different model does not guarantee an unbiased or correct assessment.
Simple example
An evaluation harness sends a support question, the evidence retrieved by the application, and the generated answer to a judge model. The rubric asks whether each factual claim is supported by that evidence and requests a structured score with a short rationale. The harness records the result beside deterministic checks and occasional human review.
Why it matters
Many useful qualities, such as relevance, completeness, and faithfulness, are difficult to capture with exact string comparison. A model judge can evaluate larger test sets more quickly than manual review and can provide diagnostic explanations that help teams find recurring failure patterns.
It is most useful as a measured evaluation instrument, not as unquestioned ground truth.
One important nuance
A judge can give different answers on repeated runs and favor outputs that resemble its own style. Its scores may not agree with human ratings. Judgments can change with rubric wording, verbosity, or the judge model. Pairwise comparisons can also be sensitive to candidate order. A judge can miss errors when the supplied reference is incomplete. Check the judge against human-labelled examples, swap candidate order where it matters, track judge versions, and keep human review for high-impact decisions.