A score such as 0.84 looks precise. On its own, it says surprisingly little. If the next run scores 0.79, you need the exact prompt, target model, dataset, evaluators, and configuration before you can explain what changed.

Create one manifest for each evaluation run. Record the system inputs that can move the result:

  • application or Git revision
  • prompt revision or content hash
  • configured deployment and observed target-model identity
  • generation, tool, retrieval, and schema configuration
  • retrieval corpus or index revision when applicable

Record the evaluation protocol as well:

  • dataset revision and case-selection rules
  • evaluator set and the revisions for rubrics, metric definitions, scales, and pass criteria
  • evaluator-model identities when using models as judges
  • repetition count and aggregation rules

Keep the run metadata and outputs tied to the same manifest:

  • run ID and timestamp
  • latency and token usage
  • aggregate and per-case results

The exact fields depend on the system. This metadata feels boring until a score drops and nobody can reconstruct the setup. Create the manifest before the first case runs. Fill in observed values as responses arrive. When the run ends, keep the completed manifest tied to the final results.

public sealed record EvaluationRunManifest(
    string RunId,
    string ApplicationRevision,
    string PromptRevision,
    string ConfiguredProvider,
    string ConfiguredDeployment,
    string? ObservedTargetModel,
    string DatasetRevision,
    string EvaluatorSetRevision,
    string MetricDefinitionRevision,
    string PassCriteriaRevision,
    IReadOnlyList<string> EvaluatorModels,
    int Repetitions,
    string AggregationRevision,
    string ConfigurationRevision,
    string? RetrievalCorpusRevision,
    DateTimeOffset StartedAtUtc);

public sealed record EvaluationMetricSummary(
    string Evaluator,
    string Metric,
    double Score,
    string Scale,
    int SampleCount);

public sealed record EvaluationSummary(
    int CaseCount,
    int PassedCaseCount,
    IReadOnlyList<EvaluationMetricSummary> Metrics);

public sealed record EvaluationArtifact(
    EvaluationRunManifest Manifest,
    EvaluationSummary Summary);

Do not replace relevance, groundedness, safety, and other metrics with one undifferentiated average. Keep a summary for each evaluator and metric, including its scale and sample count. You can calculate a documented composite, but keep the metrics behind it.

Serialize the EvaluationArtifact beside the detailed per-case results, or link both to the same immutable manifest. Both layouts work as long as the connection cannot drift.

Record configured and observed model identities

A deployment name is useful for routing, but it may not identify the model revision that handled a request. Record the configured provider and deployment. Also capture the model identity returned by the provider when it is available.

Some providers expose that identity per request. Put it on the individual case result in that situation instead of assuming one model identity covers the whole run.

If the provider offers an immutable model snapshot, use that identifier. If it exposes only a moving alias, record what you can observe. If no precise revision is available, say so in the manifest instead of inventing precision you do not have.

Apply the same rule to evaluator models when judge routing can vary during a run. A score can move because the application changed, because the judge changed, or because both changed. Without the evaluator revision and evaluator-model identity, you cannot separate those explanations.

Separate regression detection from causal attribution

Keep the evaluation protocol fixed and a lower result starts to mean something. A meaningful drop from release A to release B is evidence of a system regression, even when several parts of the system changed. Check the sample size and run-to-run variation before treating a small difference as real. The evaluation still cannot tell you which system change caused it.

Once the dataset, case-selection rules, evaluator, rubric, judge model, metric definition, or aggregation changes, the raw scores are no longer directly comparable. Release B may score lower simply because its dataset contains harder cases. Keep both manifests, but do not call the difference a regression on its own.

To find the cause, keep the other system inputs fixed and change one variable deliberately.

Versioning does not make model output deterministic. Repeated runs can differ even when the manifest is unchanged. Use repetitions, distributions, or confidence ranges when that variation matters. The manifest tells you what ran. It does not promise identical output.

Never keep an evaluation score by itself. Keep the aggregate and per-case results permanently tied to the exact manifest that describes the evaluated setup. They may live in different stores, but the link between them must be stable.