Run production evaluations outside the user response path

Suppose a chat request needs one model call. Add a relevance evaluator backed by another model and it now needs two. Await both, and the evaluation has become user-facing latency. An evaluator outage can also fail the request even though the application already produced an answer.

Microsoft’s .NET evaluation libraries call this online evaluation: scoring responses from a deployed application. I use production evaluation here because shadow evaluation can also mean sending production traffic to a candidate prompt or model without serving its result.

Here, production evaluation has a narrow job. It observes behavior without deciding the outcome of the current request. Generate the answer and store the data needed for evaluation. Then hand a small work item to a queue so a separate worker can calculate the scores.

The application abstractions below keep the evaluation-related work in the request path down to that handoff:

var response = await assistant.RespondAsync(
    request,
    cancellationToken);

var turn = await conversations.SaveAsync(
    request,
    response,
    cancellationToken);

if (productionEvaluationSampler.ShouldEvaluate(turn))
{
    var work = new ProductionEvaluationWork(
        turn.Id,
        evaluationDefinition.Version);

    if (!productionEvaluationQueue.TryEnqueue(work))
    {
        productionEvaluationMetrics.RecordDroppedSample();
    }
}

return TypedResults.Ok(response);

When the worker can load the turn from a governed store, put its identifier and the evaluation definition version on the queue. Do not copy a complete conversation into a broker just because it is convenient. If there is no existing conversation store, persist a dedicated evaluation record with the fields the worker is allowed to read.

The saved result needs provenance for both the generated answer and the evaluation. Record the generation prompt version, model, and configuration that produced the answer. The immutable evaluation definition covers the evaluator, its model configuration, and the rubric or scoring policy. Change the definition version whenever one of those inputs changes. Keep these provenance rules out of the request handler.

Make TryEnqueue a non-blocking contract. It either accepts the item immediately or reports that the bounded queue is full. A process-local Channel<T> using BoundedChannelFullMode.Wait can implement this with ChannelWriter.TryWrite; WriteAsync may wait for capacity. If evaluation data is best effort, dropping a sample can be a deliberate policy. Count drops by feature and time window because saturation during traffic spikes can bias the evaluation set toward quieter periods.

Use a durable broker when accepted work must survive a worker restart. That protects a message after the application publishes it. It does not close the crash window between saving the turn and publishing the message. A transactional outbox can close that gap only when the turn and outbox record share a transactional store. Write both records in one transaction, then let a dispatcher publish the outbox entry.

Durable queues can deliver the same work more than once. Treat the pair of turn.Id and evaluationDefinition.Version as the evaluation identity. The worker can upsert the result or skip work that has already completed for that identity.

Do not replace the queue with _ = evaluator.EvaluateAsync(...). That task still runs inside the web process and can capture request-scoped services. Failures and shutdown behavior then have no clear owner.

A hosted service removes evaluation from the synchronous control flow, but it still shares process resources with HTTP requests. A queue changes when the work runs. Move the consumer to a separate worker when it needs independent scaling or resource isolation.

Production samples may contain user input, retrieved passages, tool results, or generated text. Decide what may be retained before the worker sees it. Apply sampling and redaction before the handoff when possible, and give the evaluation pipeline its own concurrency and cost limits.

This tip covers where production evaluations run, not which metrics they should calculate. The results can reveal quality drift and inform a later release decision. They cannot protect a response that has already reached the user.

If a score can change whether the current answer may be shown, the check must finish before the response returns. That is a runtime control, not this asynchronous production evaluation.