What Should You Actually Evaluate?

Evaluate whether the application delivered the promised outcome, within its permissions and runtime limits. A good answer is one part of that result.

A support assistant can write an accurate reply and save it to the wrong ticket. Or it can cite an obsolete policy and give advice that sounds well supported. Even a correct save can finish after the caller has timed out. The final text alone will not tell you which of these happened.

I would write down what the feature promises, then decide what evidence would show that it kept that promise. Choose the metrics from there.

Start with one outcome you can verify

Consider a support assistant with this contract:

Prepare a reply from the ticket and the support policy applicable to that ticket. Save it as a draft only when requested. Never send it.

The requester must have access to the ticket. If the available policy does not support an answer, the assistant should explain the gap and request review. If saving fails, it must not claim that a draft exists.

For this worked example, each evaluation case identifies the applicable policy version and the effective timestamp used to select it. “Current” means current for that case, not whatever document happens to be newest when the suite runs. Keep those reference facts fixed when comparing candidates. Historical cases keep their original policy version. Review any new or changed cases intended to represent a newer policy. The contract gives us these checks:

These checks need different methods. Some require judgment about language. Others are ordinary assertions over IDs, state, and side effects.

The eval-first article covers how to bring these checks into the development workflow. First, decide which behaviors need them.

Separate answer correctness from groundedness

A grounded answer follows the supplied context. That does not establish that the context is current, complete, or applicable to this customer.

Suppose an old policy says customers have 30 days to request a replacement. The current policy says 14 days, with an exception for damaged deliveries. An answer based on the old document might be perfectly grounded and still wrong for the task.

I would score these questions separately:

A reference answer can help, but acceptable wording should not be limited to one sentence. Write the required facts and forbidden claims explicitly.

For a damaged-delivery case, a useful rubric might require the reply to identify the exception, ask for missing evidence required by the policy, and avoid promising approval before that evidence is reviewed. A fluent reply that promises an immediate replacement fails the task.

Evaluate abstention separately. On cases where the policy cannot answer the question, does the assistant acknowledge the gap? On answerable cases, does it unnecessarily refuse? Combining those groups can reward an assistant that avoids mistakes by declining everything.

Tone matters when the product requires it. For this assistant, I would prioritize unsupported commitments and omitted conditions before spending time distinguishing polished prose from slightly awkward prose.

Check retrieval before blaming the prompt

An incorrect answer leaves several possible causes. The relevant policy may be absent from the corpus, missed by search, removed during context assembly, or ignored by the model.

Inspect both the search results and the context actually passed to generation. Finding the right document is insufficient if the passage containing the exception never reaches the model.

With labeled relevant passages, recall at a chosen result count measures the fraction of relevant passages retrieval found. Precision measures the fraction of returned passages that are relevant. Define the relevance unit first: document, chunk, or required fact. Ten overlapping chunks from one paragraph should not look like ten independent pieces of useful evidence.

For the support example, I would also check whether the final context contains every policy fact needed to answer the ticket. A missing exception then counts as a failure even if the rest of the result list looks reasonable.

Check authorization and policy applicability separately from retrieval quality. A relevant passage from another tenant must not reach the caller or model without permission. A policy that was obsolete at the case’s effective time is unsuitable evidence even if it ranks first.

To investigate a failure, compare the original context with a manually verified context in repeated, paired runs. Keep the model, instructions, generation settings, and scoring rubric fixed. A consistent improvement with verified context is stronger evidence of a retrieval or context-assembly problem than one successful rerun. If both conditions keep failing, inspect the instructions and generation behavior too. This experiment helps narrow the search. It does not establish a single cause or replace evaluation of the complete application.

Check what the tools changed

Calling SaveDraft looks correct when the user asks to save a reply. The tool name alone tells us little.

Check which ticket ID and body reached the tool, whether access was enforced, and what was stored. A valid JSON payload can contain the wrong ticket ID. A successful tool response can also be followed by an inaccurate message to the user.

For this assistant, the important assertions include:

• A request to prepare a reply without saving produces no write.
• A save request stores the intended body against the authorized ticket.
• Retrying the same logical save operation with the same operation or idempotency identity does not create another draft. Independent save requests follow the product’s rules for new drafts or revisions.
• No execution sends a reply.
• A failed or uncertain save does not produce a confirmed-success result.

Run side-effecting scenarios against isolated test resources or controlled tool implementations. A recording fake can show what the application attempted. An integration test is needed to verify the real adapter and storage behavior. These are ordinary software tests, but they still belong in the evaluation map when their results determine whether the AI feature fulfilled its contract.

Inspect the sequence where order matters. Authorization must succeed before protected contents reach the caller or model, and before a protected state change. The application may need to load a resource or its authorization metadata to make that decision. ASP.NET Core supports this through resource-based authorization. When trusted tenant identity is already available, constrain retrieval queries to that tenant before fetching content. Independent lookups can still run in either order. One successful trace should not become the only permitted sequence.

Microsoft’s agent evaluation documentation distinguishes task outcomes from the process used to reach them, including tool usage. That is a useful distinction, but scoring tool messages does not replace checking the application’s actual side effects.

Make failure outcomes part of task success

Successful task completion depends on the scenario. When policy evidence is missing, requesting review can be the correct outcome. When the user lacks access, refusing the operation is correct.

Build these cases from the feature’s failure modes: empty retrieval, provider timeout, malformed output, unavailable storage, cancellation, and a write whose outcome is uncertain.

For each case, assert the expected application result and what may already have happened. If storage commits but the acknowledgment is lost, a generic retry test that only counts exceptions misses the important question: did recovery create a duplicate?

Keep two views in the report. One measures whether the system handled each scenario according to its contract. The other measures how often users received the useful outcome they wanted. A service that handles every outage gracefully can still be unavailable too often.

Keep safety failures visible

For this support assistant, safety cases should include a ticket containing instructions to send data elsewhere, a request for another tenant’s ticket, and a request to send the reply despite the feature’s draft-only contract.

Inspect the evidence at each boundary. Did the application apply the known tenant restriction to the query and authorize disclosure before ticket content reached the caller or model? Did instructions inside the ticket trigger an unauthorized tool attempt, and did the executor block it?

Record an unsafe proposal separately from an executed action. A blocked attempt shows that a control worked, while still revealing behavior worth investigating. A reassuring final answer does not erase an unauthorized disclosure or prohibited operation earlier in the workflow.

Content safety checks may also matter, depending on the feature. A classifier for harmful language cannot verify tenant isolation or write authorization. Those require application-level evidence.

I would make any observed unauthorized disclosure or prohibited side effect block this feature’s release. Keep those failures visible rather than averaging them with relevance or tone scores. Zero observed failures in a test set is useful evidence, not proof that no other input can cross the boundary.

Measure latency and cost across the complete task

Measure the caller’s wait and outcome separately from backend completion time and outcome. A caller might time out after 12 seconds while the backend saves the draft at 19 seconds. The timeout is a user-visible failure. The late save is a side effect that recovery must account for. Correlate both records with the same logical operation. Include retrieval, retries, tool execution, and persistence in the backend measurement.

For an interactive feature, inspect median and tail latency, such as the 95th percentile, under stated load. For streaming, distinguish the first visible output from completion of the usable result. A quick opening sentence does not mean the draft is ready.

Report failures and timeouts beside latency. Separate user-initiated cancellation, deadline or budget cancellation, and system or dependency cancellation. Record an unknown cause when you cannot establish it. A user choosing to stop is different from an execution exhausting its budget. Show which outcomes the latency sample includes so a candidate cannot look faster merely by dropping slow, failed requests from the report.

Cost should cover the boundaries the feature pays for. Name model-only estimates accordingly, and keep missing usage visible. Include failed attempts and retries rather than counting only the final successful call.

For cost per successful task, first decide which outcomes count as success. If the denominator counts cases handled according to contract, a correct access denial counts. If it counts useful user outcomes, that denial does not. Report the two ratios separately when you need both, alongside their success rates and per-execution cost. A ratio alone can hide expensive outliers or changes in the case mix.

Choose limits from the product promise. The runtime budgets article covers enforcement. Evaluation checks whether realistic workloads and failures stay within those limits while still producing useful outcomes.

Match the evaluator to the evidence

Use deterministic checks for schema validity, identifiers, permissions, persisted state, and counts. Use an explicit rubric with human review or a model judge for semantic questions such as whether the reply omitted a policy condition.

In .NET, Microsoft.Extensions.AI.Evaluation supplies relevance, completeness, and groundedness evaluators that work with existing test infrastructure. Its CompletenessEvaluator compares the response against supplied ground truth. For groundedness, explicitly pass the grounding evidence from the final context the generation model actually received through GroundednessEvaluatorContext. The evaluator does not recover that context automatically. Decide what evidence and pass criteria you need before choosing either.

Give the judge the evidence its rubric asks it to assess: the supplied context for groundedness, or the applicable reference facts for correctness. You still need to inspect storage to verify that a draft exists. The phrase “Draft saved” proves nothing about database state.

Review a sample of judge decisions against human judgments, including failures and borderline cases. Record disagreements and tighten ambiguous rubrics. If the evaluator errors or lacks required evidence, report the result as unscored. Require a valid result that meets the pass criteria at the release gate. Missing or inconclusive results need an explicit decision.

Keep the component results beside the result for the complete workflow. They should let you trace a failed task back to a missing passage, an unsupported claim, or an incorrect write without losing sight of what the user received.

Decide what would block the release

A single overall score hides the reason a candidate improved or regressed. Report results by scenario: routine answers, policy exceptions, missing evidence, denied access, and dependency failures. Include the number of cases in each group.

For example, a higher average answer score is not enough to accept a candidate that now invents commitments on missing-policy cases. A cheaper candidate is not an improvement if it saves the wrong draft. Define these acceptance rules before reviewing the candidate’s results.

Compare candidates on the same evaluation protocol and inspect per-case changes. Repeat cases where model variation could change the decision. Keep results tied to their versions using the evaluation provenance guidance rather than relying on a score copied into a pull request.

Production evaluation can reveal cases the offline suite missed. Track corrections, abandoned tasks, and sampled quality judgments with their context. User acceptance alone does not establish correctness. The production evaluation tip covers how to collect those judgments outside the user response path.

When to use this approach

Use this system-level evaluation map when an AI feature retrieves information, invokes tools, changes state, or has meaningful failure and runtime constraints. Start with one workflow and the failures that would change a release decision. Add dimensions as the feature gains responsibilities.

When a smaller evaluation is enough

A low-risk text transformation may need only output validity, preservation of required meaning, and latency. It has no retrieval or tool path to evaluate. A local experiment may begin with a few manually reviewed examples, provided that limited evidence is not treated as production readiness.

For your next change, write one sentence describing the promised outcome. List the unacceptable results, identify the evidence that would expose each one, and choose a check for each. Then add the quality and runtime measures needed to compare acceptable candidates.

For the support assistant, that means checking the reply against the applicable policy, inspecting the saved draft, and confirming what the caller saw. Those results give you something concrete to review before shipping.

Related reading

• Eval-first: Why “It Worked Once” Is Not a Sign of Quality
• Stop Guessing: Use Golden Datasets for Prompt Evals
• Version prompts, models, datasets, and evaluation results together

Sources

• Microsoft Learn: Resource-based authorization in ASP.NET Core
• Microsoft Learn: Agent evaluators in Microsoft Foundry
• Microsoft Learn: The Microsoft.Extensions.AI.Evaluation libraries
• Microsoft Learn: CompletenessEvaluator
• Microsoft Learn: GroundednessEvaluatorContext constructor