Measure retrieval misses separately from model failures

When a RAG answer is missing or wrong, the model is not always the failing component.

Retrieval may have returned no candidates, only results below the relevance threshold, or an actual dependency error. Represent those outcomes before constructing the prompt:

public enum RetrievalOutcome
{
    Hit,
    NoCandidates,
    BelowThreshold,
    FilteredOut,
    DependencyFailure
}

public sealed record RetrievalResult(
    RetrievalOutcome Outcome,
    IReadOnlyList<DocumentChunk> Chunks);

The application can then apply an explicit policy:

RetrievalResult retrieval = await retriever.SearchAsync(
    query,
    cancellationToken);

if (retrieval.Outcome is RetrievalOutcome.DependencyFailure)
{
    throw new InvalidOperationException(
        "Vector store or embedding dependency failed.");
}

if (retrieval.Outcome is not RetrievalOutcome.Hit)
{
    return SupportAnswer.WithoutContext(retrieval.Outcome);
}

return await answerGenerator.GenerateAsync(
    query,
    retrieval.Chunks,
    cancellationToken);

NoCandidates and BelowThreshold are semantic misses that can degrade gracefully. FilteredOut means candidates were removed by tenant, permission, or policy filters before context assembly. The user-facing fallback should not reveal that protected documents exist, but the internal outcome still matters. Only emit FilteredOut when the retrieval pipeline can determine it without running an unsafe unfiltered search.

DependencyFailure is different. A vector store or embedding outage must remain an operational failure, not turn into a polite no-context answer. The example throws at the application boundary. In production, preserve the original exception so operators can diagnose the failing dependency.

This prevents empty or low-confidence context from being passed to the model as if retrieval succeeded.

Record retrieval telemetry separately from model telemetry:

• outcome
• result count
• latency
• retriever or low-cardinality index alias
• configured threshold
• top score bucket

Keep metric dimensions such as Outcome low-cardinality. Put frequently changing index or collection versions in trace or log metadata instead of metric dimensions. Do not put raw queries, document text, or customer data into telemetry by default.

Production telemetry tells you how often each runtime outcome occurs. Retrieval evaluation answers a different question: whether the correct evidence appears for a representative query set. During evaluation, track whether the expected document appears in the top results and use retrieval metrics such as precision at K, recall at K, or mean reciprocal rank when they fit the test. A low exception rate does not prove good retrieval, and a high model success rate can hide that the model answered without useful evidence.

This boundary makes incident diagnosis faster. You can tell whether to inspect the query, filters, chunking, index freshness, threshold, vector store, or model instead of treating every bad answer as a prompt problem.