Evaluating Retrieval Quality in .NET

Qdrant can rank a shutdown chunk first and still miss the passage about cancellation. Check whether retrieval found the required evidence before passing results to a model.

We’ll evaluate actual Qdrant results using labeled questions and a custom IEvaluator. The evaluator calculates the metrics in C# without calling a judge model. The reporting library saves the results, and the aieval console tool produces the HTML report.

The querying article built the read path. Payload modeling and metadata filters restricted results to published runbooks, while context assembly selected the text for generation. Here, we evaluate the ranked results before assembly.

A complete, standalone version of this example is available in the sample repository.

The excerpts below explain the evaluation implementation. To run the complete experiment, use sample 07. It includes corpus preparation and validation in a standalone program that builds on the retrieval approach from the earlier articles.

Why a relevant first result is not enough

The shutdown runbook contains these passages:

For example, consider the question: “How do I stop accepting new queue items and pass cancellation to operations during shutdown?” The answer requires information from both shutdown chunks. In a measured local run:

Both result lists place a relevant chunk first. Reciprocal rank cannot distinguish these outcomes.

The two chunks also contain the same deadline paragraph. For the question “Should I drain all queued work during shutdown?”, either chunk provides the necessary condition. Labeling both as relevant makes sense, but retrieving only one doesn’t always mean the answer is incomplete. Chunk recall measures labeled passages, not individual facts. Identify the required evidence before reviewing the ranking. A plausible top result can make missing information easy to overlook.

Build a small labeled evaluation set

The original filter includes three chunks from a single published runbook, so returning all three makes Recall@3 trivial. Sample 07 adds four more published runbooks as competition:

For the complete fixture and indexing code, see sample 07 in the GitHub repository.

Including the original draft runbook and a published note, the fixture contains eleven stored points. Only seven chunks qualify for the published-runbook query. The HTTP shutdown passage is a plausible competing result: it shares shutdown terminology but does not explain a background queue.

Sample 07 uses a separate collection, blog-samples-07-eval-ollama-v1, to keep the evaluation corpus isolated from earlier examples. The pipeline continues to use the same chunking rules, payload contract, and filters. Its console application implements --index to prepare the corpus and --evaluate to validate and read it without rewriting document vectors. Both modes check the exact point count and independently read back expected identities and payloads. These checks don’t confirm vector equality or provenance.

Use .NET 10, Qdrant and Qdrant.Client 1.19.0, OllamaSharp 5.4.30, and Microsoft.Extensions.AI.Evaluation with Microsoft.Extensions.AI.Evaluation.Reporting 10.9.0. The local embeddinggemma:300m profile uses 768 dimensions and cosine distance, applying title: ... | text: ... for documents and task: search result | query: ... for queries. The standalone sample includes the provider setup and helpers. Extend its two-minute cancellation deadline if embedding calls run longer locally.

Follow the repository README to start Qdrant and configure Ollama with embeddinggemma:300m. Then run these commands from the repository root. The -- separator passes the following argument to the sample application:

dotnet run --project samples/07-retrieval-evaluation/ollama/Sample.csproj -- --index
dotnet run --project samples/07-retrieval-evaluation/ollama/Sample.csproj -- --evaluate

Run --index before the first evaluation. Repeat --evaluate against the prepared corpus when comparing retrieval settings.

The Azure OpenAI path from Indexing Your First Documents can use the same evaluator and relevance labels. It needs its own collection and document embeddings generated with the selected Azure model, along with the matching query preprocessing. The measured results in this article come from EmbeddingGemma. Evaluate the Azure configuration separately rather than assuming it produces the same rankings.

The experiment defines seven answerable cases and two unsupported questions. The Key includes the document and the current section-paragraphs-80w-overlap1-v1 chunking version, ensuring that identical chunk names from different sources do not match by mistake:

ChunkKey Key(string document, string chunk) =>
    new(document, chunkingVersion, chunk);

RetrievalCase[] cases =
[
    new("drain-deadline", "conditions", "Should I drain all queued work during shutdown?",
        [Key("worker-shutdown", "shutdown/p0"), Key("worker-shutdown", "shutdown/p1")]),
    new("cancel-draining", "paraphrase", "How do I pass cancellation to asynchronous work while draining the queue?",
        [Key("worker-shutdown", "shutdown/p1")]),
    new("shutdown-sequence", "multiple-passages", "How do I stop accepting new queue items and pass cancellation to operations during shutdown?",
        [Key("worker-shutdown", "shutdown/p0"), Key("worker-shutdown", "shutdown/p1")]),
    new("bounded-producer", "technical-term", "What happens to a producer when a bounded Channel with FullMode.Wait is full?",
        [Key("worker-shutdown", "capacity/p0")]),
    new("http-pooling", "technical-term", "How does IHttpClientFactory reuse HTTP handlers?",
        [Key("http-client-lifetime", "clients/p0")]),
    new("retry-budget", "conditions", "How should HTTP retries use backoff, jitter, and a total deadline?",
        [Key("http-request-retries", "retries/p0")]),
    new("restart-recovery", "paraphrase", "How can a worker recover unfinished jobs after a process restart?",
        [Key("durable-work-queue", "recovery/p0")])
];

string[] unsupportedQuestions =
[
    "Which exact timeout does our production ingress use?",
    "How many days does our company retain completed job records?"
];

public sealed record RetrievalCase(
    string Id, string Category, string Question, ChunkKey[] Relevant);

The baseline applies binary judgments: assign grade 2 to each listed relevant chunk and grade 0 to all other eligible chunks. For cancel-draining, only shutdown/p1 directly answers the question. The deadline overlap in shutdown/p0 doesn’t count. In future datasets, grade 1 might indicate useful supporting evidence, but that would require a separate labeling policy.

Every eligible chunk receives a judgment in this setup. Don’t treat unjudged results as irrelevant if you expand to a larger corpus. Always review these positional labels when source text or chunk boundaries change. The chunking version describes splitting rules, not the source revision.

Calculate retrieval metrics in C#

Use these definitions at a fixed cutoff K:

Precision always uses K as its denominator, even if fewer results are returned. For example, if there’s one relevant result at K=3, Precision@3 is 1/3. If you measure precision over the actual number of returned results, give that metric a different name.

Each query yields a reciprocal rank. Average these values across answerable questions to compute MRR at the same cutoff. MRR does not indicate whether a second required passage was retrieved.

For nDCG, calculate gain as 2^grade - 1 and discount as log2(rank + 1), with ranks starting at one. The ideal ranking sorts all eligible judgments by grade and selects the top K. Unsupported questions do not have a relevant-chunk denominator or positive ideal gain. Evaluate their acceptance separately.

The calculation preserves Qdrant’s returned order:

public sealed record ChunkKey(string DocumentId, string Version, string ChunkId);

public sealed record RankingScores(
    double Precision, double Recall, double ReciprocalRank, double Ndcg);

public static class RetrievalMetrics
{
    public static RankingScores Calculate(
        IReadOnlyList<ChunkKey> ranked,
        IReadOnlyDictionary<ChunkKey, int> judgments, int k)
    {
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(k);
        if (ranked.Distinct().Count() != ranked.Count)
            throw new ArgumentException("Duplicate chunk identities.");
        if (judgments.Values.Any(grade => grade is < 0 or > 2))
            throw new ArgumentException("Grades must be 0, 1, or 2.");
        if (ranked.Any(key => !judgments.ContainsKey(key)))
            throw new ArgumentException("Returned chunk has no judgment.");

        int relevantCount = judgments.Count(pair => pair.Value > 0);
        if (relevantCount == 0)
            throw new ArgumentException("Score no-answer cases separately.");

        int[] grades = ranked.Take(k).Select(key => judgments[key]).ToArray();
        int hits = grades.Count(grade => grade > 0);
        int firstHit = Array.FindIndex(grades, grade => grade > 0);
        double ideal = Dcg(judgments.Values.OrderDescending().Take(k));

        return new RankingScores(
            (double)hits / k,
            (double)hits / relevantCount,
            firstHit < 0 ? 0 : 1d / (firstHit + 1),
            Dcg(grades) / ideal);
    }

    private static double Dcg(IEnumerable<int> grades) =>
        grades.Select((grade, index) =>
            (Math.Pow(2, grade) - 1) / Math.Log2(index + 2)).Sum();
}

An empty ranking with valid relevant labels is a scored miss. Input errors include duplicate identities, missing judgments, invalid grades, or an empty relevant set. Keep these cases distinct so a broken adapter or dataset does not appear as a simple relevance failure.

Before connecting to Qdrant, test a hand-crafted order [C, A, B] with grades C=0, A=2, B=1. At K=3, this yields Precision=2/3, Recall=1, reciprocal rank=1/2, and nDCG around 0.659. Also check short lists, cutoff boundaries, duplicate identities, and evaluator cancellation. These arithmetic checks require no services.

A second controlled test uses the shutdown labels. Place the irrelevant HTTP passage before both queue passages: [0, 2, 2]. Precision@3 stays at 2/3, Recall@3 at 1, but reciprocal rank drops to 1/2 and nDCG to about 0.693. This reordered fixture simulates a ranking failure. It does not reflect an actual Qdrant regression.

Connect Qdrant to Microsoft’s evaluation framework

IEvaluator gives the C# calculations a common result format and reporting path. Assertions still test the arithmetic and enforce application acceptance criteria. The evaluator does not decide whether a release passes.

Return standard metrics

RankingContext contains the returned identities and their judgments. Its serialized evidence keeps the retrieved text, similarity scores, and eligibility metadata, allowing reports to explain low scores:

using System.Text.Json;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;

public sealed record RetrievedEvidence(
    ChunkKey Key, float Score, string Text,
    string DocumentType, string PublicationState);

public sealed class RankingContext : EvaluationContext
{
    public ChunkKey[] Ranked { get; }
    public IReadOnlyDictionary<ChunkKey, int> Judgments { get; }

    public RankingContext(RetrievedEvidence[] evidence,
        IReadOnlyDictionary<ChunkKey, int> judgments)
        : base("RetrievalRanking", JsonSerializer.Serialize(new
        {
            evidence,
            judgments = judgments.Select(pair => new { key = pair.Key, grade = pair.Value })
        }))
    {
        Ranked = evidence.Select(item => item.Key).ToArray();
        Judgments = new Dictionary<ChunkKey, int>(judgments);
    }
}

public sealed class RetrievalMetricsEvaluator : IEvaluator
{
    private readonly int _k;
    public IReadOnlyCollection<string> EvaluationMetricNames { get; }

    public RetrievalMetricsEvaluator(int k)
    {
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(k);
        _k = k;
        EvaluationMetricNames =
            [$"Precision@{k}", $"Recall@{k}", $"ReciprocalRank@{k}", $"nDCG@{k}"];
    }

    public ValueTask<EvaluationResult> EvaluateAsync(
        IEnumerable<ChatMessage> messages, ChatResponse modelResponse,
        ChatConfiguration? chatConfiguration = null,
        IEnumerable<EvaluationContext>? additionalContext = null,
        CancellationToken cancellationToken = default)
    {
        cancellationToken.ThrowIfCancellationRequested();
        NumericMetric[] metrics = EvaluationMetricNames
            .Select(name => new NumericMetric(name)).ToArray();
        var result = new EvaluationResult(metrics);

        try
        {
            RankingContext[] contexts = additionalContext?
                .OfType<RankingContext>().ToArray() ?? [];
            if (contexts.Length != 1)
                throw new ArgumentException("Supply exactly one RankingContext.");

            RankingContext context = contexts[0];
            RankingScores scores = RetrievalMetrics.Calculate(
                context.Ranked, context.Judgments, _k);
            double[] values =
                [scores.Precision, scores.Recall, scores.ReciprocalRank, scores.Ndcg];
            for (int i = 0; i < metrics.Length; i++)
            {
                metrics[i].Value = values[i];
                metrics[i].Reason = $"Scored {Math.Min(_k, context.Ranked.Length)} results at K={_k}.";
                metrics[i].AddOrUpdateContext(context);
            }
        }
        catch (ArgumentException error)
        {
            foreach (NumericMetric metric in metrics)
            {
                metric.AddDiagnostics(EvaluationDiagnostic.Error(error.Message));
                metric.Interpretation = new EvaluationMetricInterpretation(
                    EvaluationRating.Inconclusive, failed: true, reason: error.Message);
            }
        }

        return ValueTask.FromResult(result);
    }
}

The evaluator does not call a model and ignores ChatConfiguration. The runner provides an empty response to meet the interface, but this is not a generated answer. Invalid input leaves metric values unset and adds error diagnostics. A valid empty ranking results in zeroes. The runner should fail runs with input errors but include zero scores in overall averages.

The runner owns aggregation and the dataset denominator. The fixture text is authored. Before storing real document bodies, decide what evidence to retain and who can access it.

Query the fixed collection

Sample 07 includes the Qdrant client, embedding generator, query preprocessing, and payload helpers. The following excerpt shows the retrieval boundary after collection validation. The expected identities come from labels. The actual identities come from Qdrant:

var eligibility = new Filter
{
    Must =
    {
        MatchKeyword("chunking_version", chunkingVersion),
        MatchKeyword("document_type", "runbook"),
        MatchKeyword("publication_state", "published")
    }
};
float? threshold = null; // Baseline: no score-based acceptance cutoff.
SearchParams? searchParameters = null; // No per-query overrides; use server defaults.

async Task<ScoredPoint[]> SearchAsync(ReadOnlyMemory<float> vector, int k)
{
    var hits = await client.QueryAsync(collectionName, query: vector.ToArray(),
        filter: eligibility, limit: (ulong)k, scoreThreshold: threshold,
        searchParams: searchParameters,
        payloadSelector: new[] { "document_id", "chunk_id", "chunking_version", "document_type", "publication_state", "text" },
        vectorsSelector: false, cancellationToken: token);
    foreach (ScoredPoint hit in hits)
    {
        if (PayloadFields.RequiredString(hit.Payload, "publication_state") != "published" ||
            PayloadFields.RequiredString(hit.Payload, "document_type") != "runbook" ||
            PayloadFields.RequiredString(hit.Payload, "chunking_version") != chunkingVersion)
            throw new InvalidOperationException("Retrieval eligibility violation.");
    }
    return hits.ToArray();
}

static RetrievedEvidence ToEvidence(ScoredPoint hit) => new(
    new ChunkKey(
        PayloadFields.RequiredString(hit.Payload, "document_id"),
        PayloadFields.RequiredString(hit.Payload, "chunking_version"),
        PayloadFields.RequiredString(hit.Payload, "chunk_id")),
    hit.Score,
    PayloadFields.RequiredString(hit.Payload, "text"),
    PayloadFields.RequiredString(hit.Payload, "document_type"),
    PayloadFields.RequiredString(hit.Payload, "publication_state"));

async Task<ReadOnlyMemory<float>> EmbedQueryAsync(string question)
{
    ReadOnlyMemory<float> vector = await generator.GenerateVectorAsync(
        prepareQuery(question), cancellationToken: token);
    if (vector.Length != vectorSize) throw new InvalidOperationException("Query dimension mismatch.");
    return vector;
}

The mandatory filter limits candidates within Qdrant. If you apply filtering only after retrieving an unrestricted top K, you could lose eligible evidence before evaluation. The payload guard ensures that returned results meet the contract.

The runner embeds each question once and reuses that vector for K=1, 3, and 5. Keep the corpus, labels, filters, and score cutoff unchanged. Approximate searches may return different ranking prefixes as K changes. Check result identities if your analysis requires consistency. In this run, they matched. Note that recall against relevance labels differs from ANN recall with exact nearest neighbors.

Persist each question and configuration

Sample 07 constructs prepared from the complete source fixture before indexing or evaluation. The reporting excerpt builds the eligible identity set from these sources, not from the returned hits. Otherwise, a missing expected chunk could vanish from the denominator, making a miss appear as perfect recall.

The following loop shows the answerable-query reporting within --evaluate. It stores one scenario per question and K, sharing one execution ID:

using Microsoft.Extensions.AI.Evaluation.Reporting.Storage;

string execution = Guid.NewGuid().ToString("N");
var queryVectors = new Dictionary<string, ReadOnlyMemory<float>>();
foreach (string question in cases.Select(item => item.Question).Distinct())
    queryVectors.Add(question, await EmbedQueryAsync(question));

ChunkKey[] eligibleKeys = prepared
    .Where(item => item.Document.DocumentType == "runbook" &&
                   item.Document.PublicationState == "published")
    .SelectMany(item => item.Chunks.Select(chunk =>
        Key(item.Document.DocumentId, chunk.ChunkId)))
    .ToArray();

foreach (int k in new[] { 1, 3, 5 })
{
    var evaluator = new RetrievalMetricsEvaluator(k);
    var reporting = DiskBasedReportingConfiguration.Create(
        storageRootPath: "evaluation-results/ollama",
        evaluators: [evaluator],
        chatConfiguration: null,
        enableResponseCaching: false,
        executionName: execution);

    foreach (RetrievalCase testCase in cases)
    {
        if (eligibleKeys.Distinct().Count() != eligibleKeys.Length ||
            testCase.Relevant.Length == 0 ||
            testCase.Relevant.Distinct().Count() != testCase.Relevant.Length ||
            testCase.Relevant.Except(eligibleKeys).Any())
            throw new InvalidOperationException("Invalid relevance labels.");

        var judgments = eligibleKeys.ToDictionary(
            key => key, key => testCase.Relevant.Contains(key) ? 2 : 0);
        ScoredPoint[] hits = await SearchAsync(queryVectors[testCase.Question], k);
        var context = new RankingContext(hits.Select(ToEvidence).ToArray(), judgments);

        await using var scenario = await reporting.CreateScenarioRunAsync(
            $"Retrieval.K{k}.{testCase.Category}.{testCase.Id}",
            cancellationToken: token);
        EvaluationResult result = await scenario.EvaluateAsync(
            [new ChatMessage(ChatRole.User, testCase.Question)],
            new ChatResponse(new ChatMessage(ChatRole.Assistant, string.Empty)),
            additionalContext: [context], cancellationToken: token);

        if (result.Metrics.Count != evaluator.EvaluationMetricNames.Count ||
            !evaluator.EvaluationMetricNames.All(name =>
                result.Metrics.TryGetValue(name, out var metric) &&
                metric is NumericMetric { Value: not null }) ||
            result.ContainsDiagnostics(d => d.Severity == EvaluationDiagnosticSeverity.Error))
            throw new InvalidOperationException($"Unscored evaluation: {testCase.Id}");
    }
}

The label guard rejects duplicates, empty relevant sets, and references outside the eligible corpus. The result guard checks for the four declared metric names, numeric values, and the absence of error diagnostics. Disposing each scenario writes its result. Use a new execution ID for each run and keep the result directory as a CI artifact.

After evaluation finishes and all scenarios have been disposed, render the local sample’s report from the repository root. The repository includes a tool manifest that pins the matching aieval version:

dotnet tool restore
dotnet tool run aieval report --path evaluation-results/ollama --output evaluation-results/ollama/report.html

Review each metric’s evidence context along with its score. Calculate the average of per-query metrics across the seven answerable cases, including valid zeroes. This produces mean precision, mean recall, MRR, and mean nDCG. Always report the case count. Failed retrievals or missing scores should fail the execution, not simply be omitted from averages. The HTML report does not enforce any aggregation policy.

What the measured results tell us

The baseline evaluates seven eligible chunks for seven answerable questions, with no score threshold. The following are macro averages from a single local EmbeddingGemma run, rounded to three decimals:

Each case contributes equally:

There are nine relevant query-chunk pairs. At K=1, five questions achieve recall 1 while two have recall 1/2: (5 + 0.5 + 0.5) / 7 = 0.857. At K=3, all nine pairs are found in 21 result positions, resulting in mean precision 9 / 21 = 0.429.

The independently queried K=3 rankings are below. W means worker-shutdown, H means http-client-lifetime, R means http-request-retries, S means http-server-shutdown, and D means durable-work-queue. All identities use the same chunking version:

For drain-deadline, the HTTP shutdown result is irrelevant based on the reviewed labels. Precision@3=2/3 reveals the presence of this extra passage. Recall@3 and nDCG@3 both remain 1 because both relevant chunks appear earlier. At K=1, chunk recall is only 1/2, even though the returned chunk contains the deadline condition. For shutdown-sequence, both chunks are needed for a complete answer, so its K=1 result truly lacks required evidence.

All seven questions return a relevant chunk at rank one, and the relevant chunks are ideally placed within each result window. MRR and mean nDCG remain perfect. This run shows missing passages at K=1 and additional material at higher K, but no actual ordering failures. The controlled [0, 2, 2] test does not address this gap.

With only seven authored chunks, there’s limited opportunity to demonstrate ranking improvements. Add more representative user questions and naturally competing passages before comparing rerankers or making production decisions. Assess the required evidence before looking at rankings, and keep both failures and successes in a versioned dataset. Review counts by category and per-case regressions in addition to the overall mean.

What retrieval metrics cannot establish

The runner shown here covers only the seven answerable cases. Sample 07 runs unsupportedQuestions separately using the same retrieval settings. Both unsupported questions returned eligible chunks at every tested K. A policy that accepts any nonempty result list would thus accept both unsupported questions. Proximity in vector space does not prove answerability.

Exclude these outcomes from answerable-query averages, but record their question, K, result count, acceptance decision, and retrieval settings along with the run. This baseline does not calibrate a score threshold. When comparing thresholds, track both false acceptance of unsupported questions and rejection of answerable ones, using separate development and held-out sets. If there are no eligible documents, you’re testing filter logic, not semantic relevance. Always keep all necessary filters active.

Retrieval metrics evaluate ranked results, not the final context passed to the model. Even when retrieval finds both shutdown passages, deduplication or token limits during context assembly can remove required evidence.

Microsoft’s separate Microsoft.Extensions.AI.Evaluation.Quality package provides GroundednessEvaluator and RetrievalEvaluator. Both use a judge model. GroundednessEvaluator checks whether a generated answer is supported by the grounding context supplied to it. RetrievalEvaluator judges the relevance and ordering of the retrieved text for the question, but it cannot assess recall for relevant chunks it never receives.

The core Microsoft.Extensions.AI.Evaluation package provides CompositeEvaluator, which runs several IEvaluator implementations together. It can combine these built-in checks with our custom evaluator when their required inputs are available. The empty response in this setup is not enough to evaluate groundedness. The broader system evaluation process is covered in another article.

For reproducible comparisons, record the corpus, embedding model, query preprocessing, retrieval settings, and code revision. The measured baseline did not independently verify the exact embedding artifact or stored vectors, so its results should be treated as specific to this local experiment.

Use this evaluator to compare retrieval changes against the same set of labeled questions. Keep separate checks for the quality of the final context and generated answers.

Related reading

• Querying Qdrant from .NET
• Assembling Context for the Model
• What Should You Actually Evaluate?

Sources

• Microsoft Learn: Evaluation libraries and built-in evaluator catalog
• Microsoft Learn: Custom evaluators, disk-based reporting, and HTML reports
• Microsoft Learn: IEvaluator interface
• Microsoft Learn: NumericMetric result values
• Microsoft Learn: ScenarioRun persistence and disposal
• Microsoft Learn: Model-judged RetrievalEvaluator
• Microsoft Learn: Model-judged GroundednessEvaluator
• Microsoft Learn: CompositeEvaluator for combining evaluators
• Qdrant: Query API and similarity search
• Qdrant: Filtering during retrieval
• Stanford, Introduction to Information Retrieval: Evaluation of ranked retrieval results
• Ollama: Model-list API and artifact digests