Payload Modeling and Metadata Filters in Qdrant with .NET

A draft runbook can have the same shutdown procedure as the published version. Vector similarity alone cannot distinguish their publication states. If your application should only return published runbooks, require that state in the retrieval filter.

In Chunking for Qdrant in .NET Before It Becomes a Quality Problem, we split a support runbook into overlapping paragraphs and queried its chunks. We will now add sources to that console application and use document metadata to control which documents retrieval can return. In this example, the filter includes the published runbook and excludes the draft and note. The chunker and embedding setup do not change.

A complete, standalone version of this example is available in the sample repository.

Start with the chunking sample

Use QdrantCollections from the chunking article. This includes SourceSection, SourceChunk, and Chunker.Build, along with provider configuration, collection validation, an upsert loop, and a vector query. If you are starting here, refer to the indexing article for credentials and embedding configuration, and the chunking article provides the initial Program.cs.

Continue using .NET 10 and the existing packages:

Both assembled provider variants compile on .NET 10.0.102. OllamaSharp reports CS9057 for its optional source generator on this SDK, but this sample does not use generated tools.

Use the same local Qdrant 1.19.0 instance and collection for your provider. Azure OpenAI uses support-documents-v1 with the support-text-v1 profile and 1,536-dimensional cosine vectors. EmbeddingGemma uses support-documents-embeddinggemma-v1, support-embeddinggemma-v1, and 768 dimensions.

Keep the provider setup block, including prepareChunk and prepareQuery. Also keep chunkingVersion, previewOnly, and the original documentId, title, and sections declarations. The following changes begin right after the sections array. Keep the original helper types at the end of the file.

The word budget is still 80, with one paragraph of overlap. The original runbook’s embedding inputs remain unchanged. We also keep the same chunking policy and embedding profile for the additional documents.

Separate identity, evidence, and eligibility

The existing points already include useful payload. Give each field a clear purpose before adding more:

Use "runbook" and "note" for document type, and "published" and "draft" for publication state. These are scalar string fields with controlled values. The console sample sets them directly. In an application, the source adapter should read them from the document system. A model should not infer them from the snippet, and the query text should not determine whether a draft is eligible.

Stop at these fields unless a query requires more. Each value must be supplied and its meaning kept consistent across all ingestion paths.

Add this record after the existing helper types:

public sealed record SourceDocument(
    string DocumentId, string Title, string DocumentType,
    string PublicationState, SourceSection[] Sections)
{
    public void Validate()
    {
        if (string.IsNullOrWhiteSpace(DocumentId) ||
            string.IsNullOrWhiteSpace(Title) ||
            DocumentType is not ("runbook" or "note") ||
            PublicationState is not ("published" or "draft"))
        {
            throw new ArgumentException("Invalid source document metadata.");
        }
    }
}

This record describes the input for this sample. Qdrant can store other payload shapes. A keyword index does not enforce this record or validate allowed publication states.

Include document metadata in every chunk

Keep the existing SourceSection[] sections array unchanged. Immediately below it, replace everything from List<SourceChunk> chunks = Chunker.Build(...) through the closing brace of if (previewOnly) with the following block. This introduces a new documents array that reuses the existing sections.

SourceDocument[] documents =
[
    new(documentId, title, "runbook", "published", sections),
    new("worker-shutdown-draft", title, "runbook", "draft", sections),
    new("queue-capacity-note", "Queue capacity", "note", "published",
        [sections.Single(section => section.SectionId == "capacity")])
];

var prepared = new List<(SourceDocument Document, List<SourceChunk> Chunks)>();
var documentIds = new HashSet<string>(StringComparer.Ordinal);

foreach (SourceDocument document in documents)
{
    document.Validate();
    if (!documentIds.Add(document.DocumentId))
    {
        throw new ArgumentException("Document IDs must be unique.");
    }

    List<SourceChunk> chunks = Chunker.Build(
        document.DocumentId, document.Sections, chunkingVersion,
        maxWords: 80, overlap: true);
    prepared.Add((document, chunks));

    foreach (SourceChunk chunk in chunks)
    {
        Console.WriteLine(
            $"{document.DocumentId}/{chunk.ChunkId} " +
            $"[{document.DocumentType}, {document.PublicationState}]");
        Console.WriteLine(chunk.Text);
        Console.WriteLine();
    }
}

if (previewOnly)
{
    return;
}

The original worker-shutdown runbook stays published. Its draft copy deliberately uses the same title and sections. The note reuses the queue-capacity section. These fixtures let us test each filter with familiar text. They do not represent how a document transitions from draft to publication.

Run dotnet run with previewOnly = true. The original policy produces three chunks for each runbook and one for the note. Both runbooks have shutdown/p0, shutdown/p1, and capacity/p0. Their point IDs are different because Chunker.Build includes the document ID in its identity calculation.

The existing timeout, Qdrant client, and collection-contract validation continue unchanged. Preview mode still exits before opening the client or reading Azure credentials.

Index the fields used by filters

Insert this block after the collection-contract validation and before the line that calls createGenerator():

string[] filterFields =
[
    "document_id", "chunking_version", "document_type", "publication_state"
];

foreach (string field in filterFields)
{
    if (collection.PayloadSchema.TryGetValue(field, out PayloadSchemaInfo? schema))
    {
        if (schema.DataType != PayloadSchemaType.Keyword)
        {
            throw new InvalidOperationException(
                $"Payload index '{field}' must use Keyword.");
        }

        continue;
    }

    await client.CreatePayloadIndexAsync(
        collectionName: collectionName,
        fieldName: field,
        schemaType: PayloadSchemaType.Keyword,
        wait: true,
        cancellationToken: token);
}

These queries compare exact categorical values, so the fields use keyword indexes. "published" and "Published" are different values. So choose one spelling at ingestion. Indexing is not required for heading and text fields when only returning them for inspection.

The previous steps already wrote points into this collection. The block adds missing indexes and rejects any existing index with an incompatible data type. This assumes a single setup process. Concurrent schema management requires coordination.

For a new collection, create payload indexes before ingestion. Qdrant uses them for filtering and can build additional filter-aware edges when constructing the HNSW index. Adding a payload index later will not rebuild those edges in an existing HNSW graph. The small corpus here does not provide any meaningful data about search performance.

Strict mode can also reject filtering on unindexed fields. A query that works with permissive local defaults may fail with a stricter collection. Always create indexes as part of setup.

Extend the existing upsert and read-back loop

Keep using IEmbeddingGenerator<string, Embedding<float>> generator = createGenerator();. Replace the following foreach (SourceChunk chunk in chunks) ingestion loop with:

foreach (var (document, chunks) in prepared)
{
    foreach (SourceChunk chunk in chunks)
    {
        ReadOnlyMemory<float> vector = await generator.GenerateVectorAsync(
            prepareChunk(document.Title, chunk.Text), cancellationToken: token);
        if (vector.Length != vectorSize)
        {
            throw new InvalidOperationException("Embedding dimension mismatch.");
        }

        await client.UpsertAsync(
            collectionName: collectionName,
            points: [new PointStruct
            {
                Id = chunk.PointId,
                Vectors = vector.ToArray(),
                Payload =
                {
                    ["document_id"] = document.DocumentId,
                    ["chunk_id"] = chunk.ChunkId,
                    ["chunking_version"] = chunkingVersion,
                    ["section_id"] = chunk.SectionId,
                    ["title"] = document.Title,
                    ["heading"] = chunk.Heading,
                    ["text"] = chunk.Text,
                    ["document_type"] = document.DocumentType,
                    ["publication_state"] = document.PublicationState
                }
            }],
            wait: true,
            cancellationToken: token);

        var stored = await client.RetrieveAsync(
            collectionName: collectionName,
            id: chunk.PointId,
            withPayload: true,
            withVectors: false,
            cancellationToken: token);
        if (stored.Count != 1 ||
            PayloadFields.RequiredString(stored[0].Payload, "document_id")
                != document.DocumentId ||
            PayloadFields.RequiredString(stored[0].Payload, "chunking_version")
                != chunkingVersion ||
            PayloadFields.RequiredString(stored[0].Payload, "text") != chunk.Text ||
            PayloadFields.RequiredString(stored[0].Payload, "document_type")
                != document.DocumentType ||
            PayloadFields.RequiredString(stored[0].Payload, "publication_state")
                != document.PublicationState)
        {
            throw new InvalidOperationException(
                $"Read-back failed: {document.DocumentId}/{chunk.ChunkId}");
        }
    }
}

Append this helper after the type declarations at the end of the file:

public static class PayloadFields
{
    public static string RequiredString(
        IReadOnlyDictionary<string, Value> payload, string field)
    {
        if (!payload.TryGetValue(field, out Value? value) ||
            value.KindCase != Value.KindOneofCase.StringValue ||
            string.IsNullOrWhiteSpace(value.StringValue))
        {
            throw new InvalidOperationException(
                $"Payload field '{field}' must be a nonempty scalar string.");
        }

        return value.StringValue;
    }
}

RequiredString checks the payload type before accessing StringValue. Another ingestion path could write an array or a number into the same field. Qdrant can match a keyword against an element in an array, so a successful filter alone does not prove that the stored value is the scalar string this application expects.

Set previewOnly = false and run the program once the remaining query changes are in place.

The two-minute timeout now covers seven document embedding requests and one query embedding request, along with the Qdrant operations. The earlier sample embedded three chunks. Increase the deadline if you encounter slow local inference or cold model loading.

The original chunks keep their point IDs, so upserts replace those points with payload that includes the new fields. The draft copy and note receive their own points.

This small sample generates embeddings on every run. If you need to change only metadata, you can update the payload without regenerating an unchanged vector. In production, make sure metadata stays consistent across all affected chunks and handle partial failures. The per-point loop does not provide an atomic document publication change.

A filter evaluates the metadata currently stored in Qdrant. If the source system withdraws a runbook but its payload update fails, chunks still marked as published remain eligible. Make sure publication changes are synchronized and retry or reconcile failed updates. Query conditions alone do not guarantee the indexed state is current.

Put eligibility inside the vector query

Keep the existing question, query embedding, and query-dimension check. Replace the block starting at var results = await client.QueryAsync(...) and continuing through the result-printing loop with:

const string requiredDocumentType = "runbook";
const string requiredPublicationState = "published";

var eligibility = new Filter
{
    Must =
    {
        MatchKeyword("chunking_version", chunkingVersion),
        MatchKeyword("document_type", requiredDocumentType),
        MatchKeyword("publication_state", requiredPublicationState)
    }
};

var results = await client.QueryAsync(
    collectionName: collectionName,
    query: queryVector.ToArray(),
    filter: eligibility,
    limit: 3,
    payloadSelector: new[]
    {
        "document_id", "chunk_id", "chunking_version",
        "document_type", "publication_state", "heading", "text"
    },
    vectorsSelector: false,
    cancellationToken: token);

Console.WriteLine($"Question: {question}");
if (results.Count == 0)
{
    Console.WriteLine("No chunks matched the required retrieval conditions.");
}

foreach (ScoredPoint result in results)
{
    string source = PayloadFields.RequiredString(result.Payload, "document_id");
    string chunk = PayloadFields.RequiredString(result.Payload, "chunk_id");
    string version = PayloadFields.RequiredString(result.Payload, "chunking_version");
    string type = PayloadFields.RequiredString(result.Payload, "document_type");
    string state = PayloadFields.RequiredString(result.Payload, "publication_state");
    if (version != chunkingVersion ||
        type != requiredDocumentType || state != requiredPublicationState)
    {
        throw new InvalidOperationException("Unexpected retrieval eligibility.");
    }

    Console.WriteLine($"{source}/{chunk} [{type}, {state}]: {result.Score:F4}");
    Console.WriteLine(PayloadFields.RequiredString(result.Payload, "heading"));
    Console.WriteLine(PayloadFields.RequiredString(result.Payload, "text"));
    Console.WriteLine();
}

Every condition in Must must match. MatchKeyword tests for equality with a single categorical string, while the previous sample used Match with a one-element array. Both approaches support this filter, but the single-value helper makes the intention clearer.

Qdrant applies the eligibility conditions during search and ranks the returned candidates by vector similarity. Depending on the search path, candidates may be approximate nearest neighbors rather than the exact highest-scoring eligible points. A higher score does not make a draft eligible. The separation between filtering and ranking gives each part of the query a clear purpose.

The previous query also required document_id = worker-shutdown. Removing that restriction allows the query to consider any published runbook with the selected chunking version. In this fixture, only the original runbook qualifies. When a caller requests a specific document, add its ID to eligibility.Must before calling QueryAsync.

Do not retrieve an unrestricted top three and then filter out drafts in C#. This could leave you with fewer results, even if eligible passages exist further down in the unrestricted ranking. Pass the mandatory conditions to Qdrant in the query.

Check exclusions before interpreting scores

Run the original question: Should I drain all queued work during shutdown? Use the same provider, chunking policy, and result limit while comparing the filters.

With this controlled corpus, only the original runbook’s three chunks are eligible for the published-runbook query. Inspect the returned text for the shutdown deadline condition. Exact scores and ordering depend on the embedding model. Changing the metadata conditions lets you inspect the expected eligibility outcomes while keeping the same question and chunking version:

To inspect the draft fixture, set requiredPublicationState to draft. For the note, keep published and set requiredDocumentType to note. The same constants control the filter and result-loop checks. Run these as separate cases, then restore runbook and published.

Use the table for manual checks, as the sample does not automate these scenarios. The result-loop guard checks the metadata on returned points, but it cannot confirm that every exclusion case was exercised.

For a deliberate empty-result case, keep the published-runbook conditions and insert this before QueryAsync:

eligibility.Must.Add(MatchKeyword("document_id", "worker-shutdown-draft"));

That document is a draft, so it cannot satisfy the publication-state condition. The program should report that no chunks matched. Remove the extra document condition before returning to the main corpus query.

Do not silently relax required conditions after an empty result. The application may report that no eligible source was found, prompt the user to change an optional scope, or use another explicitly allowed path. An empty result does not mean the entire corpus lacks relevant information.

The previous whole-document point has no matching chunking_version, so it remains excluded. Legacy points missing the new fields also fail the required positive matches. Adding an index does not populate those values. The sample rewrites the original chunks. Other sources need their own backfill before these queries can include them. Chunks missing the publication state remain ineligible until that work is complete.

When to use metadata filters

Apply metadata filters to enforce source requirements like publication status, document type, or a specific document ID. Specify these filters in your search request to limit results to eligible documents.

Let vector similarity select relevant passages within the required retrieval scope. Use hard metadata filters for explicit source requirements, but avoid restricting results based on descriptive labels unless the application needs that restriction.

Publication state is not an authorization model. If retrieval depends on tenant membership or user permissions, derive that scope from trusted application context and enforce it on every relevant path. This console sample implements neither ACLs nor tenant isolation.

Finally, review the metadata and document IDs in your search results to verify that all returned passages meet your requirements. Context assembly still needs to account for the repeated deadline paragraph created by overlap and keep citations attached to the selected text.

Related reading

• Chunking for Qdrant in .NET Before It Becomes a Quality Problem
• Querying Qdrant from .NET
• Keep vector search filters separate from semantic ranking

Sources

• Qdrant: Payload types and payload updates
• Qdrant: Filtering clauses and keyword matching
• Qdrant: Vector search and approximate nearest neighbors
• Qdrant: Payload indexes and filter-aware HNSW
• Qdrant: Strict mode and collection configuration
• Qdrant: Official .NET client