Store a source checksum to skip unchanged documents

Do not process a document again just because the ingestion job saw it again.

Compute a checksum from the source bytes. If the checksum, pipeline version, and relevant metadata still match the last successful ingestion, skip the work.

using System.Linq;
using System.Security.Cryptography;
using System.Text.Json;

public sealed record IngestionState(
    string DocumentId,
    string SourceChecksum,
    string PipelineVersion,
    string MetadataFingerprint);

public static async Task<string> ComputeChecksumAsync(
    Stream source,
    CancellationToken cancellationToken)
{
    byte[] hash = await SHA256.HashDataAsync(
        source,
        cancellationToken);

    return Convert.ToHexString(hash);
}

Open a fresh stream for processing after hashing, or rewind the original stream only when it is seekable:

await using Stream checksumInput = await source.OpenReadAsync(
    cancellationToken);

string checksum = await ComputeChecksumAsync(
    checksumInput,
    cancellationToken);

IngestionState? previous = await states.GetAsync(
    source.Id,
    cancellationToken);

const string pipelineVersion = "plain-text-v1|chunks-v2|embeddings-v1";
DocumentMetadata metadata = source.Metadata;
string metadataFingerprint = ComputeMetadataFingerprint(metadata);

bool isUnchanged = previous is not null
    && StringComparer.Ordinal.Equals(
        previous.SourceChecksum,
        checksum)
    && StringComparer.Ordinal.Equals(
        previous.PipelineVersion,
        pipelineVersion)
    && StringComparer.Ordinal.Equals(
        previous.MetadataFingerprint,
        metadataFingerprint);

if (isUnchanged)
{
    return IngestionResult.Unchanged(source.Id);
}

await using Stream processingInput = await source.OpenReadAsync(
    cancellationToken);

await pipeline.ReplaceDocumentAsync(
    source.Id,
    processingInput,
    metadata,
    cancellationToken);

await states.SaveAsync(
    new IngestionState(
        source.Id,
        checksum,
        pipelineVersion,
        metadataFingerprint),
    cancellationToken);

ReplaceDocumentAsync must write the complete replacement set and remove or retire chunks that no longer belong to the document. Upserting the new chunks is not enough. A shorter document or a new chunking strategy can otherwise leave stale chunks in the index.

Save the ingestion state after the replacement finishes. Saving it earlier can make a partial run look complete. The index update and state update may not be atomic, so the replacement must also tolerate another attempt if the index changed but SaveAsync failed. Stable document and chunk IDs make that retry much easier to handle.

Define what unchanged means

The same bytes can still require a reindex. A change to text extraction, normalization, chunking, embedding configuration, or model selection can alter the indexed output.

Treat PipelineVersion as an application-owned compatibility contract. Change it whenever code, configuration, or model selection can change the output. A string such as plain-text-v1|chunks-v2|embeddings-v1 works only when the application owns each part and updates it deliberately.

Metadata needs its own fingerprint. A document can keep the same content while its tenant, permissions, expiry, or other filterable fields change. Use a fixed projection of the fields that affect indexed payloads or access. Sort set-like values such as permissions so their source order does not change the fingerprint:

public static string ComputeMetadataFingerprint(
    DocumentMetadata metadata)
{
    byte[] fingerprintInput = JsonSerializer.SerializeToUtf8Bytes(new
    {
        metadata.TenantId,
        Permissions = metadata.Permissions
            .OrderBy(permission => permission, StringComparer.Ordinal)
            .ToArray(),
        metadata.ExpiresAt
    });

    return Convert.ToHexString(SHA256.HashData(fingerprintInput));
}

Pass that same metadata value to the replacement operation. The skip decision and index write should use the same input. Do not let a matching content checksum suppress an authorization update.

There is no source to hash after a deletion. Detect removed documents through source reconciliation, change events, or tombstones, then delete or retire their chunks.

Hash at the boundary that saves useful work

I would start by hashing the raw source bytes. That lets the pipeline skip extraction, chunking, embedding, and index writes, although it still has to read or download the source. A trustworthy object version or ETag may avoid that transfer when the source system defines it as a stable content identity.

Raw-byte hashing sometimes reprocesses a file whose visible text did not change, such as a PDF with updated file metadata. That is usually the safer mistake. Hashing normalized extracted text avoids those runs, but extraction has already happened by then.

Use SHA-256 instead of a process-local hash code such as GetHashCode(). The checksum must stay deterministic across processes and deployments. It identifies source content for ingestion. It does not prove that an untrusted document is safe.

A checksum does not coordinate workers either. If two workers can ingest the same document at once, put a lease or conditional state transition around processing.

Use this check when ingestion rescans a source and usually finds no changes. If the source already sends only changed documents and provides a reliable content version, store that version instead of hashing every document again.