Separate ingestion health from retrieval readiness

An ingestion failure does not automatically mean retrieval is down.

Imagine yesterday’s accepted index is still queryable and meets the product’s freshness requirement. Today’s ingestion run stops halfway through. That failure needs attention, but there is no reason to stop serving yesterday’s index yet.

Report the two states separately:

Retrieval readiness combines two checks. The query dependency must be available, and the version behind the serving name must still be eligible. Freshness is part of eligibility, not proof that the retrieval backend is available.

The application decides when an index is accepted. Its schema and query contract must match. For vector search, that contract includes the embedding profile. Ingestion must finish, the required validation must pass, and the version must be published for reads. A backend reporting that an index is ready does not prove that its contents are complete, correct, or current enough for the product.

When the backend supports versioned serving indirection, record both the name used by readers and the physical index that was accepted. Depending on the backend, the serving name might be an alias, a logical index name, or an application-owned read pointer.

public sealed record AcceptedIndex(
    string ServingName,
    string PhysicalIndex,
    string Version,
    DateTimeOffset SourceAsOf,
    TimeSpan MaximumAge);

public interface IAcceptedIndexCatalog
{
    Task<AcceptedIndex?> GetCurrentAsync(
        CancellationToken cancellationToken);
}

The readiness check inspects the serving mapping and sends a request through the production query path. It does not need the status of the latest ingestion run:

public sealed record RetrievalProbeResult(
    string ResolvedIndex,
    bool CanQueryThroughServingName);

public interface IRetrievalProbe
{
    Task<RetrievalProbeResult> ProbeAsync(
        string servingName,
        CancellationToken cancellationToken);
}

public sealed class RetrievalReadinessCheck(
    IAcceptedIndexCatalog indexes,
    IRetrievalProbe retrieval,
    TimeProvider timeProvider) : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        AcceptedIndex? index = await indexes.GetCurrentAsync(
            cancellationToken);

        if (index is null)
        {
            return HealthCheckResult.Unhealthy(
                "No accepted retrieval index is published.");
        }

        DateTimeOffset now = timeProvider.GetUtcNow();

        if (index.MaximumAge <= TimeSpan.Zero || index.SourceAsOf > now)
        {
            return HealthCheckResult.Unhealthy(
                "Accepted index metadata is invalid.");
        }

        TimeSpan age = now - index.SourceAsOf;

        if (age > index.MaximumAge)
        {
            return HealthCheckResult.Unhealthy(
                $"Accepted index '{index.Version}' is too old.");
        }

        RetrievalProbeResult probe = await retrieval.ProbeAsync(
            index.ServingName,
            cancellationToken);

        if (!StringComparer.Ordinal.Equals(
            probe.ResolvedIndex,
            index.PhysicalIndex))
        {
            return HealthCheckResult.Unhealthy(
                $"Serving name '{index.ServingName}' does not resolve to " +
                $"accepted index '{index.PhysicalIndex}'.");
        }

        return probe.CanQueryThroughServingName
            ? HealthCheckResult.Healthy(
                $"Accepted index '{index.Version}' is queryable.")
            : HealthCheckResult.Unhealthy(
                $"Accepted index '{index.Version}' cannot be queried.");
    }
}

IRetrievalProbe should resolve the production serving name, then run a cheap request through that same name. It returns the resolved physical index together with the query result. The readiness check compares that index with the accepted one. The probe should not call the language model or simulate a realistic user request.

Give the health check an explicit timeout:

builder.Services
    .AddHealthChecks()
    .AddCheck<RetrievalReadinessCheck>(
        name: "retrieval",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "retrieval" },
        timeout: TimeSpan.FromSeconds(2));

The capability tag lets each health endpoint decide whether to include retrieval. The probe must honor cancellation, and the retrieval client should have its own request timeout. Passing a CancellationToken without either behavior does not put a real bound on the call.

Some backends already have this kind of read pointer. With Qdrant, an alias can remain on the accepted collection while a replacement is built in the background. The application can switch the alias atomically after accepting the replacement.

That atomic switch does not include an application-owned catalog update. A crash can leave the catalog and serving pointer out of sync, so the publication workflow needs to recover or reconcile that state.

A health check is only a point-in-time observation. The serving mapping may change between resolution and the probe request. The check catches persistent disagreement between the catalog and serving pointer, but it does not make them transactional.

Ingestion still needs its own status and telemetry. I would start with:

• current run state and duration
• last successful completion and publication time
• failed document or batch count
• pending work and oldest pending age
• source checkpoint, source watermark, and candidate index version

Those signals can page the ingestion owner before the accepted index becomes too old. They should not make retrieval unavailable merely because the newest run is delayed or failed.

In this sample, SourceAsOf is the latest source time whose required changes are known to be present in the accepted physical index. It is more useful than the publication time, but it is still a simplification. A CDC pipeline may need offsets or log sequence numbers. Several sources or partitions may each need their own checkpoint.

The acceptable delay depends on the data. A documentation assistant may tolerate yesterday’s index. A security revocation, permission change, or time-sensitive operational feed may not. Once the accepted version falls outside that contract, it is no longer eligible to serve the use case even if the retrieval backend still answers queries.

Do not make the whole application unready when retrieval is optional. Expose retrieval as feature-level health or degrade that feature explicitly. Add the check to host readiness only when the instance cannot safely serve its intended traffic without retrieval.

Keep the rule simple: ingestion reports whether new data is becoming usable; retrieval reports whether already accepted data can still be served.