Use HybridCache for expensive AI work in .NET

If five requests ask for the same expensive AI result at the same time, your app should not call the model five times.

HybridCache is a good fit for that. It gives you one API for in-process caching and optional distributed caching. The part I care about for AI workloads is stampede protection: for a given cache key, one caller runs the factory and the others wait for that result.

Add HybridCache

Install the package:

dotnet add package Microsoft.Extensions.Caching.Hybrid

Register it in DI:

builder.Services.AddHybridCache();

With this registration, HybridCache uses an in-process cache. A registered IDistributedCache, such as Redis, becomes secondary storage. Strings need no custom serializer; custom result types used with secondary storage must be serializable.

Cache an AI result

This example caches a document summary.

using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Hybrid;

public sealed class DocumentSummaryService(
    HybridCache cache,
    IChatClient chatClient)
{
    public async Task<string> SummarizeAsync(
        string documentText,
        CancellationToken cancellationToken)
    {
        const string promptVersion = "five-bullets-v1";
        const string modelKey = "configured-summary-model";
        const string optionsKey = "summary-defaults-v1";

        var normalizedText = NormalizeDocument(documentText);
        var key = $"summary:{promptVersion}:{modelKey}:{optionsKey}:{Sha256(normalizedText)}";

        var cacheOptions = new HybridCacheEntryOptions
        {
            Expiration = TimeSpan.FromHours(6),
            LocalCacheExpiration = TimeSpan.FromMinutes(30)
        };

        return await cache.GetOrCreateAsync(
            key,
            async token =>
            {
                var response = await chatClient.GetResponseAsync(
                    $"""
                    Summarize this document in five bullet points.

                    {normalizedText}
                    """,
                    cancellationToken: token);

                if (string.IsNullOrWhiteSpace(response.Text))
                {
                    throw new InvalidOperationException("Model returned an empty summary.");
                }

                return response.Text;
            },
            options: cacheOptions,
            cancellationToken: cancellationToken);
    }

    private static string NormalizeDocument(string value)
    {
        return value
            .Replace("\r\n", "\n", StringComparison.Ordinal)
            .Replace('\r', '\n')
            .Trim();
    }

    private static string Sha256(string value)
    {
        var bytes = SHA256.HashData(Encoding.UTF8.GetBytes(value));
        return Convert.ToHexString(bytes);
    }
}

The first request for a key calls the model. Later requests can reuse the cached value. Concurrent misses for the same key share the factory call. HybridCache also combines their cancellation, so the model call must be safe to cancel if all waiting callers stop.

The guard above is the minimum validation needed to avoid caching an empty result. Throwing before the factory returns means no cache entry is created. It is not complete AI-output validation. Production code should also reject refusals, truncated output, malformed structure, or any response that fails workload-specific rules.

Be strict with cache keys

The key must cover every input that can change the answer: prompt version, configured model or deployment, relevant generation options, and a hash of normalized input. If options live in the IChatClient pipeline instead of an explicit ChatOptions object, version that pipeline and use its version in optionsKey.

The sample normalizes line endings and outer whitespace before hashing. Define a different rule only when it matches what “the same document” means for your workload. The six-hour entry lifetime and 30-minute local lifetime are example policy values, not defaults.

Do not use raw user text as the key. Raw prompts can be long, sensitive, and awkward to handle as keys.

Also avoid caching results that should be fresh, personalized, permission-sensitive, or dependent on hidden context. Caching works best when the inputs are explicit and you are fine reusing the first answer for that key.

When this is useful

I would reach for HybridCache for:

• document summaries
• extracted metadata
• embeddings
• deterministic classification
• expensive retrieval results
• tool manifests or routing decisions

Keep payload size in mind. Summaries and small metadata are usually fine, but large extraction results or embedding arrays may exceed MaximumPayloadBytes. Values over that configured limit are logged and not cached.

I would not wrap every chat call in a cache. Open-ended chat responses are usually user-specific and context-specific. Cache the expensive stable pieces, not the whole conversation by default.