Table of Contents

The order of middleware affects what an AI application measures and repeats. Placing telemetry outside a cache also observes calls served from the cache. Placing it inside a retry component records every attempt, including failures that the caller does not observe.

A dashboard might show one successful call even though the provider processed two requests. The worked examples below demonstrate how this occurs, using a support-reply operation and its recorded call sequences.

This article focuses on how middleware order changes observable behavior, rather than implementing a complete middleware library. The examples use selected code excerpts from a controlled experiment to explain the resulting event traces.

Read the pipeline from outside in

An IChatClient pipeline consists of nested clients. Each wrapper can call its inner client, return early, or throw an exception. A retry wrapper might call its inner client multiple times.

ChatClientBuilder places the first registered wrapper at the outermost layer. The resulting call order looks like this:

support-reply service
  -> call probe
    -> response cache
      -> bounded retry
        -> attempt probe
          -> provider

Each wrapper can be provided as a factory function that takes the inner IChatClient and returns a decorated client. This registration matches the diagram above:

static IChatClient Compose(
    IChatClient provider,
    Func<IChatClient, IChatClient> observeCall,
    Func<IChatClient, IChatClient> cache,
    Func<IChatClient, IChatClient> retry,
    Func<IChatClient, IChatClient> observeAttempt) =>
    new ChatClientBuilder(provider)
        .Use(observeCall)
        .Use(cache)
        .Use(retry)
        .Use(observeAttempt)
        .Build();

The parameters are application-defined wrapper factories, not built-in middleware. The comparisons below show how their placement affects the events recorded by the probes.

A successful response passes back through the wrappers in reverse order. Exceptions unwind through the wrappers unless one handles the error. On a cache hit, the cache returns directly to the call probe and skips the retry component and attempt probe.

For buffered calls, probes record entry, success, failure, and cancellation. For streaming calls, probes record when enumeration starts and ends. An end event does not always mean the call succeeded.

These probes record events but do not export OpenTelemetry spans or measure durations. A probe named call observes a logical client invocation. A probe named attempt observes each invocation of the fake provider beneath it. These names reflect where the probes sit in the pipeline, not any special library behavior.

For details on the decorator implementation pattern, see Decorating IChatClient Without Tangling the Application. Here, the focus is on how wrappers interact with each other.

Define the behavior before comparing the order

The operation drafts a support reply for human review. It does not send a message or run any tools. Its fixed input asks where to find the public product manual and instructs the system to draft a reply rather than send it.

The following rules apply to these comparisons:

  • The cache accepts only this fixed, non-sensitive request and stores an expiry marker only if the completed response matches the expected text. Each entry expires one minute after being stored. Only a completed response can populate the cache. Streaming calls bypass the cache entirely.
  • The retry component permits up to two attempts. It waits 50 ms before the second attempt and only retries a designated transient failure. Other exceptions pass through without retry.
  • The outer probe records the outcome of the logical call. The inner probe records each provider attempt that reaches it.

These rules define the controlled example, not the defaults in Microsoft.Extensions.AI. I verified the event sequences with a scripted IChatClient running on .NET 10 and Microsoft.Extensions.AI 10.10.0. The fake can return a response, throw either a transient or non-transient exception, or wait for cancellation. The traces measure local call order and counts, not model latency or answer quality.

For this example, the cache key includes the fixed public request, fake model, prompt version, and text format. There is no tenant data or mutable conversation state. The one-minute expiration is used to make cache behavior explicit. It does not represent a recommended retention policy for customer messages.

The cache decorator checks cancellation and the supported request before looking up an entry. A hit returns immediately. A miss calls the inner client and stores an expiry only for the expected reply.

cancellationToken.ThrowIfCancellationRequested();
Fixture.Check(messages, options);
if (_expires.TryGetValue(Fixture.CacheKey, out DateTimeOffset expires) &&
    clock.GetUtcNow() < expires)
{
    events.Add("cache:hit");
    return Fixture.Response();
}
events.Add("cache:miss");
ChatResponse response = await base.GetResponseAsync(Fixture.Messages(), null, cancellationToken);
cancellationToken.ThrowIfCancellationRequested();
// Only this exact fixture result is cacheable. Return a fresh response on a hit.
if (response.Text == Fixture.Answer)
{
    _expires[Fixture.CacheKey] = clock.GetUtcNow().AddMinutes(1);
    events.Add("cache:store");
}
return response;

Here, _expires maps Fixture.CacheKey to its expiry. Fixture.Check accepts only the fixed text request with no options, and Fixture.Messages() reconstructs it. Because the reply is fixed, the cache stores an expiry marker rather than a mutable ChatResponse. Fixture.Response() constructs a fresh response containing that known text. The constructor supplies clock and events.

The experiment runs sequentially to keep the traces deterministic. The excerpts do not demonstrate concurrency handling or cache stampede protection.

Move telemetry across the cache boundary

With retries removed, a fresh cache makes the difference clear. Call each arrangement twice using the same request:

A: call probe -> cache -> provider
B: cache -> attempt probe -> provider

On the first call, both setups miss the cache and reach the provider. On the second call, both return the cached draft. However, the observations differ:

ArrangementFirst call: missSecond call: hitProvider calls across both
AOne recorded successOne recorded success1
BOne recorded successProbe does not run1

These are the exact event sequences. The provider counter is cumulative within an arrangement:

A miss: call:begin > cache:miss > provider:1 > cache:store > call:ok
A hit: call:begin > cache:hit > call:ok

B miss: cache:miss > attempt:begin > provider:1 > attempt:ok > cache:store
B hit: cache:hit

In A, the provider’s response passes back through the cache, which stores it before the call probe records success. In B, the inner probe records success before the cache stores the response.

If you time A, the measurement includes cache lookup and storage, plus provider work on a miss. If you time B, the measurement excludes cache work and does not cover hits. Neither placement by itself gives you a cache hit rate, so record cache outcomes explicitly.

If an inner probe reports fewer calls, that does not prove demand dropped. The cache might have served those calls.

Move telemetry across the retry boundary

For the retry comparison, each setup uses a new fake that fails once, then succeeds. The cache is removed:

C: call probe -> retry -> provider
D: retry -> attempt probe -> provider

The retry component allows up to two attempts with a 50 ms wait between them. It only retries the designated transient failure.

C recovered: call:begin > provider:1 > retry:wait > provider:2 > call:ok

D recovered: attempt:begin > provider:1 > attempt:fail > retry:wait
             > attempt:begin > provider:2 > attempt:ok

C records one successful logical call. Its elapsed time would include both attempts and the wait. D records one failed attempt and one successful attempt. The wait occurs outside both attempt measurements. Both setups make two provider calls.

With two transient failures, the exception propagates to the caller:

C exhausted: call:begin > provider:1 > retry:wait > provider:2 > call:fail

D exhausted: attempt:begin > provider:1 > attempt:fail > retry:wait
             > attempt:begin > provider:2 > attempt:fail

The first exception returns to the retry component, which handles it and retries. The second exception occurs on the final permitted attempt and propagates to the caller. The outer probe sees only the final exception, while the inner probe sees both failures.

ScriptProvider callsOuter probe (C)Inner probe (D)
Transient, success2One successOne failure, one success
Transient, transient2One failureTwo failures
Non-transient failure1One failureOne failure

Keep logical-call and provider-attempt metrics separate. Otherwise, a recovered request could hide provider failures or inflate the apparent application failure rate, depending on which layer provides the metric.

Give retries a shared deadline

A retry should use the time remaining in the operation. Creating a new timeout for each attempt quietly increases the total allowed time.

The application service can create a linked cancellation source before entering the pipeline. With an already composed client and the caller’s cancellation token, the call looks like this:

ChatMessage[] messages =
[
    new(ChatRole.System, "Draft a support reply for human review. Do not send it."),
    new(ChatRole.User, "Where can I find the public product manual?")
];

using var budget = CancellationTokenSource.CreateLinkedTokenSource(
    callerCancellationToken);
budget.CancelAfter(TimeSpan.FromSeconds(2));

ChatResponse response = await client.GetResponseAsync(
    messages,
    options: null,
    cancellationToken: budget.Token);

The two-second budget here is only an example, not a recommended value. The cancellation request covers cache work, attempts, and retry waits during the operation. Every layer must support cancellation, but this does not guarantee that remote generation or billing stops immediately.

The retry component must pass the same token into each attempt and into its backoff delay. It must check for cancellation before starting another attempt and let cancellation propagate without retry. Do not create a new deadline or call CancelAfter again inside the retry loop, as that can extend the operation’s total time budget.

After validating the fixed request, the retry decorator uses this loop. Fixture.Messages() reconstructs the supported request for each attempt. The same cancellation token reaches both the inner client and the delay.

for (int attempt = 1; ; attempt++)
{
    cancellationToken.ThrowIfCancellationRequested();
    try
    {
        // Fresh text messages for each attempt; inner clients may mutate arguments.
        return await base.GetResponseAsync(Fixture.Messages(), null, cancellationToken);
    }
    catch (TransientModelException) when (attempt < 2)
    {
        events.Add("retry:wait");
        await Task.Delay(TimeSpan.FromMilliseconds(50), cancellationToken);
    }
}

TransientModelException is the experiment’s failure type, not an exception supplied by Microsoft.Extensions.AI. events is the recorder passed to the decorator.

For retries, reconstruct the supported request from a known-safe source rather than reusing messages or options that an inner IChatClient might have mutated. A retry component needs an explicit replay contract for the content it supports. Materializing an enumerable does not clone its messages.

The fixed delay in this example keeps the scenario readable. In production, a policy needs provider-specific failure classification and backoff, as well as throttling guidance when available. Also check if the provider SDK implements its own retries. Two attempts at this layer can result in more than two network requests underneath an adapter.

When testing your own pipeline, verify cancellation during provider waits and during retry backoff. Allow the shared deadline to expire during the second attempt. It should stop that attempt without providing a new budget. For a cached path, ensure that a pre-cancelled operation does not return a cached response. These cases passed in the local verification for this example.

AI Systems Need Runtime Budgets discusses budgets across the rest of the request. Retries Are Not a Recovery Strategy covers replay safety and handling unknown outcomes.

Keep partial streams outside automatic retry

The buffered operation returns nothing until it receives a complete response. In streaming scenarios, the caller might have already consumed part of the text when a failure occurs.

Consider a stream that emits “You can find” and then fails transiently. In this example, both cache and retry wrappers pass streaming calls directly to their inner client. Streaming requests have no cache and no automatic retry, even before the first update. This is an explicit policy for this pipeline. A DelegatingChatClient subclass that only overrides GetResponseAsync leaves streaming on the base class’s pass-through path.

For streaming, create new messages and use the per-operation budget setup shown above. The following enumeration replaces the buffered call:

await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync(
    messages, cancellationToken: budget.Token))
{
    // This text may already be visible if the next MoveNextAsync fails.
    Console.Write(update.Text);
}

The local test records events from both the pipeline and the consumer:

call:stream-begin > attempt:stream-begin > provider:stream:1
  > consumer:partial > attempt:stream-end > call:stream-end

The streaming probe records enumeration lifetime. Its end event comes from a finally block:

public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
    IEnumerable<ChatMessage> messages, ChatOptions? options = null,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    events.Add($"{name}:stream-begin");
    try
    {
        await foreach (ChatResponseUpdate update in base.GetStreamingResponseAsync(
            messages, options, cancellationToken))
            yield return update;
    }
    finally
    {
        // Enumeration ended, failed, or was abandoned. This is not a success event.
        events.Add($"{name}:stream-end");
    }
}

events and name are constructor parameters for the probe. [EnumeratorCancellation] comes from System.Runtime.CompilerServices.

In the verified sequence, the caller receives the partial text once, then receives the exception. There is only one provider call.

Restarting generation after a failure could duplicate the prefix or continue with a different answer. The client needs an explicit interrupted-stream outcome and a product decision about whether to start again. Moving the retry wrapper cannot undo text already delivered to the caller.

The same restriction applies even more strongly to side effects. This experiment has no tools or writes. Do not wrap a tool loop in this retry component and assume that a failed model response means nothing happened.

Choose the observations you require

For this operation with fixed public input, I use the pipeline described at the start: call probe, cache, retry, attempt probe, provider.

The call probe observes every invocation that reaches the client. The attempt probe observes work sent to the fake provider. A cache hit results in one call success and zero provider attempts. A recovered miss results in one call success, one failed attempt, and one successful attempt.

Recovered miss:
call:begin > cache:miss > attempt:begin > provider:1 > attempt:fail
  > retry:wait > attempt:begin > provider:2 > attempt:ok
  > cache:store > call:ok

Next call:
call:begin > cache:hit > call:ok

The cache sits outside the retries, so each logical call performs one cache lookup. An expiry marker is stored only when the completed response matches Fixture.Answer. If all retries are exhausted, no cache entry is created, so the next request proceeds inward. After the one-minute expiration, the next call misses again and reaches the provider.

I keep both probes here. The outer probe runs once per logical call and describes the outcome the service awaited. The inner probe runs once per provider attempt and exposes work and failures hidden by retries. Give their metrics distinct names.

This choice does not imply that response caching is appropriate for customer support data. A shared cache requires a policy for tenant and data scope, permission changes, prompt and model versions, and retention. Authorization must run for every application operation, including those that receive a cache hit. Changing the middleware order does not provide these guarantees.

Keep prompt selection and authorization in the support-reply service. Let that service decide whether to expose an unavailable draft or reject an unacceptable answer. A successful provider call only means the client returned a response. It does not indicate whether a human should use the draft.

When to use this composition

Use a middleware pipeline when several operations share client mechanics and the order affects behavior you need to explain. Write down the outer-to-inner path, decide what each observation should record, and test cache hits, recovered failures, and exhausted attempts before connecting to a live provider. Test the streaming path separately if the feature exposes it.

Do not add this full pipeline to a feature that only needs a provider adapter and telemetry. A cache or retry component introduces policy that you will need to maintain. Only add one if the feature has a concrete requirement and a test that verifies the behavior.

Sources