Treat provider throttling as a shared capacity signal

A 429 Too Many Requests from a model provider can tell you more than “this request failed”. When the response reflects a shared limit, every caller using that capacity pool needs to reduce pressure.

If the rejected call is the only one that backs off, the others keep sending traffic. Their independent retries pile onto the same capacity pool. That is how a short throttle window turns into a retry storm.

Put an application-owned capacity gate in front of provider calls. Scope the gate to the provider capacity pool behind the limit, rather than to one user or request. The boundary can stay small:

public readonly record struct ProviderCapacityKey(
    string Provider,
    string CapacityPool);

public interface IProviderCapacityGate
{
    ValueTask WaitAsync(
        ProviderCapacityKey key,
        CancellationToken cancellationToken);

    ValueTask ReportThrottleAsync(
        ProviderCapacityKey key,
        TimeSpan retryAfter);
}

The provider adapter decides what CapacityPool means. The rest of the application treats it as an opaque identifier.

For a process-local implementation, PartitionedRateLimiter<ProviderCapacityKey> can enforce admission per capacity key. The gate still needs separate cooldown state that the provider response can update because the limiter does not interpret throttle headers for you.

Before a provider call, wait until the gate allows work for its key. When the provider returns a genuine throttle response, report its retry or reset timing to the gate. Parse Retry-After or the provider-specific equivalent inside the adapter. The gate can then delay new work for that capacity pool or reject it when the product cannot afford to wait.

Keep the request cancellation token on WaitAsync, but do not pass it to ReportThrottleAsync. The request may already be cancelled when the response arrives. That should not prevent other callers from seeing the throttle. If a distributed implementation needs cancellation for a storage call, give it a short application-owned timeout inside the gate.

Provider SDKs expose throttling differently, and some retry internally. Publish one normalized capacity signal to the rest of the application instead of leaking those differences beyond the adapter.

Treat the provider’s retry or reset timing as the earliest sensible retry time. Do not release every waiting caller at that instant. Add jitter or gradually restore concurrency so the application does not send a second traffic spike as soon as the delay expires.

A process-local gate coordinates callers in one application instance. If several replicas use the same provider pool, use a distributed limiter or coordinate provider dispatch through a shared queue with bounded consumer concurrency. Otherwise, each replica still reacts alone.

Do not merge unrelated capacity pools. Derive the key from the scope of the limit that was actually reached. Depending on the provider, that scope might be an account, project, region, model, or deployment.

Inspect the error before updating the gate. A configured rate limit, temporary provider capacity pressure, and a caller-specific limit may need different responses.

A per-request retry policy can stop one operation from looping. It cannot stop 100 concurrent operations from each making three retries. That is the job of the shared gate.