Table of Contents

A five-second model timeout does not give you a five-second AI feature.

Retrieval may take one second. The first model call takes four. A tool takes another two, and the next model call gets its own five-second timeout. Every dependency stayed within its local limit, yet the request took 12 seconds and paid for two model calls.

The complete use case needs one runtime budget. Its remaining allowance follows every model call, tool invocation, and retry.

Timeouts still matter. They are just one line in the budget.

Budget the execution, not each dependency

Consider a support-answer workflow:

load the ticket
    -> retrieve policy documents
    -> ask the model
    -> call a customer-history tool
    -> ask the model again
    -> validate the answer

Each dependency can have a sensible timeout while the workflow still runs too long. The same applies to tokens and cost. Three calls with a 1,000-token output cap can generate 3,000 tokens. A retry repeats input, and tool results enlarge the next prompt.

A runtime budget gives the workflow one shared envelope:

DimensionWhat it boundsWhat happens when it is exhausted
Wall-clock timeTotal useful lifetime of the executionRequest cancellation and return a deadline result
Input tokensContext sent across all model callsReduce context before the call or stop
Output tokensGenerated tokens across all model callsLower the next call’s cap or stop
Model callsInitial calls, follow-ups, repairs, and retriesDo not start another call
Tool callsAutomatic and application-directed invocationsStop the loop before another tool runs
RetriesRepeated attempts after transient failuresReturn the chosen failure or recovery outcome
Estimated costAccumulated cost of metered workStop or use a fallback whose cost already fits

These limits belong together because the dimensions interact. A retry can spend more time, tokens, and money. A tool call spends time and may make the next prompt larger. A longer output consumes more of the deadline.

A budget will stop some work that might have succeeded. Good. A late or overpriced result is not a successful execution for that product.

Define one execution policy

Keep the configured limits immutable. The live ledger belongs to one execution, which may or may not be an HTTP request.

public sealed record RuntimeBudget(
    TimeSpan MaxDuration,
    TimeSpan MaxModelAttemptDuration,
    TimeSpan MinUsefulModelAttemptDuration,
    TimeSpan CompletionHeadroom,
    int MaxModelCalls,
    int MaxToolCalls,
    int MaxRetries,
    long MaxInputTokens,
    long MaxOutputTokens,
    decimal MaxEstimatedCostUsd);

Validate the policy when configuration is loaded. The three attempt and execution durations must be positive. Headroom, counters, token limits, and cost limits cannot be negative. Require a positive output cap. The time limits must satisfy both invariants:

MinUsefulModelAttemptDuration <= MaxModelAttemptDuration
MinUsefulModelAttemptDuration + CompletionHeadroom <= MaxDuration

Apply the same checks to reservations. Negative estimates must never create allowance. Use checked arithmetic or subtraction helpers to avoid overflow.

Keep counters private and protect every compound read or mutation with the same lock. Return immutable snapshots created under that lock. Making one reservation method atomic does not make unrelated public properties thread-safe.

Start the deadline and ledger together

I use four separate cancellation sources:

  • caller cancellation
  • host shutdown
  • the execution deadline
  • a per-attempt timeout

The execution scope owns its deadline. Both elapsed-time accounting and cancellation use the same TimeProvider and start when the scope is created:

long startedAt = timeProvider.GetTimestamp();
using var deadlineCts = new CancellationTokenSource(
    limits.MaxDuration,
    timeProvider);

TimeSpan RemainingTime()
{
    TimeSpan remaining = limits.MaxDuration
        - timeProvider.GetElapsedTime(startedAt);

    return remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero;
}

Using one clock keeps tests honest when they use FakeTimeProvider. A system timer should not expire independently from the fake clock used by the ledger.

The per-attempt timeout must fit inside the time still available. If an execution has 1.4 seconds left, starting a model call with its normal five-second timeout is dishonest. Reserve completion headroom first, then calculate:

attempt timeout = min(
    configured attempt timeout,
    remaining execution time - completion headroom)

Reject the call when the result is below MinUsefulModelAttemptDuration. A one-millisecond attempt is traffic, not a serious chance of success.

For each dependency call, create a time-provider-aware attempt source and link the sources only for delivery to that dependency:

using var attemptCts = new CancellationTokenSource(
    reservation.AttemptTimeout,
    timeProvider);

using var callCts = CancellationTokenSource.CreateLinkedTokenSource(
    callerToken,
    hostStoppingToken,
    deadlineCts.Token,
    attemptCts.Token);

Keep the original tokens. The linked token asks observing work to stop, but it does not record which source won. In this example, classification precedence is host shutdown, caller cancellation, execution deadline, then attempt timeout. Pick an order deliberately and use it everywhere.

Cancellation is cooperative. It does not prove that local work stopped immediately, or that a remote provider stopped processing and billing.

Reserve before handoff and reconcile afterward

Post-call accounting is too late. The workflow has already spent the money and time.

A model reservation needs a stable ID, estimated input tokens, an output cap, estimated cost, and an attempt timeout. Create it under the ledger lock only when every required dimension has enough allowance.

The reservation lifecycle matters:

Reserved -> Started -> Reconciled
  • Release Reserved work only when the application never handed it to the inner client or tool.
  • Reconcile Started work when required usage fields are available.
  • When a started call returns uncertain or partial usage, charge the unresolved fields against the reservation or stop with an unknown-usage outcome.

At this abstraction level, Started means control was handed to the next layer. Local validation, serialization, or option conversion may still fail before a network request. Precise dispatch tracking requires instrumentation inside the provider or tool adapter. Without it, charge pessimistically after handoff. A caller timeout does not prove that the provider consumed nothing.

The reservation operation should validate its arguments and calculate the attempt timeout inside the same lock that reserves calls, tokens, and cost. The core calculation is:

available attempt time = remaining execution time - completion headroom
attempt timeout = min(configured attempt timeout, available attempt time)
output allowance = limit - committed output - reserved output

Reject the reservation when the attempt timeout is below the minimum or the output allowance is zero. Subtract validated non-negative values and return zero once committed or reserved usage reaches the limit. Use the same approach for input and cost.

The reservation supplies the per-call ChatOptions.MaxOutputTokens. That option limits one model request when the underlying client honors it. It is not a cumulative workflow limit.

Put enforcement inside automatic tool loops

FunctionInvokingChatClient.GetResponseAsync(...) can make several requests to its inner client. It sends tool results back to that inner client and continues until the tool loop ends. The final response aggregates available usage from those turns.

One reservation around the outer call is wrong. It counts one model call even when the wrapper makes three, and its MaxOutputTokens value can be applied to each inner request. It also sees none of the locally invoked tools.

Place model-call enforcement inside the function-invocation wrapper. Put any retry middleware you control outside that budgeting boundary so each attempt must reserve again:

application
    -> FunctionInvokingChatClient
        -> application retry layer
            -> BudgetingChatClient
                -> provider SDK

There is a lifetime trap here. AddChatClient registers its pipeline as a singleton by default. A budgeting adapter that captures one execution’s ledger or cancellation sources must not live in that singleton pipeline. Create the budgeting layer and its FunctionInvokingChatClient for each execution, or keep the adapter stateless and pass the current ledger through explicit per-call context.

The following fragment is illustrative pseudocode. Types such as PromptEstimate, ModelReservation, and the ledger are application contracts, not a companion library. callCts.Token is the linked per-attempt token from the previous section.

List<ChatMessage> messageList = [.. messages];
ChatOptions callOptions = options?.Clone() ?? new ChatOptions();

PromptEstimate estimate = promptEstimator.Estimate(
    messageList,
    callOptions,
    modelProfile);

ModelReservation reservation = budget.ReserveModelCall(
    estimate,
    callOptions.MaxOutputTokens);

callOptions.MaxOutputTokens = reservation.OutputTokenCap;

budget.MarkStarted(reservation); // Controlled handoff, not confirmed dispatch.

ChatResponse response = await innerClient.GetResponseAsync(
    messageList,
    callOptions,
    callCts.Token);

budget.ReconcileModel(reservation, response.Usage);

Materializing messages once avoids enumerating an arbitrary IEnumerable<ChatMessage> twice. The estimator needs more than message text. Depending on the provider, the effective prompt can include ChatOptions.Instructions, tool declarations and JSON schemas, structured-output schemas, multimodal content, and provider-specific framing. Estimate is the honest name because local preflight cannot always produce an exact count.

Stateful clients make that limit clearer. If ConversationId refers to history stored by the provider, the application may not possess the complete context. Reserve conservatively or disable local claims of exact input accounting for that path.

After the controlled handoff, charge the reservation on an exception unless reliable usage data allows reconciliation. The fragment omits that branch and the streaming override. A production adapter needs both.

Guard locally executed tools through FunctionInvokingChatClient.FunctionInvoker. Validate known preconditions before reserving, then mark the reservation Started immediately before context.Function.InvokeAsync(context.Arguments, cancellationToken).

On success or a known failure, reconcile from the request, response metadata, usage record, and failure details available to that tool adapter. If those sources cannot resolve the cost after handoff, charge the reservation. This is deliberately pessimistic. Parallel tool calls need atomic reservations before any task starts.

Hard tool-call limits apply only to execution you control. Provider-managed or server-side tools can be opaque. In that case, enforce the controls the provider exposes, account for reported usage, and avoid claiming that the application counted every tool call.

An application-owned tool loop is the other valid design. It is more code, but it makes every model and tool boundary visible without relying on wrapper placement.

Reconcile usage field by field

ChatResponse.Usage being non-null does not mean every count is present. InputTokenCount, OutputTokenCount, cached input, and reasoning counts are nullable. Reconcile each field required by your policy. For a missing required field, charge its reservation or stop with unknown usage.

Per-call interception avoids another ambiguity. FunctionInvokingChatClient adds the non-null usage objects returned by its inner calls. If one turn omits usage and another reports it, the outer response can contain a plausible but incomplete total.

Do not add cached input tokens to InputTokenCount again. Do not add reasoning tokens to OutputTokenCount again. Those detail counts classify pricing; the documented totals already include them.

Cost estimation needs a reviewed pricing key, not only a model ID. Depending on the provider, it may include the provider, deployment or model snapshot, region, service tier, modality, batch mode, and cached-token rules.

An application-estimated cost budget is not a provider billing ceiling. Rounding, missing usage, and price changes can make the invoice differ. Use provider-side quotas and billing controls separately, and know whether they alert or block spend.

The total cost dimension must include every paid boundary the article claims to budget. Retrieval services, rerankers, paid tools, and external APIs need their own reservations. If the ledger only accounts for model calls, name the limit MaxEstimatedModelCostUsd instead.

Retries spend the original execution budget

A retry does not receive a fresh deadline or token allowance. Do not start one that cannot finish inside the remaining time or would exceed another dimension.

Put application retry middleware outside BudgetingChatClient, as shown above. Each retry attempt then crosses the budgeting boundary and needs a new reservation.

Before repeating an attempt, the retry layer must atomically reserve one retry slot. The inner budgeting client separately reserves the repeated model or dependency call.

Retries inside the provider SDK are different. Configure or disable them when possible. If the SDK retries internally without an attempt callback, the application cannot claim an exact retry or model-call count. Treat that layer as opaque and reserve conservatively.

A cost-exhausted execution may use a fallback only when that fallback is free, has separately reserved capacity, or still fits inside the remaining allowance.

Budget exhaustion is an application result

OperationCanceledException alone does not explain what happened. It might mean the user disconnected, the host stopped, a dependency attempt timed out, or the complete runtime deadline expired.

Select a stop reason from the original source tokens, not from the linked token. Apply the same precedence at every boundary. This is a policy decision based on the tokens observed at inspection time; it does not prove which token caused the cancellation first.

For example, a SelectStopReason helper can check host shutdown first, then caller cancellation, the execution deadline, and finally the attempt timer. Map those to unavailable, cancelled, time-budget exhaustion, and dependency timeout results. Ledger failures should carry the exhausted dimension directly rather than masquerading as cancellation.

The user-facing message need not mention tokens or internal prices. Keep the reason in the application result for telemetry, support, and fallback decisions.

A partial answer needs its own policy. Some features can label and return one. Others should return nothing because an incomplete answer would be misleading.

Observe the allowance and the spend

Record both sides of the ledger. A trace that says model.duration = 3.2s does not say whether 3.2 seconds was healthy for this feature.

For one execution, I would record:

  • the budget policy or a stable policy version
  • elapsed and remaining time at each boundary
  • model calls, tool calls, and retries used
  • reported input and output tokens
  • estimated cost and the price configuration version
  • the exhausted dimension, when execution stops

Do not copy prompts, retrieved documents, or tool results into telemetry just to explain the numbers. Operational metadata usually answers the budget question without duplicating sensitive content.

Per-execution budgets stop one workflow from running away. They do nothing about ten thousand executions that each stay inside budget. Rate limits, concurrency limits, tenant quotas, and dependency-level retry budgets protect that shared capacity.

Choose numbers from the product promise

There is no universal runtime budget. A chat interaction and an overnight evaluation run have different definitions of “too late.” Start with how long the caller will wait, what one result may cost, which tools it needs, and whether a fallback still has value.

Test the complete path under normal load and injected failure. Measure tail latency as well as averages. Include prompt growth, tool output, throttling, and retry delays. Test the effects of opaque SDK retries even when the SDK does not expose an exact attempt count.

Leave headroom between the internal deadline and the external timeout. The application still needs to map the result, write telemetry, and respond. If both expire together, the caller may receive a connection failure instead of the intended outcome.

Treat the first values as a hypothesis. Production traces and cost data will show where the budget is too loose, too strict, or spent on the wrong step.

When runtime budgets are useful

Use an execution-scoped budget when a workflow can make more than one metered or remote call, invoke tools, retry, or fan out. It gives the orchestration layer one answer to the question: may this execution start more work?

For interactive work, that execution normally observes caller cancellation. Deliberate background work needs an independent budget instead of silently outliving the request. Durable workflows also need persisted reservations and idempotent reconciliation so a restart cannot reset or double-charge the ledger.

When a full budget object is unnecessary

A local spike or one low-risk model call may only need cancellation, a timeout, and an output cap. Add a shared ledger when the workflow can multiply time or spend, not when the object would add ceremony without changing a decision.

Practical takeaway

Pick one AI use case and write down the maximum wall-clock time, cumulative tokens, model calls, tool calls, retries, and estimated cost for one execution.

Put those limits in one policy. Create one ledger when the execution begins. Before every new attempt, reserve what it can spend. After each metered operation, reconcile its reservation with the available usage or cost data. When a required dimension lacks enough allowance, return a named application outcome instead of starting one more call.

The next model call now starts only when the complete execution can still afford it.

Sources