Table of Contents

A successful model call proves that one request reached one model and produced one response. It does not prove that the feature is reliable, secure, observable, affordable, or useful.

That gap contains most of the engineering work.

The model call might be ten lines of code. The production feature is everything that happens before, around, and after those ten lines. It still has to authenticate the user, retrieve authorized data, manage time and cost, validate the result, handle failures, record what happened, and decide whether the output may trigger an action.

Do not design around the model call. Design around the complete application flow that contains it.

A successful response proves very little

The first working version of an AI feature often looks like this:

ChatResponse response = await chatClient.GetResponseAsync(
    "Summarize this support ticket.",
    cancellationToken: cancellationToken);

return response.Text;

This is a useful spike. It verifies credentials, connectivity, request formatting, and basic model compatibility.

It does not answer the questions that determine whether the feature can run in production:

  • Was the caller allowed to read the ticket?
  • Which ticket data was sent to the model?
  • Was sensitive content removed or retained?
  • What happens when retrieval returns incomplete context?
  • How long may the operation run?
  • How many retries are allowed?
  • What is the token and cost budget?
  • How is the output checked before it is displayed or used?
  • What happens when the provider is unavailable?
  • Can an operator explain why this request failed?
  • How will a model, prompt, or retrieval change be evaluated?

The model response is one intermediate value in a larger application flow.

Trace the complete request path

Take a support-reply feature. A user asks the application to prepare a response for a ticket.

The actual path is closer to this:

authenticated user
    -> request validation
    -> ticket authorization
    -> ticket and customer context
    -> prompt construction
    -> model call
    -> output validation
    -> policy decision
    -> draft persistence
    -> response to the user
    -> traces, metrics, and audit data

Each arrow is a boundary where latency, failure, cost, or incorrect assumptions can enter the system.

The model may answer successfully while the feature still fails because the wrong customer context was retrieved. A technically valid response may contain an unsupported claim. A good draft may be stored under the wrong tenant. A timeout may leave the user unsure whether the request is still running. A retry may create a duplicate side effect.

This is why production readiness cannot be measured by asking whether the model returned text.

The model call is not the architecture. It is one step inside a request path whose other parts can fail even when the model succeeds.

Give each boundary an explicit contract

The application becomes easier to reason about when the model call is surrounded by normal software boundaries.

For the support-reply example, I would separate at least these responsibilities:

BoundaryContract
RequestThe input is valid, bounded, and associated with an authenticated caller.
AuthorizationThe caller may access this ticket and its supporting data.
ContextThe application can identify which data was retrieved and why it was included.
Model interactionThe request has an explicit model, prompt version, options, timeout, and budget.
OutputThe result can be parsed, checked, and rejected before use.
PolicyApplication code decides whether the output may be displayed, stored, or acted on.
PersistenceRepeated processing does not create inconsistent or duplicate state.
OperationsTraces and metrics reveal where time, tokens, failures, and retries occurred.

A few named responsibilities and clear enforcement points are enough. A large framework is optional.

Keep the model behind an application service

Application code should not scatter model calls across endpoints, background workers, and controllers. Put the use case behind a service that owns the complete flow.

public sealed class SupportReplyService(
    ITicketContextReader contextReader,
    IChatClient chatClient,
    IReplyPolicy replyPolicy,
    IReplyDraftStore draftStore)
{
    public async Task<SupportReplyResult> PrepareAsync(
        Guid ticketId,
        ClaimsPrincipal user,
        CancellationToken cancellationToken)
    {
        TicketContext context = await contextReader.ReadAuthorizedAsync(
            ticketId,
            user,
            cancellationToken);

        ChatResponse response = await chatClient.GetResponseAsync(
            BuildMessages(context),
            cancellationToken: cancellationToken);

        ReplyDecision decision = replyPolicy.Evaluate(
            context,
            response.Text);

        if (!decision.IsAccepted)
        {
            return SupportReplyResult.Rejected(decision.Reason);
        }

        Guid draftId = await draftStore.SaveAsync(
            ticketId,
            response.Text,
            cancellationToken);

        return SupportReplyResult.Prepared(draftId, response.Text);
    }
}

This example is deliberately incomplete. A production implementation still needs telemetry, budgets, failure mapping, prompt versioning, and tests.

The ownership boundary is the point. The endpoint asks for a support reply. The application service coordinates the use case. The model client is only one dependency inside it.

That gives you a place to add policy without teaching the UI, controller, or provider SDK about business rules.

Reliability is a set of decisions, not a retry policy

Retries are useful for some transient failures. They are not a complete reliability strategy.

A retry can make the situation worse when:

  • the original request is still running
  • the model call is expensive
  • the operation already changed state
  • the provider is throttling the entire workload
  • the failure is caused by invalid input or an unsupported capability
  • the next attempt silently uses different context

For every dependency, decide what the system should do when it is slow, unavailable, or returns an unusable result.

FailurePossible system decision
Model timeoutCancel the attempt and return a retryable application result.
Provider throttlingApply bounded backoff or queue the work when the result can arrive later.
Retrieval missReturn an explicit no-context result instead of asking the model to guess.
Invalid structured outputReject the output or make one bounded repair attempt.
Policy violationStop before persistence or tool execution.
Persistence failureDo not report success; retry only through an idempotent operation.
Telemetry outageContinue only if telemetry is not part of an audit requirement.

The correct response depends on the use case. Make those decisions before production traffic makes them for you.

Quality, latency, and cost are one operating envelope

AI features do not have one definition of success.

A response can be accurate but too slow. It can be fast but insufficiently grounded. It can satisfy the user while consuming an unacceptable number of tokens. It can pass an offline evaluation while failing under production concurrency.

Define an operating envelope for the complete feature. For a retrieval-backed support reply, that means an acceptable answer quality and retrieval miss rate, a maximum useful latency, a token or cost budget, and strict limits on tool calls or side effects. It also means deciding what happens after a timeout, which fallback is acceptable, and what data the application retains in telemetry.

These constraints interact. Adding more context may improve one evaluation score while increasing latency and cost. Adding retries may improve completion rate while amplifying throttling. A fallback model may improve availability while changing behavior.

Treat those as system tradeoffs, not model settings.

Evaluation and observability answer different questions

Evaluation asks whether the system behaves well enough across representative cases.

Observability asks what happened during a specific execution in a running environment.

You need both.

An evaluation set can detect that a prompt change reduced groundedness for refund questions. A trace can show that one production request spent six seconds in retrieval, made two model calls, invoked a tool, and then failed during persistence.

Logs and traces do not prove answer quality, and a quality score does not explain a production timeout. Unit tests cover deterministic application behavior, but they do not measure nondeterministic result quality. One successful manual prompt establishes neither quality nor reliability.

Keep deterministic application tests around authorization, routing, parsing, policy, and state changes. Add evaluations for model and retrieval behavior. Add telemetry across the boundaries that operators must diagnose.

Define production readiness at the feature level

Before calling an AI feature production-ready, I would want explicit answers to these questions:

Behavior

  • Which user outcome does the feature produce?
  • Which examples show acceptable and unacceptable behavior?

Boundaries

  • Where are authentication and authorization enforced?
  • Which inputs and outputs are treated as untrusted?
  • Which actions require deterministic policy or human approval?

Runtime

  • What are the time, token, cost, retry, and tool-call budgets?
  • Can the caller cancel the work, and should it stay in the request, stream, or become a background job?

Failure handling

  • Which failures are retryable?
  • What does the user see when a dependency is unavailable?
  • Can partial work be resumed without repeating a side effect?

Operations

  • Can traces separate retrieval, model, tool, and persistence time?
  • Are sensitive inputs and outputs excluded or deliberately governed?
  • Can operators detect degraded quality, latency, cost, or error rate?

Change control

  • How are model, prompt, retrieval, and tool changes evaluated before release?
  • Can the application roll back without changing stored data contracts?
  • Are provider-specific capabilities explicit, or are they hidden behind an abstraction that pretends all models behave the same?

If these questions do not have answers, the system may still be a useful prototype. It is not ready to be operated as a dependable product feature.

When this framing is useful

This framing matters whenever an AI result affects a user workflow, accesses private data, depends on retrieval, invokes tools, changes state, has meaningful cost, or creates an operational promise. At that point, the model call is no longer the architecture. The complete request path is.

When a direct model call is enough

A small direct call is reasonable for a local experiment, an internal spike, or a low-risk disposable tool where failure has little consequence and no sensitive data or side effects are involved.

Do not build a platform before you have a use case. Once the feature creates a real product promise, stop treating the model call as the architecture.

Practical takeaway

Start with one important AI request in your application and draw its complete path.

Mark every dependency, policy decision, data boundary, side effect, and operational signal. Then define what should happen when each part is slow, unavailable, or wrong.

The model call will probably remain one of the smallest boxes in the diagram.

Sources