Table of Contents

LLM calls introduce unpredictable network latency, but latency alone does not decide the execution model. An AI request should become a background job when its result still matters after the HTTP request has ended.

A response that takes 30 seconds may still belong in an interactive streaming request. A five-second operation may already need a background job if it triggers durable work, must survive a deployment, or needs retries and duplicate handling outside the original HTTP request.

The boundary is ownership:

  • A synchronous request is owned by the current HTTP exchange.
  • A streaming request is still owned by that exchange, but returns partial output while work continues.
  • A background job is owned by the application after the request has been accepted.

That distinction changes the API contract, cancellation model, retry behavior, state storage, and operational responsibilities.

Start with the lifecycle decision

Before adding a queue, ask what should happen when the client disconnects or the application restarts.

Execution modelGood fitImportant limitation
Synchronous requestShort, bounded work whose result is needed immediatelyThe caller and server must keep the request alive
Streaming requestInteractive output where partial progress improves the experienceStreaming does not make the work durable
Background jobWork that must survive disconnects, expose status, or continue independentlyRequires persistent state and an explicit job lifecycle

Keep the operation inside the request when all of these are true:

  • the caller needs the result immediately
  • the work has a bounded execution time
  • a disconnected caller no longer needs the result
  • retrying the complete request is safe
  • the hosting path can keep the connection open long enough

Move it to a background job when one or more of these are true:

  • the result must survive a browser refresh, network interruption, or deployment
  • the operation contains multiple model, retrieval, or tool steps
  • the caller needs progress or a status page
  • retries must happen without resubmitting the original HTTP request
  • the workload needs queue-based load leveling or concurrency limits
  • the operation creates an artifact that can be retrieved later
  • duplicate submissions could create duplicate cost or side effects

This is not limited to chat. Document extraction, large summarization jobs, batch classification, evaluation runs, ingestion pipelines, audio transcription, and report generation are common candidates.

Streaming is not a durability mechanism

Streaming solves a user-experience problem. It lets the caller see tokens, events, or progress before the complete result is available.

It does not answer what happens when:

  • the browser closes
  • a reverse proxy ends the connection
  • the application instance restarts
  • the client retries after losing the final response
  • the workflow must continue for several minutes
  • another device needs to retrieve the result later

A streaming operation can also be backed by a durable job, but that is a separate design. The stream then becomes one view over persisted job events rather than the only place where progress exists.

Use streaming when the interaction itself is the product. Use a job when completing and recording the work is the product.

The request-bound version is simple, but fragile

The direct version of a document-analysis endpoint is easy to write:

app.MapPost(
    "/documents/{documentId:guid}/analysis",
    async (
        Guid documentId,
        DocumentRepository documents,
        IChatClient chatClient,
        AnalysisRepository analyses,
        CancellationToken cancellationToken) =>
    {
        var document = await documents.GetAsync(
            documentId,
            cancellationToken);

        var response = await chatClient.GetResponseAsync(
            $"Analyze this document:\n\n{document.Text}",
            cancellationToken: cancellationToken);

        var analysis = await analyses.SaveAsync(
            documentId,
            response.Text,
            cancellationToken);

        return Results.Ok(analysis);
    });

This is a valid design when the caller is expected to wait and the operation can be safely repeated.

It becomes a poor fit when the analysis must outlive the request. Passing the request cancellation token through the complete operation correctly stops work when the caller leaves, but that is the opposite of durability. Ignoring the token is not a durable solution either. The work still has no stable identifier, no persisted status, and no reliable recovery path after a process restart.

The endpoint needs a different contract.

Return a job resource, not an unfinished result

For asynchronous processing, the initial request should validate and accept the work, create a stable job identifier, and return quickly.

The common HTTP shape is:

POST /documents/{documentId}/analysis-jobs
    -> 202 Accepted
    -> Location: /analysis-jobs/{jobId}
    -> Retry-After: 5

GET /analysis-jobs/{jobId}
    -> Pending | Running | Succeeded | Failed | Canceled

202 Accepted does not mean the analysis succeeded. It means the server accepted responsibility for processing it.

A minimal endpoint can make that contract explicit:

app.MapPost(
    "/documents/{documentId:guid}/analysis-jobs",
    async (
        Guid documentId,
        HttpContext httpContext,
        AnalysisJobSubmissionService submissions,
        CancellationToken cancellationToken) =>
    {
        var idempotencyKey =
            httpContext.Request.Headers["Idempotency-Key"].ToString();

        if (string.IsNullOrWhiteSpace(idempotencyKey))
        {
            return Results.BadRequest(new
            {
                error = "An Idempotency-Key header is required."
            });
        }

        var submission = await submissions.SubmitAsync(
            documentId,
            httpContext.User,
            idempotencyKey,
            cancellationToken);

        var statusUrl = $"/analysis-jobs/{submission.JobId}";
        httpContext.Response.Headers["Retry-After"] = "5";

        return Results.Accepted(statusUrl, new
        {
            submission.JobId,
            submission.Status,
            StatusUrl = statusUrl
        });
    })
    .RequireAuthorization();

The AnalysisJobSubmissionService is the important application boundary. It should:

  1. Verify that the authenticated caller may analyze the document.
  2. Scope the idempotency key to the caller or tenant and operation.
  3. Bind the key to a request fingerprint.
  4. Check whether that scoped key already belongs to an existing submission.
  5. Return the existing job only when the stored and incoming fingerprints match.
  6. Reject the request when the key exists with a different fingerprint.
  7. Persist the new job as Pending.
  8. Arrange for the job identifier to reach the worker.
  9. Return the job to the endpoint.

The fingerprint should cover every input that changes the meaning of the work, such as the documentId, operation type, prompt or workflow version, and relevant options. Reusing the same key for a different fingerprint should produce a conflict response instead of returning an unrelated job. Enforce the scoped key with a unique database constraint so concurrent submissions cannot both create a job.

Persisting the row and publishing a queue message are two separate writes. If losing a job between them is unacceptable, use an outbox or another design that can reliably recover pending jobs. A database commit followed by an unprotected queue send leaves a failure window.

Persist the job lifecycle

The queue message should not be the only record that the job exists. Keep a persistent job resource that the API and worker can both use.

A practical job record usually contains:

  • job ID
  • owner or tenant ID
  • operation type
  • input reference
  • idempotency key
  • request fingerprint
  • state
  • creation and update timestamps
  • application processing attempt count
  • prompt or workflow version
  • model or deployment identifier
  • result reference
  • safe error code and message
  • cancellation request timestamp

Do not place raw prompts, private documents, or credentials in the queue message unless the queue is explicitly designed and governed for that data. A small message containing a job ID is easier to retry, inspect, and secure. The worker can load the authorized input from the system of record.

Use a closed set of states. For example:

public enum AnalysisJobStatus
{
    Pending,
    Running,
    Succeeded,
    Failed,
    CancellationRequested,
    Canceled
}

Terminal states should remain terminal. If a duplicate queue delivery reaches a job that is already Succeeded, Failed, or Canceled, the processor should exit without running the model again.

Keep the worker separate from the HTTP request

The worker owns execution after submission. A BackgroundService is a useful .NET hosting boundary, but the durability comes from the queue and job store, not from BackgroundService itself.

public sealed class AnalysisJobWorker(
    IAnalysisJobQueue queue,
    IServiceScopeFactory scopeFactory,
    ILogger<AnalysisJobWorker> logger)
    : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        await foreach (var delivery in queue.ReadAllAsync(stoppingToken))
        {
            await using var scope = scopeFactory.CreateAsyncScope();
            var processor = scope.ServiceProvider
                .GetRequiredService<AnalysisJobProcessor>();

            try
            {
                await processor.ProcessAsync(
                    delivery.JobId,
                    stoppingToken);

                await delivery.CompleteAsync(stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                using var settlementTimeout =
                    new CancellationTokenSource(TimeSpan.FromSeconds(10));

                await delivery.ReleaseAsync(settlementTimeout.Token);
                throw;
            }
            catch (Exception exception)
            {
                logger.LogError(
                    exception,
                    "Analysis job {JobId} failed",
                    delivery.JobId);

                using var settlementTimeout =
                    new CancellationTokenSource(TimeSpan.FromSeconds(10));

                await delivery.FailAsync(exception, settlementTimeout.Token);
            }
        }
    }
}

The sample uses application abstractions because the queue semantics matter. CompleteAsync acknowledges successful processing. FailAsync should map the exception to the queue’s retry or dead-letter behavior. ReleaseAsync makes an interrupted delivery available again when the host shuts down. Settlement uses a short, independent timeout because the host shutdown token may already be canceled. Choose the timeout and settlement behavior for the backing broker.

The backing queue determines what those operations can guarantee:

  • A bounded Channel<T> can provide backpressure inside one process, but queued items disappear when that process stops.
  • A durable broker such as Azure Service Bus can survive restarts and support redelivery, but the processor must handle duplicate delivery.
  • A database-backed queue can keep job state and dispatch close together, but needs safe claiming and concurrency control.

Do not describe an in-memory queue as durable. It is appropriate only when the work is disposable or can be reconstructed from the persistent job store.

Broker delivery count and application processing attempts are related but different signals. Redelivery can happen before business processing starts, while one delivery may contain multiple internal attempts. Use broker metadata for transport diagnostics and record the processing attempts that matter to the workflow in the job store.

The worker is registered as a singleton hosted service. Creating a dependency-injection scope for each job lets the processor use scoped services such as DbContext without keeping one instance alive for the complete worker lifetime.

Make processing idempotent

There are two places where duplication can occur:

  1. The caller retries the submission because it did not receive the first response.
  2. The queue redelivers a job because processing failed before acknowledgement.

The submission idempotency key handles the first case. The persistent job state and an idempotent processor handle the second.

Before making an expensive model call, the processor should load the job and decide whether work is still required. Before committing a side effect, it should verify that the same effect has not already been applied.

That check must include an atomic claim. Move the job from Pending to Running with a conditional update guarded by the current state, an optimistic concurrency token such as a row version, or a time-bound lease. If the claim fails, another worker owns the job and the current delivery should exit or be released. A lease also needs an expiry and recovery rule so a crashed worker does not leave the job stuck in Running forever.

For model-only work, repeating a call may only duplicate cost. For jobs that send messages, update records, or invoke tools, repeating the job can also duplicate business effects. Those effects need their own idempotency boundary. The model should not be responsible for deciding whether an action already happened.

Separate request cancellation from job cancellation

The cancellation token on the submission endpoint belongs to the HTTP request. It is useful while validating and persisting the submission. It should not become the lifetime of the accepted job.

Once the API returns 202 Accepted, cancellation becomes an application operation. A typical API exposes something like:

POST /analysis-jobs/{jobId}/cancellation

That endpoint records CancellationRequested. The worker checks that state before expensive boundaries and passes a job-specific cancellation signal to model, retrieval, and tool calls where possible.

DELETE /analysis-jobs/{jobId} is also defensible when deleting the resource intentionally means requesting cancellation. If the job remains available and only changes state, an explicit cancellation operation makes that contract clearer.

Cancellation is cooperative. The application still needs to decide what happens to partial results and external side effects. A completed email cannot be canceled. A partially written report may need cleanup. A tool call may need compensation rather than cancellation.

Host shutdown is different again. The stoppingToken tells the worker that the process is stopping. With a durable queue, unfinished work should be released for redelivery rather than silently marked as a business failure.

Expose useful status without leaking internals

The status endpoint is part of the public contract, not an operational log viewer.

Return information the caller can act on:

{
  "jobId": "c7d8d96c-45ef-4c40-9a96-d6d12b9db4ec",
  "status": "Running",
  "createdAt": "2026-07-19T08:30:00Z",
  "lastUpdatedAt": "2026-07-19T08:30:07Z",
  "resultUrl": null,
  "error": null
}

Do not return raw exception messages, model payloads, credentials, or internal queue details. Map failures to stable error codes and safe descriptions. Keep trace IDs in the response when they help support teams correlate the public failure with internal telemetry.

Authorize every status and cancellation request against the job owner or tenant. Knowing a job ID must not be enough to read another user’s inputs, results, or failure details.

Push completion without making notifications durable state

Polling the status URL is the simplest client contract, but it is not the only user experience. SignalR or WebSockets can push progress and completion events to connected clients, including events sent by a background worker.

The persisted job resource must remain the source of truth. A client can disconnect before receiving an event, reconnect after the event was sent, or miss a notification during deployment. Treat push as a fast notification path, then have the client read the status resource to confirm the current state. Do not make delivery of a SignalR or WebSocket event the durability boundary.

Common implementation mistakes

Returning 202 without a status resource

The caller needs a durable identifier and a place to observe the outcome. A bare 202 only moves uncertainty from the server to the client.

Starting Task.Run from the endpoint

Task.Run does not persist work, provide backpressure, survive a process restart, or create a retry contract. It is not a replacement for a job store and queue.

Using only the queue as state

Queue delivery state is not the same as business state. Persist whether the job is pending, running, complete, failed, or canceled.

Holding a database transaction open during the model call

Persist the state transition, commit it, call the remote model outside that transaction, and persist the result in a later short transaction. Do not hold database locks while waiting on an unpredictable network dependency.

Retrying every failure automatically

Retry transient transport failures deliberately. Do not retry invalid input, authorization failures, a request that exceeds the model’s context limit, a non-retryable content-policy rejection, or deterministic tool errors as if they were temporary network problems.

Classify these terminal failures before applying a generic retry policy. Persist the job as Failed with a stable, safe error code. In a broker-backed design, dead-letter the delivery immediately when it represents poison input or requires operator inspection; otherwise complete the delivery after recording the terminal business failure. Either way, do not spend the full retry budget repeating a request that cannot succeed unchanged. A dead-letter queue also needs monitoring and an explicit inspect, correct, and resubmit process.

When to use a background job

Use this pattern when:

  • the work must continue after the request ends
  • the result needs a stable URL or status page
  • the operation is expensive enough to require queueing or concurrency control
  • retries and duplicate delivery need explicit handling
  • users may return later for the result
  • the workflow produces a durable artifact

When not to use a background job

Keep the normal request or streaming model when:

  • the interaction is short and the caller needs the immediate answer
  • partial output is useful and losing the stream should stop the work
  • the operation has no durable result or side effect
  • adding a job store, queue, worker, and status API would create more complexity than the workload justifies

Do not move work to a queue only because model calls feel slow. First decide whether the application needs durability, independent retries, progress, or load leveling. If it does not, a bounded synchronous or streaming request is usually easier to operate.

Practical takeaway

The decision is not “HTTP or queue.” It is who owns the work after the request ends.

If the request owns the work, keep it synchronous or stream the response and propagate cancellation correctly.

If the application owns the work, persist a job, return 202 Accepted with a status URL, claim it atomically, process it through an explicit worker boundary, and make retries, cancellation, and terminal failure handling part of the application contract.