Design queue handlers for duplicate delivery

A queue can deliver the same logical job more than once. The worker may finish the business operation and crash before acknowledging the message. When the lock expires, another worker receives it again.

Treat duplicate delivery as part of the handler contract, not as an exceptional edge case.

Start with persistent job state and an atomic claim:

public async Task HandleAsync(
    GenerateReport message,
    CancellationToken cancellationToken)
{
    JobClaim claim = await jobs.TryClaimAsync(
        message.JobId,
        leaseFor: TimeSpan.FromMinutes(5),
        cancellationToken);

    if (!claim.Acquired)
    {
        return;
    }

    Report? existing = await reports.FindByJobIdAsync(
        message.JobId,
        cancellationToken);

    if (existing is not null)
    {
        await jobs.MarkSucceededAsync(
            message.JobId,
            existing.Id,
            cancellationToken);
        return;
    }

    Report report = await generator.GenerateAsync(
        message.DocumentId,
        cancellationToken);

    await reports.SaveForJobAsync(
        message.JobId,
        report,
        cancellationToken);

    await jobs.MarkSucceededAsync(
        message.JobId,
        report.Id,
        cancellationToken);
}

TryClaimAsync must be one conditional database operation, use an optimistic concurrency token, or create a time-bound lease. A separate read followed by an update allows two workers to claim the same job. A lease needs an expiry rule so a crashed worker does not leave the job stuck forever.

The job lease is an application-level claim, not the broker’s message lock. Both need deliberate timeout behavior. For long-running handlers, renew the broker lock intentionally or persist enough state that redelivery can safely continue the work.

SaveForJobAsync should enforce a unique constraint on JobId or behave as an upsert. If another worker wins the insert race, treat the unique-constraint conflict as an existing result and read it back. When the report and job state share a database, save the report and mark the job as succeeded in one local transaction or Unit of Work. That commits the business effect and local job state atomically, while the unique constraint still protects against competing workers.

If the handler calls another system, pass a stable operation identifier when that system supports idempotency. If it must publish a follow-up message, an outbox table lets the business data and outgoing message intent commit in one local transaction; a relay publishes the message afterward. An inbox record applies the same idea on the receiving side by committing the consumed message identifier with the local changes.

Use a stable business identifier such as JobId for handler idempotency. Broker MessageId is useful for detecting duplicate sends within its configured window, but it does not replace the business idempotency key or prevent receive-side redelivery after a lost lock, restart, or failed acknowledgement.

Complete the queue message only after the durable state and required effects are committed. On transient failure, let the message be retried deliberately. On permanent business failure, record a terminal state and complete the message so the same invalid work is not retried forever. Use the broker’s dead-letter behavior for poison messages or failures that require operator inspection.

Assume at-least-once delivery. Make the second execution observe the first result instead of producing a second effect.