await foreach does not automatically use your cancellation token
A CancellationToken in scope does nothing unless the async enumeration receives and observes it.
Reliability
Run production evaluations outside the user response path
Hand production samples to a separate evaluation worker instead of making users wait for an evaluator model.
AI Systems Need Runtime Budgets
How to bound time, tokens, retries, tool calls, and estimated cost across a complete AI execution instead of limiting each call in isolation.
Return Retry-After when a client should back off
Tell API clients when another attempt may succeed instead of leaving them to guess a retry delay.
Choose histogram boundaries from the service objective
Put the latency objective into the histogram boundaries so compliance can be calculated without interpolating between buckets.
Use concurrency tokens to detect lost updates
Make stale writes fail explicitly instead of silently overwriting newer database state.
Retries Are Not a Recovery Strategy
Why bounded retries help with transient faults but cannot resolve unknown outcomes, partial workflows, or duplicated side effects in AI systems.
Separate ingestion health from retrieval readiness
A failed ingestion run should not stop retrieval while an accepted index remains queryable and eligible to serve.
Cap tool-call loops explicitly
Set a hard iteration limit on automatic function invocation and treat hitting it as an explicit application outcome.
What Happens When Your AI Feature Fails?
A practical way to identify failure modes, choose explicit application behavior, and test an AI feature before its successful path hides the hard decisions.