Use OpenTelemetry before you need production debugging

Do not wait until production is already slow or failing before adding observability. At that point, you are trying to understand a live system with missing evidence, partial logs, and no reliable view of where time is spent.

Add OpenTelemetry while the system is still small enough to reason about. Start with the boring signals: request duration, dependency calls, database queries, queue operations, cache misses, model calls, retries, and failures. Those traces and metrics become the baseline you will need later.

In .NET, keep the OpenTelemetry packages at the hosting edge where you configure collection, sampling, and export. Application and domain code should normally emit through the BCL APIs: System.Diagnostics.ActivitySource for traces, System.Diagnostics.Metrics.Meter for metrics, and ILogger for logs. That keeps the core code independent from the telemetry backend.

The host setup can be small:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(builder.Environment.ApplicationName)
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation())
    .WithMetrics(metrics => metrics
        .AddMeter(builder.Environment.ApplicationName)
        .AddRuntimeInstrumentation());

Those instrumentation calls give you a lot of the boring signals without hand-written spans: incoming ASP.NET Core requests, outgoing HttpClient calls, SQL client activity, and runtime metrics. Add your own ActivitySource only around boundaries the framework cannot see, such as a business workflow step, a tool invocation, or a queue handoff.

For AI calls built on Microsoft.Extensions.AI, prefer the built-in UseOpenTelemetry(...) chat client middleware over custom timing wrappers. Configure it with the source name you collect in the host, then add the same source and meter names through AddSource(...) and AddMeter(...). That is usually enough to see model calls and token-related metrics when the provider returns usage information.

The point is not to collect everything. The point is to make important boundaries visible. When a request crosses into a database, HTTP dependency, background job, storage service, or AI model call, there should be enough telemetry to answer what happened, how long it took, and which part failed.

Keep the signal safe and useful. Do not put secrets, prompts with sensitive user data, access tokens, raw payloads, or personally identifiable information into tags and logs. If you enable sensitive AI telemetry such as raw inputs, outputs, tool arguments, or tool results, treat that as production data with explicit access control and retention rules.

Use this early for services where production debugging would cross process, database, queue, cache, or model boundaries. Do not use telemetry as a dumping ground for payloads you would not store in a normal audit log.