Observability for Agents with OpenTelemetry, Aspire, and Application Insights

In the previous article, we looked at AG-UI and the frontend boundary for agents. The main point was that agent behavior should not be flattened into plain chat text. Tool calls, approvals, state updates, and cancellations are separate events.

The same rule applies in production.

Do not treat an agent run as one opaque log line. An agent run is a chain of model calls, tool calls, retries, approvals, state changes, and external dependencies. If you cannot see that chain, you cannot debug cost, latency, failure, or unsafe behavior.

I want the trace to look more like this:

HTTP request
-> agent run
-> model call
-> tool call
-> model call
-> approval request
-> tool execution
-> final response

From that trace, I want to answer normal production questions:

• Which model was called?
• How many input and output tokens were used?
• Which tool ran?
• How long did each step take?
• Did the tool fail or retry?
• Did a human approve the side effect?
• Was sensitive prompt or tool data captured?
• Can I inspect the same flow locally before sending it to the cloud?

OpenTelemetry is the shared telemetry format. Aspire gives me the local loop. Application Insights is where the same signals become searchable in Azure, including the agent-focused views in Azure Monitor.

Start with the questions

Before adding packages, decide what the telemetry must explain.

For most agent systems, I would start with these questions:

Notice what is not in the table:

Log the whole prompt and hope it helps later.

This can help in a development environment. In production, it is often the wrong default. Prompts, tool arguments, retrieved documents, and tool results may contain customer data, secrets, internal tickets, file contents, or personally identifiable information.

Observability does not mean storing everything. Treat it like production data, with the same access and retention discipline you would apply anywhere else.

The basic OpenTelemetry setup

Agent Framework builds on Microsoft.Extensions.AI. That matters because Microsoft.Extensions.AI provides an OpenTelemetry middleware for chat clients. The extension method is UseOpenTelemetry on ChatClientBuilder.

Start with the chat client:

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

AzureOpenAIClient azureClient = new(
    new Uri(builder.Configuration["AZURE_OPENAI_ENDPOINT"]!),
    new DefaultAzureCredential());

IChatClient chatClient = azureClient
    .GetChatClient(builder.Configuration["AZURE_OPENAI_DEPLOYMENT"]!)
    .AsIChatClient()
    .AsBuilder()
    .UseOpenTelemetry(
        sourceName: builder.Environment.ApplicationName,
        configure: telemetry =>
        {
            telemetry.EnableSensitiveData =
                builder.Environment.IsDevelopment()
                && builder.Configuration.GetValue<bool>(
                    "AI:Telemetry:EnableSensitiveData");
        })
    .UseFunctionInvocation()
    .Build();

WebApplication app = builder.Build();

AIAgent agent = chatClient.AsAIAgent(
    name: "deployment-agent",
    instructions: """
    You help operators inspect deployment state and prepare safe release actions.
    Use tools when current deployment state is needed.
    Ask for approval before production changes.
    """,
    tools:
    [
        AIFunctionFactory.Create(CheckStagingStatusAsync),
        AIFunctionFactory.Create(DeployToProductionAsync)
    ],
    services: app.Services);

Two details matter here.

First, the sourceName is deliberate. OpenTelemetry tracing only collects activity sources that the tracer provider listens to. Aspire service defaults already add the application name as a trace source. Using builder.Environment.ApplicationName keeps the chat client activity source aligned with the application telemetry source.

Second, sensitive data is disabled unless development configuration explicitly enables it. By default, the OpenTelemetry chat client records metadata such as model information and token counts, but not raw prompts, raw outputs, tool arguments, or tool results. That default is boring in the best possible way.

Note: The exact telemetry attributes follow the OpenTelemetry Generative AI semantic conventions. Those conventions are still moving, so verify attribute names against your package version before you bake them into dashboards or KQL queries.

Add the AI source to Service Defaults

If you use Aspire, your app probably has a ServiceDefaults project with a ConfigureOpenTelemetry method. The starter template already configures ASP.NET Core, HttpClient, runtime metrics, OTLP export, and trace collection.

For agent telemetry, make sure the same source name used by UseOpenTelemetry is collected for both traces and metrics.

In the service defaults project, extend the OpenTelemetry setup:

using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

public static TBuilder ConfigureOpenTelemetry<TBuilder>(
    this TBuilder builder)
    where TBuilder : IHostApplicationBuilder
{
    builder.Logging.AddOpenTelemetry(logging =>
    {
        logging.IncludeFormattedMessage = true;
        logging.IncludeScopes = true;
    });

    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics =>
        {
            metrics
                .AddMeter(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation();
        })
        .WithTracing(tracing =>
        {
            tracing
                .AddSource(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.Filter = context =>
                        !context.Request.Path.StartsWithSegments("/health")
                        && !context.Request.Path.StartsWithSegments("/alive");
                })
                .AddHttpClientInstrumentation();

            if (builder.Environment.IsDevelopment())
            {
                tracing.SetSampler(new AlwaysOnSampler());
            }
        });

    builder.AddOpenTelemetryExporters();

    return builder;
}

These two lines matter:

.AddMeter(builder.Environment.ApplicationName)
.AddSource(builder.Environment.ApplicationName)

The chat client creates an ActivitySource and a Meter. If your OpenTelemetry configuration does not listen to them, the code can be instrumented and still produce nothing in the dashboard.

What the chat client gives you

The OpenTelemetry chat client is the easiest part of the setup. It sits in the IChatClient pipeline and observes calls made through that client.

On a normal agent run, it can record model metadata such as:

• operation name
• requested model
• response model
• provider name
• server address
• streaming flag
• finish reason
• response id
• input token count
• output token count
• cached input token count, when available
• reasoning output token count, when available
• operation duration
• time to first streamed chunk
• time between streamed chunks

With that alone, you can answer the first set of production questions.

For example:

Why did this request take 34 seconds?
-> The first model call took 1.4 seconds.
-> The search tool took 29 seconds.
-> The final model call took 2.1 seconds.

Why did this request cost more than usual?
-> The input token count jumped after retrieval.
-> The agent included eight document chunks instead of two.
-> The route used a larger model than the normal path.

Why did streaming feel broken?
-> The model call started quickly.
-> Time to first chunk was 18 seconds.
-> The delay is provider-side or prompt-side, not frontend rendering.

A trace turns “the agent was slow” into something you can actually fix.

Token usage is an operational signal

Token usage is billing data, but it is also behavior data.

High input tokens can mean:

• the chat reducer is not running
• retrieval returned too much context
• the agent is carrying stale state
• the frontend is resending too much history
• tool results are too verbose
• an inner agent is dumping its full conversation into the outer agent

High output tokens can mean:

• the agent is overexplaining
• the instructions are too broad
• structured output is not constrained
• the model is stuck in a repair loop
• the user asked for a large artifact

The OpenTelemetry middleware records token usage when the provider returns usage information. You can also inspect usage directly from responses when you need a local guardrail:

AgentResponse response = await agent.RunAsync(
    "Check release-2026-07-08 and tell me whether it is ready.",
    session,
    cancellationToken: cancellationToken);

if (response.Usage is { } usage)
{
    logger.LogInformation(
        "Agent run used {InputTokens} input tokens and {OutputTokens} output tokens.",
        usage.InputTokenCount,
        usage.OutputTokenCount);
}

I would not turn that into the main production telemetry path. Use OpenTelemetry for durable traces and metrics. Use direct response usage checks for local assertions, tests, or immediate guardrails.

For example, an integration test can fail when a supposedly small route starts consuming a huge prompt:

response.Usage?.InputTokenCount.Should().BeLessThan(8_000);

That catches prompt and retrieval regressions before they turn into a cloud bill.

Tool calls need their own spans

Model telemetry is not enough. Many agent failures happen outside the model:

• a search index is slow
• an MCP server is down
• a deployment API rejects a change ticket
• a database query times out
• a tool result is too large
• approval is granted but execution fails

If your tool is just normal C#, instrument it like normal C#.

using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;

public static class AgentTelemetry
{
    public static readonly ActivitySource ActivitySource =
        new("DeploymentAgent");
}

public static async Task<DeploymentStatus> CheckStagingStatusAsync(
    string releaseId,
    IServiceProvider services,
    CancellationToken cancellationToken)
{
    using Activity? activity = AgentTelemetry.ActivitySource.StartActivity(
        "tool deployment.check_staging",
        ActivityKind.Internal);

    activity?.SetTag("agent.name", "deployment-agent");
    activity?.SetTag("ai.tool.name", "CheckStagingStatusAsync");
    activity?.SetTag("ai.tool.risk", "read_only");
    activity?.SetTag("deployment.environment", "staging");

    var deployments = services.GetRequiredService<IDeploymentService>();

    try
    {
        DeploymentStatus status = await deployments.CheckStagingStatusAsync(
            releaseId,
            cancellationToken);

        activity?.SetTag("deployment.status", status.State);
        activity?.SetStatus(ActivityStatusCode.Ok);

        return status;
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        activity?.AddException(ex);
        throw;
    }
}

Then collect the tool source:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource(builder.Environment.ApplicationName)
            .AddSource("DeploymentAgent")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

Use low-cardinality tags for dashboards. For example, deployment.environment=production is fine. Raw releaseId, user messages, customer names, email addresses, and ticket descriptions need more care.

When you need correlation without raw values, log a stable hash or an internal id that your support process can resolve under access control.

Approval telemetry is separate from tool telemetry

In the human-in-the-loop article, the rule was:

The model requests the action. The application decides whether the action continues.

Telemetry should preserve that split.

The approval request, human decision, and eventual tool execution are related, but they are not the same event.

I usually want these approval fields:

• approvalId
• conversationId
• agentRunId
• traceId
• toolName
• toolArgumentsHash
• requestedBy
• approvedBy
• decision
• decisionReason
• createdAt
• decidedAt
• executedAt
• executionResult

An approval log might look like this:

logger.LogInformation(
    "Agent tool approval decided. ApprovalId={ApprovalId} ToolName={ToolName} Decision={Decision} TraceId={TraceId}",
    approval.Id,
    approval.ToolName,
    decision.Approved ? "approved" : "rejected",
    Activity.Current?.TraceId.ToString());

Do not log full tool arguments by default. For production, I usually prefer:

logger.LogInformation(
    "Approval request created. ApprovalId={ApprovalId} ToolName={ToolName} ArgumentsHash={ArgumentsHash}",
    approval.Id,
    approval.ToolName,
    argumentsHash);

The user interface can show exact arguments at approval time. That does not mean the telemetry backend should keep those arguments forever.

Sensitive data is a feature flag, not a casual toggle

The OpenTelemetry chat client has an EnableSensitiveData setting. There is also an environment variable:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

When enabled, telemetry can include raw inputs and outputs such as:

• system instructions
• user messages
• assistant messages
• tool definitions
• tool call arguments
• tool call results
• additional request and response properties

This helps during local debugging. It can also become a data-retention problem in production.

I would treat it as an explicit environment policy:

.UseOpenTelemetry(
    sourceName: builder.Environment.ApplicationName,
    configure: telemetry =>
    {
        telemetry.EnableSensitiveData =
            builder.Environment.IsDevelopment()
            && builder.Configuration.GetValue<bool>(
                "AI:Telemetry:EnableSensitiveData");
    })

Make the configuration explicit:

{
  "AI": {
    "Telemetry": {
      "EnableSensitiveData": false
    }
  }
}

For production, decide this with security, privacy, and compliance input. If you enable prompt capture, also decide:

• which environments may capture it
• who can query it
• how long it is retained
• whether sampling applies
• which fields are redacted before export
• whether customer data can cross region or tenant boundaries
• whether tool results include secrets or confidential documents

The default should be metadata only. Turn on content capture only when the debugging value justifies the data risk.

Redact before export when possible

OpenTelemetry processors can modify telemetry before it leaves the process. For agent systems, this matters because sensitive content often appears in attributes, logs, or exception messages.

A minimal custom processor can remove known high-risk attributes:

using System.Diagnostics;
using OpenTelemetry;

public sealed class AgentRedactionProcessor : BaseProcessor<Activity>
{
    private static readonly HashSet<string> SensitiveTags =
    [
        "gen_ai.input.messages",
        "gen_ai.output.messages",
        "gen_ai.system.instructions",
        "gen_ai.tool.definitions"
    ];

    public override void OnEnd(Activity activity)
    {
        foreach (string tagName in SensitiveTags)
        {
            activity.SetTag(tagName, null);
        }
    }
}

Register it in tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource(builder.Environment.ApplicationName)
            .AddProcessor(new AgentRedactionProcessor())
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

This is just a sketch. Real redaction usually needs to happen earlier, closer to the application boundary. I prefer this order:

avoid collecting raw data
-> redact known sensitive fields
-> sample high-volume telemetry
-> restrict access in the telemetry backend
-> set retention intentionally

Do not rely on one processor as your entire privacy story.

Local debugging with Aspire

Aspire is a good local loop for agent observability.

With an AppHost and service defaults, you can run the agent backend locally and inspect:

• resources
• environment variables
• logs
• traces
• metrics
• outbound HTTP calls
• failed dependencies
• model-call spans
• tool-call spans, if you added them

What I like about this setup is that the local flow uses OTLP. Your app emits OpenTelemetry data. The Aspire dashboard receives it. You can inspect the same telemetry before sending it to Azure Monitor.

A minimal AppHost might look like this:

var builder = DistributedApplication.CreateBuilder(args);

var api = builder.AddProject<Projects.DeploymentAgent_Api>("deployment-agent-api")
    .WithEnvironment("AI__Telemetry__EnableSensitiveData", "true");

builder.Build().Run();

Then run the AppHost:

dotnet run --project src/DeploymentAgent.AppHost

In the dashboard, inspect one request end to end:

POST /chat
-> deployment-agent model call
-> CheckStagingStatusAsync tool span
-> deployment API HTTP dependency
-> deployment-agent model call
-> approval request log

This is what I want to see locally. If you only see ASP.NET Core request spans, the app is observable as a web API but not yet observable as an agent.

If spans are missing, check the boring parts first:

• UseOpenTelemetry is in the IChatClient pipeline before creating the agent.
• sourceName matches an AddSource(...) entry.
• The chat client’s meter name is added with AddMeter(...) if you want token metrics.
• Custom tool ActivitySource names are added with AddSource(...).
• OTEL_EXPORTER_OTLP_ENDPOINT is set by Aspire or by your local environment.
• The provider actually returns usage information if token counts are missing.

Also remember that the Aspire dashboard can display sensitive configuration and telemetry. If you enable prompt capture locally, treat the dashboard as sensitive too.

Cloud monitoring with Application Insights

For Azure Monitor and Application Insights, the usual .NET path is the Azure Monitor OpenTelemetry distro. For ASP.NET Core applications, install:

dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore

Then enable Azure Monitor export:

using Azure.Monitor.OpenTelemetry.AspNetCore;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

if (!string.IsNullOrWhiteSpace(
        builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
{
    builder.Services.AddOpenTelemetry()
        .UseAzureMonitor();
}

In production, set the connection string through configuration, not as a hard-coded value:

APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=...;IngestionEndpoint=...

The Aspire service defaults template already includes a commented Azure Monitor exporter section. I usually wire it there so the same service defaults project controls local OTLP export and cloud export.

Once telemetry reaches Application Insights, you can use the normal views:

• Application map
• Transaction search
• End-to-end transaction details
• Failures
• Performance
• Logs
• Metrics
• Workbooks
• Alerts

In a workspace-based Application Insights setup, the underlying trace and log data is queryable in the connected Log Analytics workspace. That is where the KQL queries for custom dashboards, incident review, and agent-specific investigations usually live.

For AI systems, Application Insights also has an Agents view. It is built around OpenTelemetry Generative AI semantics and is meant to help inspect agent runs, model usage, tool calls, errors, token usage, and cost-related behavior.

Still, keep normal application telemetry. Agent observability should sit beside request, dependency, database, queue, cache, and runtime telemetry. The agent is part of the application, not a separate universe.

What to alert on

Start with boring alerts.

For agents, I would usually begin with:

• model call failure rate
• tool call failure rate
• high run duration
• high model duration
• high tool duration
• high input token count
• high output token count
• repeated max-iteration exits
• approval rejection spike
• approval timeout spike
• missing telemetry from an expected agent service

Avoid alerting on every weird model response. That belongs in evaluation and product analytics more than infrastructure alerting.

I split it this way:

For token spikes, tie the alert back to the meter you configured earlier with AddMeter(...). If the OpenTelemetry pipeline exports the chat client token metrics to Azure Monitor, you can use Azure Monitor metric alerts for the fast signal and a workbook or KQL query for the investigation view.

Observability tells you what happened. Evaluation tells you whether the behavior was good. You need both.

Production checklist

For a production agent, I would want this before launch:

• Every service has service.name or an equivalent cloud role name.
• The agent has a stable name.
• UseOpenTelemetry is configured on the chat client.
• The chat client source is collected by tracing.
• The chat client meter is collected by metrics.
• Tool spans exist for important application tools.
• Tool spans include low-cardinality tags such as tool name, risk, and result class.
• Approval decisions are logged with correlation ids.
• Raw prompt and tool content capture is disabled by default.
• Any enabled content capture has a documented retention and access policy.
• The Aspire dashboard shows a full local agent run.
• Application Insights receives traces, metrics, and logs in the target environment.
• A failed tool call can be found from a trace id.
• A high-token request can be found and explained.
• Alerts exist for failure rate, latency, and missing telemetry.

A decent first version is small:

UseOpenTelemetry on the chat client
AddSource and AddMeter for the AI source
custom spans around important tools
metadata-only telemetry by default
Aspire locally
Application Insights in Azure

This is enough to stop saying “the agent acted weird” and start looking at the actual execution path.

When I would use this

I would add this setup when:

• the agent calls real tools
• the agent runs in a web API or worker service
• token cost matters
• support needs to debug user incidents
• approval flows need auditability
• the agent has multiple model calls per request
• retrieval or tool latency affects user experience
• the system is moving from demo to production

The earlier you add it, the easier it is. Retrofitting observability after a production incident usually means you first have to guess where the system is opaque.

When I would not start here

I would not start with full production telemetry when:

• the agent is still a throwaway prototype
• prompts contain data you are not allowed to store
• no one has decided telemetry retention and access rules
• the team has no baseline for normal token usage
• the agent has no stable tool boundaries yet
• the main problem is answer quality, not runtime behavior

In those cases, start smaller. Log request ids, model names, token usage, and tool names locally. Use Aspire for development. Add cloud export once the boundaries are stable enough to monitor intentionally.

Conclusion

Agent observability is more than “turn on logs”. You need traces for the execution path, metrics for volume and cost, logs for decisions, and a clear policy for sensitive data.

For .NET agents, I usually start here:

Microsoft.Extensions.AI UseOpenTelemetry
-> Agent Framework agent
-> Aspire dashboard locally
-> Application Insights in Azure

Then fill the gaps that the model-call telemetry cannot know about:

• custom tool spans
• approval logs
• application-specific correlation ids
• redaction and retention rules
• alerts that match real operational failure modes

Do not store every thought-like artifact an agent produces. Store enough evidence to explain failures, cost, latency, and side effects without guessing.

The follow-up article moves from observing agent behavior to testing it. I will look at how to test Microsoft Agent Framework applications without treating the model as an untestable black box: fake model clients, tool contract tests, structured output tests, routing tests, workflow tests, and eval-style regression checks.

Further reading

• Microsoft Agent Framework docs: Agent Framework documentation
• Microsoft Agent Framework tools: Tools overview
• Microsoft.Extensions.AI: Microsoft.Extensions.AI libraries
• Microsoft.Extensions.AI API reference: UseOpenTelemetry
• Microsoft.Extensions.AI API reference: OpenTelemetryChatClient
• Aspire docs: Telemetry
• Aspire docs: Dashboard overview
• Aspire docs: C# Service Defaults
• Azure Monitor docs: Application Insights OpenTelemetry observability
• Azure Monitor docs: Enable Azure Monitor OpenTelemetry
• Azure Monitor docs: Monitor AI agents with Application Insights
• OpenTelemetry: Generative AI semantic conventions repository