Privacy Boundaries for AI Telemetry

Turning off prompt capture is not enough to enforce a complete telemetry recording policy. Exceptions can still include customer text in spans. If an investigation is limited to ten samples, it can end up with forty if four replicas each track their own counters. Queued samples can exceed their intended retention if the destination resets the expiry upon arrival.

The recording-policy tip establishes what to allow. This article focuses on enforcement: inspect the actual SDK output, authorize capture with a narrow scope, reserve quota across replicas, and carry an absolute expiry through every copy of evidence.

The example is a support assistant that reads a private ticket, fetches a refund policy, calls an eligibility tool, and drafts a reply. An investigation requires the policy excerpt and the generated reply, but should not collect the ticket or account-tool payload.

Verify what the SDK actually records

Keep ordinary message capture explicitly disabled. This composition fragment assumes an existing provider client and uses Microsoft.Extensions.AI 10.10.0:

using Microsoft.Extensions.AI;

IChatClient client = new ChatClientBuilder(providerClient)
    .UseOpenTelemetry(
        sourceName: "SupportReply.AI",
        configure: telemetry => telemetry.EnableSensitiveData = false)
    .Build();

Setting this option explicitly overrides OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT for this wrapper. This is not a telemetry allowlist. Reviewing the 10.10.0 source shows there are additional paths:

For example, if a provider exception includes a synthetic customer marker, that marker remains in Activity.StatusDescription even if raw message capture is disabled. The conversation identifier can also link a trace to a private application record.

For a strict routine destination, use application-owned instrumentation built from approved values. Do not connect its tracer provider to the wrapper’s source until the source’s output complies with policy. Review the configured logger and its providers separately. Excluding the activity source does not prevent exception logging. Catching exceptions outside the wrapper does not remove a completed span or an exception already passed to a logger.

Treat any allowed sanitization as specific to the exact fields and signal paths it covers. Removing message attributes alone does not address status descriptions, events, resource attributes, log bodies, and other instrumentation. Collector redaction is an extra safeguard, but it cannot reverse data already sent to an exporter or persistent queue.

These are observations specific to this version, not a guarantee for all SDK releases. Repeat inspections and marker checks with every upgrade.

Keep routine evidence useful and bounded

An operator investigating an ineligible refund often starts with operational evidence:

A result count alone does not establish relevance or provide sufficient evidence. To determine if the model saw a conflicting refund clause, you may need separately authorized access to content.

Constrain not only field names but also the values. Use controlled failure codes. Exception messages, document titles, URLs, and log scopes can leak customer details or secrets. Keep record lookups behind authorized application access. Opaque IDs and hashes still allow correlation and should not be used in high-cardinality metric dimensions.

Review OpenTelemetry Baggage separately, since propagation can expose it to downstream services via HTTP headers. Exclude customer content and credentials, treat incoming baggage as untrusted, and configure propagation at external boundaries.

Make diagnostic capture require a versioned grant

Keep ordinary capture disabled during an incident. Use a dedicated capture service to select permitted fields and write only to a restricted evidence store. A global content flag would gather unrelated conversations as well.

An authorized service owner can issue a grant with the following limits. These values are illustrative, not SDK defaults or universal limits:

An operation ID comes from trusted application context. It is not a model argument or an investigation header used for authorization. If the incident is linked to a conversation, resolve authorized operations on the server and list those IDs. A broad tenant-wide capture requires separate justification.

This application supports a single fixed diagnostic schema: policy excerpt and generated reply. A grant authorizes only this schema and cannot select arbitrary fields. Adding or changing fields requires a separately reviewed schema and capture path with new grants.

Assign each grant a version. Changes to scope, collection deadline, evidence lifetime, or byte limit invalidate old authorizations at reservation time. The capture writer cannot expand a grant or read evidence. Investigator access is a separate role.

Put selection, reservation, and expiry on one path

The following C# code shows the enforcement boundary inside a capture service. The application-specific collaborators described below are not features provided by OpenTelemetry:

public sealed record CaptureGrant(
    Guid Id, int Version, string TenantId, string OperationType,
    IReadOnlySet<Guid> OperationIds, DateTimeOffset CollectUntilUtc,
    TimeSpan EvidenceTtl, int MaxUtf8Bytes);

public async Task TryCaptureAsync(
    CaptureGrant grant, TrustedOperation operation, CancellationToken ct)
{
    if (grant.TenantId != operation.TenantId
        || grant.OperationType != operation.Type
        || !grant.OperationIds.Contains(operation.Id))
        return;

    using var budget = CancellationTokenSource.CreateLinkedTokenSource(ct);
    budget.CancelAfter(TimeSpan.FromSeconds(2));

    var slot = await reservations.TryReserveAsync(grant, operation, budget.Token);
    if (slot is null)
        return;

    // This type exposes only the two fields authorized by this design.
    var selected = new DiagnosticFields(operation.PolicyExcerpt, operation.Reply);
    byte[]? safePayload = redactor.TrySerializeUtf8(
        selected, grant.MaxUtf8Bytes, budget.Token);
    if (safePayload is null || safePayload.Length > grant.MaxUtf8Bytes)
        return;

    var now = clock.GetUtcNow();
    var expiresAt = slot.CollectedAtUtc + grant.EvidenceTtl;
    if (now >= slot.CollectUntilUtc || now >= expiresAt)
        return;

    var sample = new EvidenceSample(slot.SampleId, grant.Id, grant.Version, operation.Id,
        slot.CollectedAtUtc, expiresAt, safePayload);
    await evidence.WriteIfFreshAsync(sample, budget.Token);
}

grant is an immutable, server-authorized snapshot loaded after verifying the investigation purpose and the requesting identity. TrustedOperation is provided by the application. The first guard checks for context mismatches before selecting content.

reservations revalidates the current grant, including its version, scope, revocation status, deadline, and shared quota. It returns a unique sample ID and database timestamp only after committing the reservation. CollectedAtUtc records the time the reservation was admitted, before payload selection and redaction, ensuring that processing or transport delays do not extend retention.

redactor serializes only DiagnosticFields, respects the byte and cancellation budgets, removes required recognized secrets, and returns no payload if it cannot verify removal. This does not guarantee that arbitrary text is non-sensitive, so the destination must be authorized for any remaining content. Do not serialize the entire operation and then try to remove private fields afterward.

evidence accepts the same sample ID idempotently and checks the absolute expiry again when admitting the sample. The fragment omits exception handling. Its calling capture boundary must catch operational exceptions from these collaborators, including capture timeouts, without failing the user request. On failed redaction or an operational failure, skip capture and record only a controlled outcome such as capture_skipped. Do not log the payload or exception text as a fallback. The example’s two-second budget requires all collaborators to honor cancellation and keep their work bounded.

Reserve quota atomically across replicas

A separate read of the remaining quota followed by an unconditional write races. In PostgreSQL, use a short transaction: acquire a lock on the grant row, check the deadline, and then perform a conditional increment:

BEGIN;

SELECT id FROM capture_grants
WHERE id = @grant_id FOR UPDATE;

WITH admission AS MATERIALIZED (
    SELECT clock_timestamp() AS collected_at_utc
)
UPDATE capture_grants
SET samples_used = samples_used + 1
FROM admission
WHERE id = @grant_id AND version = @version
  AND tenant_id = @tenant_id AND operation_type = @operation_type
  AND @operation_id = ANY(operation_ids)
  AND revoked_at IS NULL
  AND collect_until_utc > admission.collected_at_utc
  AND samples_used < max_samples
RETURNING admission.collected_at_utc, collect_until_utc;

COMMIT;

Here, id is the primary key, operation_ids is a UUID array, and timestamps use timestamptz. The @ names are driver parameters. Each replica locks the same grant row. Evaluate database time after acquiring the lock so the deadline check uses the current time rather than a timestamp from before the wait. The materialized CTE provides a single collection timestamp for both the expiry check and the returned receipt. A successful committed increment uses up one reservation. If no row is returned, the capture is denied.

Apply grant changes and revocations through the same authority, incrementing the version when the scope or lifetime changes. Include lock waits in the capture budget. Using database time prevents replica clocks from making independent decisions about when reservations expire.

Reservations are not refunded after failed redaction or storage. This approach can result in fewer than ten samples but will never authorize more than ten captures. An ambiguous reservation response may mean capacity was consumed. Do not reset the counter when retrying. Reuse a sample ID for evidence write retries, keeping the original timestamps.

The reservation is the authorization point. Revocation stops new reservations but does not retroactively cancel any that have already been issued. The producer must complete its work within the short budget and the collection deadline. If you need to immediately revoke in-flight capture, the evidence-admission service must check the current grant state before committing.

Carry absolute expiry through queues and copies

Every sample needs explicit lifecycle fields. The sample created above has this shape:

public sealed record EvidenceSample(
    Guid SampleId, Guid GrantId, int GrantVersion, Guid OperationId,
    DateTimeOffset CollectedAtUtc, DateTimeOffset ExpiresAtUtc,
    byte[] Payload);

GrantId and GrantVersion identify the authorization used, even if the live grant changes or is revoked later. Preserve that authorized version for audit purposes.

Suppose reservation admission happens at 10:00 UTC and the evidence lifetime is twenty-four hours. ExpiresAtUtc will be 10:00 UTC the next day. If a queue delivers the redacted payload at 09:50 the next morning, the destination has ten minutes left, not another full day. Any arrival after 10:00 must be rejected.

The destination should compute any storage TTL based on the original expiry minus the current time. Apply the absolute deadline on reads, since a scheduled deletion sweep may run later. Clock synchronization and destination-side time checks are still needed. Producer-side checks alone cannot prevent delayed arrivals.

Stopping collection after thirty minutes does not delete accepted samples, and deleting from the active store does not remove all exported copies. Maintain a copy register with concrete enforcement and clear ownership:

A dashboard role is insufficient if a routine operator can still access evidence through an API or export permission. Also verify service identities and administrator access. Placing data in a separate table only restricts access if the backend enforces it. If a dashboard forwards content to an AI diagnosis service, include that destination in the register.

Backups, replicas, and preservation requirements need a defined lifecycle outside the active-store sweep. If an unexpected copy appears, contain the recording path and involve the responsible incident team in its resolution. Disabling the source flag does not dispose of existing evidence.

Treat reuse for evaluation as a new authorization

A production failure can become a regression case, but its investigation grant does not authorize keeping the dataset indefinitely. Prefer using a synthetic case that preserves the failure mechanism without including the original ticket.

If real content is needed, separately approve its new purpose, readers, destination, and expiry. Maintain lineage to the source sample and record the new authorization. Copying into CI or a golden dataset must not silently remove the original deadline.

A judge model is another possible destination. Its explanations and reviewer comments may quote the input and require content controls. A numeric score with approved labels can go into routine telemetry, but explanations must remain restricted. Versioning a dataset does not extend retention permission.

Verify failure paths before enabling real capture

Use distinct synthetic markers in messages, policy excerpts, tool payloads, conversation IDs, tool names, and provider exceptions. Include a fictional email address and a fake credential. Inspect spans, status descriptions, events, logs, queues, backend queries, and exports at their destinations.

Marker tests detect known paths, not every possible private sentence. Repeat these tests after any SDK, exporter, Collector, or evaluation changes. For a real incident, also test the copy register and deletion process. A successful source-side unit test does not confirm backend access or erasure.

When to use this approach

Use this approach when production debugging requires selected user or business content across multiple service instances. Start with a single feature, a versioned grant, a shared reservation authority, and a restricted store that enforces absolute expiry. The existing observability walkthrough covers instrumentation setup, and trust-boundary guidance provides the broader authorization model.

When not to use this approach

Experiments with only synthetic data may need simpler controls. Mandatory audit evidence and investigations that require unredacted records need a separately designed evidence process with the necessary availability, preservation, and retention guarantees. This best-effort diagnostic path is a single application control, not a complete privacy or governance program.

Related reading

• Define what AI telemetry may contain before recording content
• Observability for Agents with OpenTelemetry, Aspire, and Application Insights
• Trust Boundaries Around AI Features

Sources

• Microsoft Learn: EnableSensitiveData defaults and environment-variable precedence
• .NET Extensions source: OpenTelemetryChatClient at the 10.10.0 package revision
• .NET Extensions source: Error status and exception recording at the same revision
• OpenTelemetry: Handling sensitive data
• OpenTelemetry: Baggage security considerations
• OpenTelemetry Collector Contrib: Redaction processor behavior
• PostgreSQL 18: Read Committed isolation and concurrent updates
• PostgreSQL 18: UPDATE and RETURNING
• PostgreSQL 18: Current-time functions and clock_timestamp