OpenTelemetry telemetry reduction for AI agents: practical filtering with the Collector

What changed is the introduction of an "optimized" pipeline alongside the existing raw pipeline in the OpenTelemetry Collector, using filter and transform processors to strip out bulky fields, drop whole events, and hash identifiers before the data reaches the backend. This matters to AI engineers, platform teams, and SREs because the unfiltered output of an AI coding agent can generate hundreds of records and megabytes of JSON per run, inflating storage bills and obscuring the signals that actually answer operational questions.

Why Raw Telemetry Is Unsustainable

When an agent runs a loop of prompts, model calls, tool invocations, and decisions, each step emits spans, logs, and metrics. A single run with all content flags enabled produced 181 telemetry records and 289 KB of JSON, most of which contained full prompts, file bodies, or tool parameters that no dashboard ever queries. The default instrumentation was designed for debugging a single session, not for a fleet handling dozens of sessions daily. The result is high cardinality (e.g., a new session.id creates a fresh time‑series) and unnecessary storage of sensitive payloads.

Building an Optimized Collector Pipeline

The architecture keeps one OTLP receiver that feeds two pipelines:

• raw – batch processor only, exported to a control‑group endpoint.
• optimized – batch plus a series of processors that perform the reduction.

Key processors in the optimized path are:

• filter/drop_api_request – drops any log record where attributes["event.name"] == "api_request", cutting roughly a quarter of log volume.
• transform/logs_delete_fields – removes large attributes such as prompt, response, tool_parameters, workspace.host_paths, user.email, and terminal.type. The remaining fields still convey size trends without the raw text.
• transform/logs_hash_and_trim – replaces session.id and user.id with a 16‑character SHA‑256 substring, limiting exposure of personally identifiable information while preserving uniqueness for correlation.

A minimal

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch: {}
  filter/drop_api_request:
    error_mode: ignore
    logs:
      log_record:
        - 'attributes["event.name"] == "api_request"'
  transform/logs_delete_fields:
    error_mode: ignore
    log_statements:
      - delete_key(log.attributes, "prompt")
        ...
  transform/logs_hash_and_trim:
    error_mode: ignore
    log_statements:
      - set(log.attributes["session.id"], Substring(SHA256(log.attributes["session.id"]), 0, 16))
        ...
service:
  pipelines:
    logs/raw:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/lgtm_raw_logs]
    logs/optimized:
      receivers: [otlp]
      processors: [batch, filter/drop_api_request, transform/logs_delete_fields, transform/logs_hash_and_trim]
      exporters: [otlp/lgtm_opt_logs]

snippet illustrates the essential configuration without adding unrelated settings.

Operational and Security Considerations

Running two pipelines doubles the export paths, so operators should decide whether the raw path is needed only for occasional validation. In production the raw exporters can be pointed at cheap object storage or disabled entirely, reducing network traffic.

From a security perspective, hashing identifiers and removing full tool payloads lowers the risk of leaking sensitive code or user information. The filtering performed here is a data‑reduction step, not an access‑control boundary; it does not replace authentication or authorization mechanisms.

Metrics remain untouched except for the attribute cleanup, but the reduced cardinality of resource attributes (e.g., fewer distinct session.id values) improves query performance in downstream observability tools.

Related CloudNinjas coverage: AI engineering.

What This Means For Practitioners

Adopt a dual‑pipeline Collector when instrumenting AI agents that emit verbose telemetry. Start by mirroring the existing receiver into a raw pipeline for baseline comparison, then incrementally add filter and transform processors until the optimized pipeline retains only the counters and short attributes needed for cost and performance monitoring. Verify that dashboards still receive the required signals, and monitor storage metrics to confirm the expected reduction. Finally, treat the hashing and field‑deletion steps as a privacy‑enhancing practice rather than a security control, and ensure that any downstream authorization remains unchanged.