The team swapped a long‑standing StatsD‑based pipeline for a full OpenTelemetry collector migration while keeping the external StatsD UDP contract unchanged. This let them modernise the ingest, aggregation, and forwarding layers without forcing any service to change its instrumentation.
OpenTelemetry collector migration – architecture overview
The original stack consisted of a gostatsd sidecar on every host, an in‑house proxy called nomad for sharding, custom aggregators, and a bespoke forwarder to back‑ends such as SignalFx and S3. The migration introduced four purpose‑built OpenTelemetry Collector distributions that map to the same logical stages: collection, ingest, aggregation, and forward. The external interface – StatsD over UDP – remained identical, so applications continued to emit metrics exactly as before.
Implementation steps
- Collection sidecar replacement: The
gostatsdsidecar was removed and the existing tracing‑team Collector distribution was deployed instead. It accepts StatsD packets and also exposes an OTLP receiver, enabling native OpenTelemetry metrics without touching the application code. - Ingest routing improvement: The previous
nomadproxy hashed on(service, environment), which produced hot shards for large services. The newloadbalancingexporterhashes on the stream ID of each time‑series, spreading traffic evenly across the aggregator pool and flattening CPU usage. - Aggregation processor: A custom delta‑aggregation processor was added (open‑sourced under
atlassian‑labs) to collapse the ~4.8 billion incoming datapoints per minute down to ~220 million, a 96 % reduction. This cut CPU consumption roughly in half and removed the need to parsegostatsdpayloads. - Forwarder simplification: The bespoke forwarder was replaced by a stateless Collector distribution (named
metrics‑gateway) that uses community exporters. It provides built‑in retry, queuing, and back‑pressure handling, and adding a new destination is a configuration change rather than a new service. - Serverless support: An OpenTelemetry Lambda extension was built to mimic the
gostatsdendpoint, preserving the same address and environment variables, so no code changes were required for Lambda functions.
Operational impact
Eliminating the dedicated StatsD sidecar reduced per‑service CPU by about 3.9 %, translating to a roughly 30 % reduction in sidecar cost at fleet scale. The even ingest distribution tightened autoscaling bands, allowing off‑peak scale‑down and removing hot‑shard pressure. Aggregation now runs on about half the CPU compared to the previous stack. Overall, the combined CPU share of the old gostatsd aggregators and nomad proxy represented ~38 % of requests in the metrics clusters, with nomad alone accounting for ~13 % of total resources; removing them yields direct cost savings.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Keeping the public StatsD contract while moving the internal pipeline to OpenTelemetry Collectors lets you upgrade observability stacks without coordinating a massive re‑instrumentation effort. The approach demonstrates that a phased, sidecar‑swap migration can deliver CPU and cost reductions, smoother load distribution, and easier extensibility through configuration‑driven exporters. Teams should evaluate whether their own metrics pipelines have a stable external contract that can be preserved while modernising the backend, and consider adopting the Collector’s built‑in routing and aggregation extensions to gain similar operational efficiencies.
