OpenTelemetry for Go: Measuring overhead costs

Summary

Everything comes at a cost — and observability is no exception. When we add metrics, logging, or distributed tracing to our applications, it helps us understand what’s going on with performance and key UX metrics like success rate and latency. But what’s the cost? I’m not talking about the price of observability tools here, I mean the instrumentation overhead. If an application logs or traces everything it does, that’s bound to slow it down or at least increase resource consumption. Of course, that doesn’t mean we should give up on observability. But it does mean we should measure the overhead so we can make informed tradeoffs. These days, when people talk about instrumenting applications, in 99% of cases they mean OpenTelemetry. OpenTelemetry is a vendor-neutral open source framework for collecting telemetry data from your app such as metrics, logs, and traces. It’s quickly become the industry standard. In this post, I want to measure the overhead of using OpenTelemetry in a Go application. To do that, I’ll use a super simple Go HTTP server that increments a counter in an in-memory database Valkey (a Redis fork) on every request. The idea behind the benchmark is straightforward: First, we’ll run the app under load without any instrumentation and measure its performance and resource usage. Then, using the exact same workload, we’ll repeat the test with OpenTelemetry SDK for Go enabled and compare the results. Test setup For this benchmark, I’ll use four Linux nodes, each with 4 vCPUs and 8GB of RAM. One will run the application, another will host Valkey, a third will be used for the load generator, and the fourth for observability (using Coroot Community Edition). I want to make sure the components involved in the test don’t interfere with each other, so I’m running them on separate nodes. This time, I’m not using Kubernetes, instead, I’ll run everything in plain Docker containers. I’m also using the host network mode for all containers, to avoid docker-proxy introd...