Go is portable, until it isn't

Summary

We thought Go would give us a single, portable agent binary for every Linux distro. Turns out… not exactly. But also, kind of yes.This post kicks off a series about the traps we fell into while building a cross-platform server monitoring agent.First, some theory. simob is our open source server monitoring agent that powers the Simple Observability platform. We like to think of it as a passive sensor, not a long running program or daemon. Because in the real world a passive sensor does not come with a long list of requirements. It’s small, self contained and can fit inside the existing system. That is the same goal we have for simob: a lightweight standalone binary with no requisites or external dependencies.The same idea also applies to how we wanted to ship it. We wanted a project that you can compile from source on your development machine and run anywhere across your infrastructure. No complicated pipelines. No third party build services. Just a simple build that produces a portable binary.Why we chose GoIn the observability world, if you're building an agent for metrics and logs, you're probably writing it in Go. Promtail, Telegraf, Grafana Alloy and many others are all written in Go.And there are good reasons for that. First it’s compiled. A whole class of runtime errors gets caught before you even run the binary.Then there is the garbage collector. For something that’s constantly ingesting and forwarding data, not having to manage memory is a massive advantage.The Goroutines are also an excellent abstraction. We knew our agent would need to manage a lot of parallel task: tailing log files, reading from input plugins, and sending data upstream. We could write clear, sequential-looking code for each task and let the runtime handle the concurrencyAnd of course, because we thought we could compile it for any platform. "Just set GOOS and GOARCH at compile time and you're done"The simple stuffMost of the early work was simple. The Go ecosystem is more than a decade ...