Golang optimizations for high‑volume services

Summary

Building services that sit on top of a Postgres replication slot and continuously stream data into Elasticsearch is a great way to get low‑latency search without hammering your primary database with ad‑hoc queries. But as soon as traffic ramps up, these services become a stress test for Go’s memory allocator, garbage collector, and JSON stack.This post walks through optimizations applied to a real-world service that:Connects to a Postgres replication slotTransforms and enriches the change eventsUses Elasticsearch’s bulk indexer to index and delete documentsThe constraints: the service cannot stop reading from the replication slot for long (or Postgres disk will grow), and it cannot buffer unbounded data in memory (or Go’s heap will). The goal is to keep latency and memory stable under sustained high volume.Replication slots are relentless: as long as your primary receives writes, the slot will keep producing changes. If your consumer slows down, Postgres has to retain more WAL segments, increasing disk usage on the database server. If your consumer tries to “just buffer more” in memory, the heap will balloon and garbage collection will kick in more frequently, stealing CPU from useful work.In this setup, you typically have three competing forces:Backpressure from Elasticsearch bulk indexingThe continuous stream of changes from the replication slotThe Go runtime’s allocator and garbage collector trying to keep up with allocations in your hot pathThe design work is about turning this into a stable flow: limit in-flight work, keep memory usage predictable, and reduce per-message overhead.One of the earliest hot spots in services like this is JSON encoding/decoding for documents going into Elasticsearch. The standard library’s encoding/json is correct and convenient, but it trades some performance for safety and reflection-based flexibility.High-volume services often switch to jsoniter (github.com/json-iterator/go) for a few reasons:Faster encoding/decoding for common p...