Latency Profiling in Python: From Code Bottlenecks to Observability

Summary

In the previous article, we've mapped the terrain where latency hides: from the wire through the NIC, into kernel space, across the user-space boundary, and finally into application code. The next logical step is to quantify time spent at each stage of request processing, attributing cost to specific code paths, system calls, and data movement operations. 1. Introduction: Why Averages Are Dangerous The trading system reports an average latency of 10ms. The monitoring dashboard is green. Everything looks fine ... until it isn't. Suddenly, a big losing trade flashes across the screen. Post-mortem shows the system took 83ms to respond during a volatility spike, and by the time the order hit the exchange, the price had already moved. This is the problem with averages: they hide the disasters. This is why seasoned engineers obsess over percentiles such as the 95th, 99th, and 99.9th. Recommended reading: How to Lie with Statistics by Darrell Huff 2. The Anatomy of Latency To understand why averages fail, we first need to understand what time actually means inside a system. Performance Profiling vs Latency Profiling Performance profiling tells you where your code spends time on average. Latency profiling chases the tail (e.g. the 99th percentile): the rare outliers that wreck performance guarantees. Dimensions of Time Latency is multi-dimensional. Several clocks tick inside every system, each telling a slightly different story. Wall Clock Time is what users experience: the real duration between start and finish. CPU Time is how long your code actually ran on a CPU core. If your process spent 1ms computing and 2ms waiting for a database query to return, the CPU time is 1ms even though wall time is 3ms. 💡 Demo: Fork the companion GitHub repo and run python 01_wall_time_vs_cpu_time.py You'll see something like: Wall time: 1.530 s CPU time: 0.529 s One function burns CPU cycles, the other just waits. Both consume time but for different reasons. Sources of Latency Worth Measuri...