The Curious Case of Beam CPU Usage (2019)

Summary

While benchmarking Go vs Elixir vs Node, we discovered that Elixir (running on the BEAM virtual machine) had much higher CPU usage than Go, and yet its responsiveness remained excellent. Some of our readers suggested that busy waiting may be responsible for this behavior. Turns out, busy waiting in BEAM is an optimization that ensures maximum responsiveness. In essence, when waiting for a certain event, the virtual machine first enters a CPU-intensive tight loop, where it continuously checks to see if the event in question has occurred. The standard way of handling this is to let the operating system kernel manage synchronization in such a way that, while waiting for an event, other threads get the opportunity to run. However, if the event in question happens immediately after entering the waiting state, this coordination with the kernel can be wasteful. An interesting side effect of the busy wait approach is that CPU utilization reported by the operating system becomes misleading. Since kernel waiting (even if implemented as busy waiting) does not figure in CPU utilization results, busy waiting of the BEAM certainly does. To confirm the theory of BEAM busy waiting contributing to high CPU utilization in our benchmarking test, we re-ran the test with a modified virtual machine. The modification was to enable extra microstate accounting by running ./configure on our build with the --with-microstate-accounting=extra parameter. We then used msacc to collect microstate accounting during the 10 minutes of the sustained phase of the 100k connections test, on a c5.9xlarge AWS instance. Average thread real-time : 600000658 us Accumulated system run-time : 21254177063 us Average scheduler run-time : 586759264 us Thread alloc aux bifbusy_wait check_io async 0.00% 0.00% 0.00% 0.00% 0.00% aux 0.02% 0.06% 0.00% 0.00% 0.02% dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.00% poll 1.99% 0.00% 0.00% 0.00% 19.50% scheduler 4.89% 2.63% 6.41% 56.2...