Diagnosing a Linux Performance Regression

Summary

Systems Report: Diagnosing a Linux Performance Regression Automattic · January 14, 2025 From time to time, our systems engineers write up a case study detailing a notable moment on the infrastructure front lines. This month’s comes from Ale Crismani and Joshua Coughlan, systems wranglers who work on WordPress VIP. At Automattic, we use Kubernetes to orchestrate the infrastructure running WordPress VIP applications. We have firewall rules that ensure an application cannot connect to resources that are dedicated to other applications, and we monitor those firewall rules in real time. During routine maintenance of our servers, we noticed that our firewall monitoring had started failing. Our ensuing investigation uncovered a regression in the Linux kernel ipset module that resulted in some operations running up to 1,000 times slower. Read on to learn how we went from failure to fix. The first symptom As mentioned in the introduction, we have monitoring on our Kubernetes hosts to ensure that they conform to our security policies. One of our monitoring scripts checks if the host has the correct IPs assigned to it, if the file system has been tampered with, if firewall rules are the ones we expect, and if too much traffic is getting rejected/dropped by them. It usually runs in about 2 seconds: time ./security-checks.sh ALL OK real 0m2.085s After updating packages on a host for maintenance, though, we noticed the same monitoring checks were taking much longer: time ./security-checks.sh ALL OK real 1m13.848s That’s . . . a lot slower. A quick debugging session found that the slowness was due to running iptables-save to list rules on the host. The host has many iptables rules, but not so many that it should take more than a minute to enumerate them. Digging deeper revealed that iptables was taking a long time to return information via getsockopt: # strace -tT iptables-save ... getsockopt(5, SOL_IP, 0x53 /* IP_??? */, "\7\0\0\0\7\0\0\0KUBE-SRC-MVTE6UDDXDN4KYV"..., [40]) = 0 <0...