OpenBSD IO Benchmarking: How Many Jobs Are Worth It?

Summary

This post explores these questions through detailed fio(1) benchmarking, looking at random reads, random writes, and latency — all running on a recent build of OpenBSD 7.7-current. OpenBSD 7.7 (GENERIC.MP) #624: Wed Apr 9 09:38:45 MDT 2025 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP Test Setup # Storage: 1TB Crucial P3 Plus SSD M.2 2280 PCIe 4.0 x4 3D-NAND QLC (CT1000P3PSSD8) Tool: fio, installed via OpenBSD packages Test File Size: 64 GB (to bypass RAM cache) Block Size: 4 KiB I/O Depth: 32 Job Counts Tested: 1 to 32 Runtimes: 30s runtime per test, 10s ramp-up Results at a Glance # Throughput vs. Job Count (Random Read) Throughput vs. Job Count (Random Write) Average Read Latency Average Write Latency Summary Tables # Random Read Performance # numjobs Total BW (MiB/s) IOPS Avg Latency (µs) Notes 1 473.0 121,318 26.41 Baseline 2 808.3 207,333 30.80 Strong scaling 4 1,302.4 334,219 37.97 Excellent parallel read gain 8 1,712.0 439,728 58.22 Near peak performance 18 1,715.6 439,661 117.12 Saturation reached 32 1,618.6 415,603 180.56 Slight regression, high latency Random Write Performance # numjobs Total BW (MiB/s) IOPS Avg Latency (µs) Notes 1 265.6 68,223 58.64 Baseline 2 476.6 122,246 63.27 Good scaling 4 829.9 212,610 70.84 Steady performance increase 8 1,259.1 323,439 95.72 Approaching write peak 18 1,428.1 366,830 172.17 Plateau with rising latency 32 1,408.2 361,404 230.42 Regression due to contention Latency Overview (Read vs Write) # numjobs Read Latency (µs) Write Latency (µs) Notes 1 26.41 58.64 Minimal latency, sequential load 2 30.80 63.27 Low contention 4 37.97 70.84 Balanced performance 8 58.22 95.72 Sweet spot for throughput vs latency 18 117.12 172.17 Steep latency increase 32 180.56 230.42 High CPU & queue contention Observations # OpenBSD scales I/O quite well up to a point — notably better than expected. Job count sweet spot: Between 6 and 8 jobs gave the best balance of IOPS and latency. Too many jobs degrade performance du...