Optimizing My Disk Usage Program

Summary

In my previous post, Maybe the Fastest Disk Usage Program on macOS, I wrote about dumac. A very fast alternative to du -sh that uses a macOS-specific syscall getattrlistbulk to be much faster than the next leading disk usage program.I received some great technical feedback in the Lobsters thread. After implementing some of the suggestions, I was able to increase performance by ~28% on my large benchmark.hyperfine --warmup 3 --min-runs 5 './before temp/deep' './after temp/deep'Benchmark 1: ./before temp/deep Time (mean ± σ): 910.4 ms ± 10.1 ms [User: 133.4 ms, System: 3888.5 ms] Range (min … max): 894.5 ms … 920.0 ms 5 runs Benchmark 2: ./after temp/deep Time (mean ± σ): 711.9 ms ± 10.5 ms [User: 73.9 ms, System: 2705.6 ms] Range (min … max): 700.1 ms … 725.0 ms 5 runs Summary ./after temp/deep ran 1.28 ± 0.02 times faster than ./before temp/deepThe main performance gain came from reducing thread scheduling overhead, while minor gains were from optimizing access to the inode hash-set shards.Better ParallelismThe previous version of dumac used Tokio to spawn a task for each directory.// Process subdirectories concurrentlyif !dir_info.subdirs.is_empty() { let futures: Vec<_> = dir_info.subdirs.into_iter() .map(|subdir| { let subdir_path = Path::new(&root_dir) .join(&subdir) .to_string_lossy() .to_string(); tokio::spawn(async move { calculate_size(subdir_path).await }) }) .collect(); // Collect all results for future in futures { match future.await { Ok(Ok(size)) => total_size += size, // .. } }}And then in the middle of this task, calling getattrlistbulk required a blocking call:// In the middle of the async task// .. let dir_info = tokio::task::spawn_blocking({ let root_dir = root_dir.clone(); move || get_dir_info(&root_dir) // Calls getattrlistbulk}).await.map_err(|_| "task join error".to_string())??;Tokio runs many tasks on a few threads by swapping the running task at each .await. Blocking threads are spawned on demand to avoid blocking the core threads which are h...