The order of files in your ext4 filesystem does not matter

Summary

home the title is a cheeky reference to something at the front page of the orange site today. i don’t want to be misleading here; glob order in bash is “alphanumeric”-ish. this is more about documenting a wierd bug we encountered recently after a node image patch update, which in-turn caused a multi-hour outage since we could not get ahead of it in time. we have JVM workloads on production, with dockerfiles that look like this. CMD ["java", "-cp", "/jars/*", "-server", ..., "com.acmecorp.app.Application"] the wildcard here, is not a glob, since the thing is not running in a bash shell. the actual argument value the JVM receives is "/jars/*", and in turn decides to be helpful, and expand the wildcard anyway. in posix systems, this happens to use the readdir syscall. tl;dr overlayfs delegates readdir to the underlying filesystem the underlying filesystem happened to be an ext4 ext4 readdir optimizes by caching the entries of a directory in a “hashed b-tree” with a specific “directory hash seed” the directory hash seed changed with the node image patch update, causing the jar order in the classpath to change, causing an uncaught throwable leading to “stuck” application initialization the rest of the blog is about how i went about trying to figure out how we got here. 1. red herring - buildah squashing layers by default with a recent overhaul a couple years ago of our CI/CD setup (from Jenkins to GHA), and a couple other reasons, we switched from docker to buildah for building the container images, and we noticed that some of the buildah built images would not start up. we were copying the files into the container image in a specific order, to possibly save bandwidth with ‘shared’ layers. this involved copying jars in a specific order of “volatility” (bottom-to-top); the specific project jars internal dependencies’ jars kotlin stdlib, and dependencies’ jars other dependency jars the intent was that layers 3 and 4 would “rarely” change, and surely this should help a bunc...