Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

Summary

Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise This blog post recounts a recent debugging session that uncovered a surprising set of issues involving Bash, getcwd(), and OverlayFS. What began as a simple customer bug report evolved into a deep dive worth sharing. Initial Bug Report A customer reported that OpenSSH scp failed after switching to OverlayFS. We found the following error in the logs: shell-init: error retrieving current directory: \ getcwd: cannot access parent directories: Inappropriate ioctl for device After analyzing the report, we realized the message didn’t come from scp itself but from the Bash shell. We asked the key question: why couldn’t Bash determine the current working directory, and why did it fail with ENOTTY (Inappropriate ioctl for device)? Ruling Out the Kernel Because the issue appeared after the introduction of OverlayFS, we reviewed the OverlayFS source code in the Linux kernel for any code paths that return ENOTTY. Although such paths exist, we considered hitting them highly unlikely. Bash uses glibc and is written in C. We examined the glibc system call wrapper for getcwd() but found no logic that could return ENOTTY. The wrapper mainly handles buffer allocation and falls back to a generic implementation if the system call fails. To test this theory, we enabled system call tracing. Surprisingly, the trace revealed that the getcwd() system call never got called. Since glibc offers multiple getcwd() implementations depending on the system, we double-checked that we had reviewed the correct Linux-specific one. We found no code path that bypassed the system call. Bash’s home made getcwd() A hunch led us to check how Bash links to the getcwd symbol: $ nm -D bash | grep getcwd ... 000c7b10 T getcwd ... This showed that Bash includes its own getcwd() function rather than relying on glibc’s version. We expected this output instead: $ nm -D bash | grep getcwd ... U getcwd ... Surprised, we inspected the Bash source an...