A Misleading “File Not Found”: When the File Is Right There

Every so often, a debugging session produces an error message that seems to contradict reality. That happened recently when I tried to run a Go binary inside an Alpine-based Docker container and got a "file not found" error — despite the file clearly existing and being executable.

The root cause was subtle: the binary existed, but its ELF interpreter did not. Here’s how I tracked it down, and what you can learn from the experience.

Reproducing the Error

I had a Go program that I wanted to package into a Docker image. The Dockerfile was straightforward: build the Go source into a binary, then copy that binary into a minimal Alpine container. Yet, when I tried to execute /app/serve, the shell reported:

$ docker build .
$ docker run -it broken-container:latest /app/serve
standard_init_linux.go:228: exec user process caused: no such file or directory

But the file was definitely there:

$ docker run -it broken-container:latest ls -l /app/serve
-rwxr-xr-x    1 root     root       6220237 Nov 16 13:27 /app/serve

I started with the obvious hypothesis: permissions. But that didn’t hold up — permission errors wouldn’t produce No such file or directory, and ls -l showed the file as executable.

Narrowing It Down With strace

To pin down where the failure occurred, I reached for strace. Running it against /app/serve/ gave:

$ docker run -it broken-container:latest /bin/sh
$ /app/static # apk add strace
(apk output omitted)
$ /app/static # strace /app/serve
execve("/app/serve", ["/app/serve"], 0x7ffdd08edd50 /* 6 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

The error appears immediately at the execve system call — a useful data point. Interestingly, strace on a truly nonexistent binary produced a different pattern:

$ strace /app/asdf
strace: Can't stat '/app/asdf': No such file or directory

That contrast hinted that the problem wasn’t with the binary’s existence on disk but something about how the kernel was attempting to load it.

The Interpreter Connection

Searching for “ENOENT but file exists execve” surfaced a key insight on Stack Overflow:

When execve() returns the error ENOENT, it can mean more than one thing: the program doesn’t exist; or the program itself exists, but it requires an “interpreter” that doesn’t exist. ELF executables can request to be loaded by another program, in a way very similar to #!/bin/something in shell scripts.

The suggested diagnostic was readelf -l $PROGRAM | grep interpreter. I didn’t have readelf available inside the container, so I mounted the container’s filesystem and ran it from the host. That trick works on Linux; on a Mac you would need a different approach.

The output confirmed the suspicion:

$ mount | grep docker
overlay on /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/326ILTM2UXMVY64V7JFPCSDSKG:/var/lib/docker/overlay2/l/MGGPR357UOZZWXH3SH2AYHJL3E:/var/lib/docker/overlay2/l/EEEKSBSQ6VHGJ77YF224TBVMNV:/var/lib/docker/overlay2/l/RVKU36SQ3PXEQAGBRKSQRZFDGY,upperdir=/var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/diff,workdir=/var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/work,index=off)
$ # (then I copy and paste the "merged" directory from the output)
$ readelf -l /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged/app/serve | grep interp 
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
   01     .interp 
   03     .text .plt .interp .note.go.buildid

The binary’s interpreter was /lib64/ld-linux-x86-64.so.2 — which, of course, doesn’t exist in an Alpine container:

$ docker run -it broken-container:latest ls /lib64/ld-linux-x86-64.so.2

The Fix: Two Options

With the root cause identified, I had a couple of paths forward.

The first was to build the Go program using the golang:alpine Docker image, which is designed to produce binaries compatible with Alpine’s musl-based environment. That resolved the problem.

The second, arguably cleaner fix was to disable cgo, which forces Go to produce a statically linked binary. Setting CGO_ENABLED=0 in the build environment achieved this:

$ # first let's build it without that flag
$ go build serve.go
$ file ./serve
./serve: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=UGBmnMfFsuwMky4-k2Mt/RaNGsMI79eYC4-dcIiP4/J7v5rNGo3sNiJqdgNR12/eR_7mqqrsil_Lr6vt-rP, not stripped
$ ldd ./serve
	linux-vdso.so.1 (0x00007fff679a6000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f659cb61000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f659c995000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f659cbb0000)
$ # and now with the CGO_ENABLED_0 flag
$ env CGO_ENABLED=0 go build serve.go
$ file  ./serve
./serve: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Kq392IB01ShfNVP5TugF/2q5hN74m5eLgfuzTZzR-/EatgRjlx5YYbpcroiE9q/0Fg3zUxJKY3lbsZ9Ufda, not stripped
$ ldd  ./serve
	not a dynamic executable

That binary ran fine in the Alpine container without needing to change the build image. As an added bonus, the static binary wasn’t larger — if anything, it came out slightly smaller, which I hadn’t expected. Even though the Go binary was dynamically linked by default (I hadn’t explicitly set CGO_ENABLED=1), simply disabling cgo avoided the interpreter dependency altogether.

What’s Really Going On

This bug is a great illustration of what can happen when you compile a dynamically linked executable on one platform and run it on another. The ELF format allows an executable to declare an interpreter — a separate loader — which must be present on the target system. When it’s missing, the kernel reports ENOENT even though the executable itself is right there.

If you ever hit a "file not found" that defies all logic, it’s worth asking: is the file’s interpreter present? A quick readelf or ldd can save you a lot of head-scratching.