Fixing the Docker Buildx EOF Error by Pinning Buildkit to v0.17.2

The error

The build would run to completion, push layers to the registry, and then crash right at the end.

ERROR: failed to receive status: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: EOF", received prior goaway: code: NO_ERROR, debug data: "graceful_stop"

The build itself was fine. The image was pushed. But the CI job failed, which meant the pipeline stopped. Rerunning the job fixed it about half the time. The other half, it failed again. This was not a flaky network. Something was wrong with Buildx itself.

The environment

Our pipeline uses a shared library function that gets loaded at runtime from a centralized script file. Here is roughly what it does:

1. Creates a Docker context pointing to a DinD service (docker:27.3.1-dind, TLS disabled)
2. Creates a Buildx builder with the docker-container driver
3. Runs docker buildx build --push to build and push the image

The builder was not pinned to any specific Buildkit version. It was pulling whatever moby/buildkit:buildx-stable-1 resolved to at the time. That turned out to be the problem.

Here is the relevant part of the function before the fix:

docker buildx create --name ${builder_name} --driver docker-container --use ${context_name}

No --driver-opt flag. No image pinning. Whatever Buildkit version Docker decided to pull, that is what we got.

The investigation

I spent the first hour blaming the network. Our DinD service talks to the runner over TCP, and EOF errors usually mean a connection dropped. I checked the DinD logs. Nothing. The connection was healthy. The build was completing. The push was completing. The error happened after the push, when the client was waiting for the final status from the Buildkit daemon.

Then I searched for the exact error message and found docker/buildx#2789. The issue had multiple reports from users on self-hosted GitHub runners, all describing the same pattern: builds complete, push succeeds, then EOF at the very end.

The Buildx maintainer, crazy-max, confirmed it was a gRPC transport bug in Buildkit. The Buildkit daemon was sending a graceful shutdown signal (graceful_stop) to the gRPC connection while the client was still waiting for status updates. The client saw the connection close, got an EOF, and reported it as an error. The fix was merged in moby/buildkit#5530.

The version that had the bug was whatever buildx-stable-1 resolved to at the time. The fix landed in Buildkit v0.17.2.

The fix

One line changed. I added --driver-opt image=moby/buildkit:v0.17.2 to the docker buildx create command.

docker buildx create --name ${builder_name} --driver docker-container \
  --driver-opt image=moby/buildkit:v0.17.2 \
  --use ${context_name}

That is it. No more EOF errors. The build runs to completion, the status is received cleanly, and the CI job passes.

The full updated function, simplified, now looks like this:

# ... setup docker context, trap cleanup, TLS config ...

docker buildx create --name ${builder_name} --driver docker-container \
  --driver-opt image=moby/buildkit:v0.17.2 \
  --use ${context_name}
docker buildx inspect --bootstrap

# ... resolve registry and tag based on environment ...

docker buildx build -f $DOCKERFILE --progress plain --platform ${PLATFORMS} $ARGS \
  -t <registry>/<namespace>/<project>:<tag> \
  --provenance=false . --push

Closing

The mistake was not pinning the Buildkit image. Relying on a floating tag like buildx-stable-1 means your CI can break silently when a new version ships with a bug. The fix is one line, but the debugging took longer than it should have because the error message does not point to Buildkit at all. It looks like a network problem.

If you run Buildx with a docker-container driver, pin the Buildkit image. Always. It is a Docker container running inside your CI. It gets updated. It can break. The fix is --driver-opt image=moby/buildkit:v0.17.2, or whatever version you have tested and trust.

The GitHub issue thread also mentions that the bug was fixed in Buildkit v0.17.2, so if you are running a newer version, you should not hit this. But if you are not pinning, you never know what version you are actually running until something breaks.

I might revisit this later if we ever need multi-architecture builds. For now, we only build linux/amd64, so we could even drop Buildx entirely and use plain docker build and docker push. But keeping the Buildx setup means we are ready for multi-arch when the time comes. Just with a pinned Buildkit image this time.