Fixing Athens Go Proxy for Private GitLab Modules

Last week I set up Athens Go Proxy in our Kubernetes cluster. Public modules cached fine, but private modules from our internal GitLab always failed. I wrote about the compromise I settled on: use GOPRIVATE to bypass Athens for private modules, and let each Docker build handle GitLab authentication directly with a build secret.

I called it tech debt. I said the fix was probably simple, I just had not figured out why Athens's Go worker processes were not picking up the git configuration. I was wrong about it being simple. But I was right that it was fixable.

Here is the full investigation and the actual fix.

The symptom

When Athens tried to download a private module from our GitLab, it returned a 404 with this error embedded in the response:

not found: <internal-gitlab-host>/<group>/<private-module>@v0.0.0-20260803051323-9fc1b59e4974:
invalid version: git ls-remote -q https://<internal-gitlab-host>/<group>/<private-module>.git
exit status 128:
    fatal: could not read Username for 'https://<internal-gitlab-host>': terminal prompts disabled

But if I exec'd into the pod and ran go mod download manually with GOPROXY=direct and the same gitconfig, it worked. So the git configuration was there, the token was valid, the network was fine. The problem was something specific to how Athens spawns its child processes.

The investigation

I started by having DeepSeek-v4-pro read the Athens source code for me. The version we are running is v0.18.1. When Athens needs to download a module, it creates a GoGetFetcher and calls downloadModule, which runs go mod download as a child process. The AI pointed me to the critical function: prepareEnv in pkg/module/prepare_env.go.

Here is what prepareEnv does. It creates a completely clean environment for the child Go process. It sets GOPATH, GOCACHE, CGO_ENABLED=0, and GO111MODULE=on. Then it copies only a hardcoded whitelist of environment variables from the parent process:

HOME, PATH, GIT_SSH, GIT_SSH_COMMAND, HTTP_PROXY, HTTPS_PROXY, NO_PROXY, GOROOT

That is it. Everything else is stripped. GONOSUMDB, GOPRIVATE, GONOPROXY, GIT_CONFIG_GLOBAL, NETRC. All gone.

This is why the gitconfig at /etc/gitconfig was not working. Go's go mod download sets GIT_CONFIG_NOSYSTEM=1, which blocks the system gitconfig. And the chart's gitconfig feature mounts the config at /etc/gitconfig, which is exactly the system gitconfig that gets blocked.

The gitconfig at $HOME/.gitconfig (/root/.gitconfig) should work though, because HOME is in the whitelist and GIT_CONFIG_NOSYSTEM=1 only blocks /etc/gitconfig, not $HOME/.gitconfig. But the checksum verification was also failing because GONOSUMDB was stripped. The error was:

verifying go.mod: reading sum.golang.org/...

So even if git auth worked, the checksum verification against the public sum.golang.org would fail for private modules.

To confirm this, I ran an isolated test inside the Athens pod. I recreated the exact clean environment that prepareEnv creates, using env -i:

kubectl exec -it deploy/athens-go -n athens -- sh -c '
env -i \
  HOME=/root \
  GOPATH=/tmp/gopath \
  GOCACHE=/tmp/gopath/cache \
  CGO_ENABLED=0 \
  GO111MODULE=on \
  PATH="$PATH" \
  git ls-remote https://<internal-gitlab-host>/<group>/<private-module>.git
'

It worked. The git ls-remote returned the commit hashes. So the insteadOf gitconfig at /root/.gitconfig was being read by git in the clean environment. The git auth was not the problem. The real problem was only GONOSUMDB being stripped.

The fix: ATHENS_GO_BINARY_ENV_VARS

Athens has a configuration option called GoBinaryEnvVars. It is a list of KEY=VALUE pairs that get appended to the child process environment. They are added after the whitelisted variables, so they can even override them. The environment variable name is ATHENS_GO_BINARY_ENV_VARS.

The format uses semicolons to separate multiple pairs:

ATHENS_GO_BINARY_ENV_VARS="GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct"

The default value is GOPROXY=direct. When you set ATHENS_GO_BINARY_ENV_VARS, it completely replaces the default, so you have to include GOPROXY=direct if you want Athens to download from source. I found this out from the EnvList.Decode function in the Athens config package, which the AI pointed me to. The comment says "env vars must override config file".

I added this to the Helm chart's configEnvVars:

- name: ATHENS_GO_BINARY_ENV_VARS
  value: "GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct"

This injects GONOSUMDB and GOPRIVATE into every child Go process that Athens spawns. The wildcard pattern means "skip checksum verification for all modules" and "treat all modules as private". This is fine for our use case since Athens is an internal proxy and we do not need the public checksum database for modules cached through it.

The full Athens Helm values now look like this:

configEnvVars:
  - name: ATHENS_GO_BINARY_ENV_VARS
    value: "GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct"

lifecycle:
  postStart:
    exec:
      command:
        - sh
        - -c
        - |
          cp /etc/netrc/.netrc /root/.netrc
          TOKEN=$(awk '/password/ {print $2}' /etc/netrc/.netrc)
          cat > /root/.gitconfig << EOF
          [url "https://oauth2:${TOKEN}@<internal-gitlab-host>/"]
              insteadOf = https://<internal-gitlab-host>/
          EOF

netrc:
  enabled: true
  existingSecret: netrcsecret

upstreamProxy:
  enabled: false

The postStart hook copies the .netrc file to /root/.netrc and creates a /root/.gitconfig with the insteadOf URL rewriting. This is needed because the chart's netrc feature mounts .netrc at /etc/netrc/.netrc, but git looks for it at $HOME/.netrc. The insteadOf gitconfig embeds the GitLab token directly into the URL, so git does not need to prompt for credentials.

The upstreamProxy is disabled because we want Athens to download from source. When enabled, Athens redirects to the upstream proxy (proxy.golang.org) for modules it does not have cached. But the filter file was empty, so private modules were being redirected to the public upstream, which returned 404, and the client fell back to direct, which failed because git auth was not configured.

The Dockerfile side

On the client side, the Dockerfile needed a change too. In the first article, I used GOPRIVATE to tell Go to bypass the proxy for private modules. But now that Athens can handle private modules, I want the opposite. I want Go to use the proxy for everything, including private modules.

The problem is that GOPRIVATE implicitly sets both GONOPROXY and GONOSUMDB. GONOPROXY tells Go to bypass the proxy. So even if I point GOPROXY to Athens, the private modules are fetched directly from GitLab. That is exactly what I wanted in the first article, but now I want the opposite.

The fix is to replace GOPRIVATE with explicit GONOSUMDB and GONOPROXY. GONOPROXY set to empty means "use the proxy for everything". GONOSUMDB set to the GitLab host means "skip checksum verification for these modules". The Go documentation says GOPRIVATE is deprecated since Go 1.21 anyway.

Here is the final Dockerfile ENV block:

ENV GO111MODULE=on
ENV GONOSUMDB=<internal-gitlab-host>
ENV GONOPROXY=
ENV GOPROXY=http://<athens-proxy-host>,https://proxy.golang.org,direct

And the RUN commands no longer need the gitlab_token secret or the git config insteadOf:

COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod/ \
  go mod download -x && go mod verify

COPY . .

RUN --mount=type=cache,target=/go/pkg/mod/ \
  --mount=type=cache,target=/root/.cache/go-build/ \
  CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build -ldflags="-w -s" -o <binary-name> ./cmd/api

The go mod download -x output confirms that both public and private modules are now going through Athens. Here is a snippet from the build log:

# get http://<athens-proxy-host>/github.com/sirupsen/logrus/@v/v1.9.4.info: 200 OK (0.001s)
# get http://<athens-proxy-host>/<internal-gitlab-host>/<group>/<private-module>/@v/v0.0.0-20260803051323-9fc1b59e4974.info: 200 OK (0.001s)
# get http://<athens-proxy-host>/go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho/@v/v0.69.0.info: 200 OK (0.001s)

The private module is being fetched from Athens, not from GitLab directly. And it is fast because Athens has it cached.

What I got wrong in the first article

I said the fix was probably adding HOME=/root to GoBinaryEnvVars. That was wrong. HOME was already in the prepareEnv whitelist. The insteadOf gitconfig at /root/.gitconfig was working fine. The real problem was GONOSUMDB and GOPRIVATE being stripped, which caused the checksum verification to fail and the Go module download to error out.

I also said I did not fully understand how Athens spawns its Go processes. I was right about that. The prepareEnv function is the key piece I was missing. Once the AI walked me through the source code, everything clicked.

The .netrc file being at /etc/netrc/.netrc instead of /root/.netrc was another detail I missed. The chart's netrc feature mounts it at /etc/netrc/.netrc and sets ATHENS_NETRC_PATH, but git looks for .netrc at $HOME/.netrc. The postStart hook that copies it to /root/.netrc fixes this.

If you are setting up Athens for your team, here is my advice: forget the chart's gitconfig feature. It does not work for private modules because of GIT_CONFIG_NOSYSTEM=1. Instead, use a postStart hook to create /root/.gitconfig with the insteadOf URL rewriting. And use ATHENS_GO_BINARY_ENV_VARS to inject GONOSUMDB and GOPRIVATE into the child processes. Two lines of configuration, and everything works.