Caching Go Modules with Athens Proxy on Kubernetes

Every time our CI pipeline runs a Go build, it downloads Go modules from the internet. go mod download pulls from proxy.golang.org, GitHub, and our internal GitLab. Over dozens of pipeline runs per day across multiple services, this adds up. It slows down builds, it wastes bandwidth, and it is an external dependency that does not need to be one.

The obvious fix: run a Go module proxy inside the cluster. I picked Athens because it is the most mature option, supports disk storage, and can proxy both public and private modules. Here is how it went.

The setup

Athens runs as a single pod in its own namespace on our Kubernetes cluster. I deployed it using the official Helm chart (gomods/athens-charts, chart version 0.17.1, Athens version v0.18.1). The chart is managed through ArgoCD, stored in our GitOps repo.

It uses disk storage backed by a 20Gi PVC with resourcePolicy: keep, so the cache survives pod restarts and redeployments. The upstream proxy is proxy.golang.org. Any module Athens does not have cached, it fetches from there.

The values file is simple. Athens listens on port 80, exposed through an nginx ingress at <athens-proxy-host>. A single replica, 250m CPU request, 1Gi memory limit, and 5 Go module download workers.

The gitconfig is managed as a separate Kubernetes secret and mounted through the chart's gitconfig section. The secret contains the insteadOf rule that injects the GitLab token into URLs pointing to our internal GitLab.

The first thing I hit was TLS. Our ingress uses a wildcard certificate from our cloud provider for <internal-domain>. But the certificate subject is ingress.local, not the actual hostname. So when Go tried to fetch modules over HTTPS, it got:

tls: failed to verify certificate: x509: certificate is valid for ingress.local,
not <athens-proxy-host>

The fix was simple: disable TLS on the ingress and use plain HTTP. Since Athens is only accessed from inside the cluster (by CI runners and Docker builds), this is fine. The GOPROXY in our Dockerfile uses http://.

The private module problem

Here is where things got interesting. Our Go services import a private module from our internal GitLab at <internal-gitlab-host>. The idea was to have Athens cache everything, including private modules. That way, every service in the company could benefit from the cache.

I set up a gitconfig on the Athens pod with an insteadOf rule that injects the GitLab token into the URL:

[url "https://oauth2:<token>@<internal-gitlab-host>/"]
    insteadOf = https://<internal-gitlab-host>/

I verified this works from inside the pod. Running git ls-remote and even go mod download with GOPROXY=direct worked perfectly. The token was being picked up, git could authenticate, and the module downloaded fine.

But when Athens itself tried to download the private module, it failed with a 404. The error message embedded in the response was:

not found: <internal-gitlab-host>/<group>/<private-module>@v0.0.0-20260803051323-9fc1b59e4974:
invalid version: git ls-remote -q --end-of-options https://<internal-gitlab-host>/<group>/<private-module>.git
exit status 128:
    fatal: could not read Username for 'https://<internal-gitlab-host>': terminal prompts disabled

Athens was running the same go mod download internally, but the git process it spawned was not picking up the system gitconfig at /etc/gitconfig. I spent a few hours debugging this. I checked the service routing, the ingress, the pod environment, the gitconfig location. Everything was correct. The gitconfig worked for manual commands but not for Athens's internal Go worker processes.

The Athens logs showed the real story. Or rather, the lack of it. During the entire time I was testing, the Athens pod logged zero module requests. Only health checks. The 404 responses were coming from somewhere inside Athens's internal download pipeline, not from the HTTP handler I could see in the logs.

At this point I had a choice. I could dig deeper into Athens's Go module downloader internals. Maybe the GoBinaryEnvVars, maybe the process isolation, maybe the HOME directory the workers run under. Or I could admit that I do not fully understand how Athens spawns its Go processes and take the pragmatic route.

I took the pragmatic route.

The compromise: GOPRIVATE

Instead of forcing Athens to handle private modules, I told Go to bypass the proxy for our internal packages. The GOPRIVATE environment variable does exactly this:

GOPRIVATE=<internal-gitlab-host>

Modules matching <internal-gitlab-host> are fetched directly from the source, using the git token configured in the Docker build. Everything else goes through Athens.

The final GOPROXY configuration in our Dockerfile:

ENV GOPRIVATE=<internal-gitlab-host>
ENV GOPROXY=http://<athens-proxy-host>,https://proxy.golang.org,direct

The fallback to proxy.golang.org and direct means if Athens is ever down, the build still works. It just falls back to the public internet, which is slower but not broken.

The git authentication for private modules uses a Docker build secret. The pipeline passes the GitLab token as a secret, and the Dockerfile mounts it during the RUN step:

RUN --mount=type=secret,id=gitlab_token \
  --mount=type=cache,target=/go/pkg/mod/ \
  sh -c 'TOKEN=$(sed "s/^Private-Token: *//" /run/secrets/gitlab_token) && \
  git config --global url."https://oauth2:${TOKEN}@<internal-gitlab-host>/".insteadOf "https://<internal-gitlab-host>/" && \
  go mod download -x && go mod verify'

I added the -x flag to go mod download to see what is actually happening. The output confirms that public modules are hitting Athens:

# get http://<athens-proxy-host>/go.opentelemetry.io/otel/sdk/@v/v1.44.0.zip: 200 OK (0.006s)
# get http://<athens-proxy-host>/go.opentelemetry.io/otel/trace/@v/v1.44.0.zip: 200 OK (0.005s)
# get http://<athens-proxy-host>/gopkg.in/go-playground/validator.v9/@v/v9.31.0.zip: 200 OK (0.004s)

The private module is fetched directly from our GitLab, authenticated by the gitconfig insteadOf rule. The GOPRIVATE flag tells Go to skip the proxy, the sum database, and the checksum database for those packages. Which is what you want for internal code that is not published.

Closing

I do not have benchmark data. No before-and-after pipeline timings, no egress bandwidth measurements. Our pipeline speeds were inconsistent before this change (standard deviation was high), so any comparison would be noisy. I could call this CV-driven development, and honestly, that is not entirely wrong. But the reasoning is sound: fewer external downloads means less dependence on internet speed, less bandwidth cost, and a faster feedback loop for developers.

The real value here is not just for my team. This Athens instance is a shared resource. Any Go service in our Kubernetes cluster can point its GOPROXY to <athens-proxy-host> and get the same caching. The Dockerfile pattern (GOPRIVATE for internal packages, GOPROXY pointing to Athens with fallbacks) is something other teams can copy directly.

The private module situation is tech debt. Athens should be able to cache private modules too. The fix is probably simple: figure out why Athens's Go workers do not read the system gitconfig, and either fix the environment or switch to a different auth mechanism. I suspect adding HOME=/root to the GoBinaryEnvVars in the Athens config would do it, but I have not tested that yet.

If you are setting up Athens for your team, my advice is: start with GOPRIVATE for your internal packages. Get the caching working for public modules first. Then tackle private module caching as a separate effort. The public module cache alone is worth it, and the private module problem can be a follow-up.