Caching Terraform Modules in a Kubernetes CI Pipeline
Building a manual module cache on a shared PVC to avoid terraform init git-cloning every module from scratch in ephemeral Jenkins agents.
I wrote about caching Terraform providers a while back. That solved one part of the terraform init slowness. But there was a bigger fish.
The modules.
Our Terraform code uses git-sourced modules. A lot of them. Each environment pulls in 30-50 modules from our internal git repos. In an ephemeral Kubernetes Jenkins agent, every pipeline run starts with an empty .terraform directory. So terraform init has to git-clone every single module. One by one. Over SSH.
Here is what it looked like before any caching:
09:57:46 - <module_name> in .terraform/modules/<module_name>
09:57:46 Downloading git::ssh://git@<git-server>/<org>/<repo>.git?ref=<tag> for <module_instance>...
09:57:46 - <module_instance> in .terraform/modules/<module_instance>
09:57:46 Downloading git::ssh://git@<git-server>/<org>/<repo>.git?ref=<tag> for <module_instance>...
...
09:58:24 Downloading git::ssh://git@<git-server>/<org>/<repo>.git for <module_instance>...
Start at 09:57:46. The last module lands at 09:58:24. That is 38 seconds of nothing but git clones.
Not terrible in isolation. But this is just the init phase. In the MR pipeline, we run init separately for plan and apply stages. And in the full pipeline, init runs for every environment. Those seconds add up.
The Problem
Terraform has a built-in cache for providers. Set TF_PLUGIN_CACHE_DIR, point it at a shared directory, and providers get cached automatically.
There is no equivalent for modules.
Providers are just binaries. Terraform downloads them, checks the hash, stores them in the plugin cache. Next time, it reads the binary directly.
Modules are trickier. Each module is a git checkout. A directory with .tf files. Terraform handles module source by cloning the git repo into .terraform/modules/<name>/. There is no config option to redirect that to a shared cache.
So I had to build my own.
How Terraform Decides to Skip a Module
Before getting into the solution, it helps to understand how Terraform decides whether a module is "already installed." This is the key insight that makes the cache work.
During terraform init, Terraform reads .terraform/modules/modules.json. This manifest maps each module "..." {} block in your code to a directory on disk. For each module, if three things match (Key, the module name in your code like vpc or tke_cluster, Source, the git URL and ref like git::...?ref=<tag>, and Dir, the local directory exists on disk), Terraform skips the git clone. It prints "already installed" for that module and moves on. If any of these fields don't match, Terraform re-downloads only that module.
This is called manifest-gated skipping. The modules.json file is the gatekeeper. And it means the cache approach is straightforward. Save the entire .terraform/modules/ directory (including modules.json) to a PVC after init. Restore it before the next init. If the source refs haven't changed, everything matches and nothing needs to download.
The Approach
The same PVC from the provider caching post is mounted at /cache. The module cache lives in /cache/terraform/modules/.
Per-Project Isolation
The cache key is built from the Jenkins job name and environment name. This keeps modules from different projects or environments from mixing.
MODULE_CACHE_KEY=$(echo "${JOB_NAME}_${environment}" | tr '/' '_')
MODULE_CACHE_DIR="/cache/terraform/modules/${MODULE_CACHE_KEY}"
JOB_NAME is a Jenkins env variable like <org>/<team>/<project>/<pipeline>. After replacing / with _, the cache path becomes /cache/terraform/modules/<org>_<team>_<project>_<pipeline>/.
Restore Before Init
mkdir -p .terraform/modules
if [ -d "${MODULE_CACHE_DIR}" ] && [ -n "$(ls -A ${MODULE_CACHE_DIR} 2>/dev/null)" ]; then
cp -rn "${MODULE_CACHE_DIR}"/* .terraform/modules/
fi
This copies the cached modules.json and all module directories into the workspace before init runs. Terraform reads the manifest and decides what to skip.
Save After Init
mkdir -p "${MODULE_CACHE_DIR}"
if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then
cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/"
fi
After init resolves and downloads everything, the save step writes the full .terraform/modules/ back to the PVC. This overwrites the previous cache entirely. That is intentional. When a module ref changes, modules.json is rewritten by init. The save step must replace the old manifest with the new one.
How Changes Propagate
The cache is self-updating. Here is what happens for each scenario:
Cache hit on second run. All module sources match the restored modules.json. Terraform sees everything as "already installed." Zero git clones. The init phase goes from 38 seconds to about 4 seconds (the overhead of copying the cached files from the PVC into the pod).
Ref bump. A developer changes ?ref=v1.0 to ?ref=v2.0 in one module source. The restored modules.json still references v1.0 for that module. Terraform detects the source mismatch. It re-downloads only the changed module from git. Every other module is still "already installed." Save step overwrites the cache with the updated manifest.
New module added. A developer adds a new module "foo" {} block. The restored modules.json has no entry for foo. Terraform downloads only the new module. Save updates the cache.
Module removed. The stale directory stays on the PVC but is never referenced by modules.json. Terraform ignores it. Harmless and tiny (< 200 KB).
The Full Pipeline Stage
Here is the relevant block from the Jenkins shared library. It runs in every stage that calls terraform init.
stage('Terraform Init') {
steps {
container('terraform') {
sh '''
cd "${environment}"
MODULE_CACHE_KEY=$(echo "${JOB_NAME}_${environment}" | tr '/' '_')
MODULE_CACHE_DIR="/cache/terraform/modules/${MODULE_CACHE_KEY}"
LOCK_CACHE_DIR="/cache/terraform/lock-files/${MODULE_CACHE_KEY}"
# Restore modules from cache
mkdir -p .terraform/modules
if [ -d "${MODULE_CACHE_DIR}" ] && [ -n "$(ls -A ${MODULE_CACHE_DIR} 2>/dev/null)" ]; then
cp -rn "${MODULE_CACHE_DIR}"/* .terraform/modules/
echo "[CACHE] Modules restored"
fi
# Restore lock file from cache
mkdir -p "${LOCK_CACHE_DIR}"
if [ -f "${LOCK_CACHE_DIR}/.terraform.lock.hcl" ]; then
cp "${LOCK_CACHE_DIR}/.terraform.lock.hcl" .terraform.lock.hcl
echo "[CACHE] Lock file restored"
fi
terraform init -input=false
# Save modules to cache
mkdir -p "${MODULE_CACHE_DIR}"
if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then
cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/"
echo "[CACHE] Modules saved"
fi
# Save lock file to cache
if [ -f .terraform.lock.hcl ]; then
cp .terraform.lock.hcl "${LOCK_CACHE_DIR}/"
echo "[CACHE] Lock file saved"
fi
'''
}
}
}
In the pipeline there are 5 call sites wrapped with this pattern. Every place that runs terraform init gets the same restore/save logic. Manual plan, manual import, MR triggered plan. All of them.
The Results
Here is the save operation after a cached init. Terraform found all modules already present, so it did zero git downloads. The only work was copying the modules back to the PVC.
16:59:37 + mkdir -p /cache/terraform/modules/<org>_<team>_<project>_<pipeline>
16:59:37 + '[' -d .terraform/modules ]
16:59:37 + ls -A .terraform/modules
16:59:37 + '[' -n '<module_1> <module_2> ... <module_n>' ]
16:59:37 + cp -r .terraform/modules/<module_1> ... <module_n> /cache/terraform/modules/<org>_<team>_<project>_<pipeline>/
Performance on a project with 30+ git-sourced modules:
+------------------------+-----------------------+-----------------------------+
|Scenario |Before |After |
+------------------------+-----------------------+-----------------------------+
|First run (cold cache) |~38s (download all) |Same (populates cache) |
|Second run (cache hit) |~38s (download all |~4s (PVC copy overhead) |
| |again) | |
|Module version bump |~38s |~5s (only changed module) |
+------------------------+-----------------------+-----------------------------+Gotchas
Alpine cp Syntax
Our Jenkins agents run Alpine-based images. Busybox cp does not support the dir/. syntax for copying directory contents. It silently copies nothing.
# Silent failure on Alpine
cp -rn /cache/modules/. .terraform/modules/
# Works everywhere
cp -rn /cache/modules/* .terraform/modules/
I found this the hard way. The cache directory was populated on the PVC. The restore step ran without errors. But terraform init still downloaded every module because nothing was actually copied. The /* glob is the portable approach and works on Alpine, Ubuntu, and every other base image.
Self-Healing on Cache Corruption
The cache is best-effort only. If the PVC is unreachable or a cache directory is corrupted, the cp fails silently. .terraform/modules/ stays empty. terraform init runs normally and downloads everything from git. The next run's save step writes a fresh copy to the PVC. No pipeline ever breaks because of the cache.
The Empty Cache Bug
There is one edge case that slipped through during development and was reported by a user after release. It was blocking their pipeline entirely. The original save check was too simple:
# Before -- only checks directory exists
if [ -d .terraform/modules ]; then
cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/"
fi
I assumed that if .terraform/modules/ exists, it has content. That is not always true.
Here is the scenario. A user pushed an incomplete Terraform project that had no modules defined yet. The pipeline creates .terraform/modules/ with mkdir -p in the restore step. The cache is cold. Since the code has no module "..." {} blocks, .terraform/modules/ stays empty through the init step.
The save step checks [ -d .terraform/modules ] – true, the directory exists. So the cp -r runs. But the directory is empty. In bash, with default settings, the * glob expands to the literal character *. The command becomes cp -r "..."/* "${MODULE_CACHE_DIR}/", trying to copy a file literally named *. That file does not exist. The cp command fails. The pipeline fails with it.
The fix adds a content check alongside the existence check:
# After -- also checks directory has content
if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then
cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/"
fi
The [ -n "$(ls -A ...)" ] part returns false when the directory is empty. The cp never runs. The pipeline continues normally. Terraform init then downloads modules from git as if there was no cache at all.
I wanted to document this because the symptom was confusing. The cache directory existed on the PVC after a successful mkdir -p, so it looked like a successful cache. But there was nothing inside. The failure happened during the save step, not the restore step, which made it harder to trace.
Concurrency
The PVC is backed by CFS with ReadWriteMany. Multiple pods can mount and write to it at the same time. The module cache key is unique per job and environment, so concurrent runs of different pipelines do not collide. Two runs of the same pipeline at the same time would normally race to write to the same cache directory. But Jenkins is configured to only allow one concurrent run per pipeline. If a job is triggered while another is already running, it queues instead of running in parallel. So this edge case never actually happens.
Combined With Provider Cache
The full cache directory structure on the PVC:
/cache/
.terraformrc
terraform/
plugins/ # Provider cache (built-in, TF_PLUGIN_CACHE_DIR)
modules/ # Module cache (manual, per-project)
lock-files/ # Lock file cache (manual, per-project)
tflint/
plugins/ # TFLint plugin cache
The provider cache and the module cache sit side by side. The lock file cache completes the picture. Together they turn terraform init from a network-heavy multi-second operation into a mostly-local file copy.
What I Would Do Differently
The first run after a cache miss still downloads everything from git. This is unavoidable. But if the cache was prepopulated with known module versions from a build image or a periodic seeding job, even the first run would be fast. I haven't done this because the cache hit rate is already high.
One thing I did not explore: using rsync instead of cp for the save step. It would reduce PVC I/O by only writing changed files. For now cp -r is good enough. The entire .terraform/modules/ directory is maybe 50-200 MB across all projects.