Caching Terraform Providers in a Kubernetes CI Pipeline

At work, everything is already IaC. Terraform runs in Jenkins pipelines.

The flow is simple. Someone pushes IaC code to our repo, a webhook triggers the pipeline, Jenkins clones the code, runs terraform init, then terraform plan or terraform apply. Done.

The Jenkins agents run on Kubernetes. Every agent pod is ephemeral. Created for the job, destroyed after. This means every single pipeline run starts from scratch. And that includes terraform init downloading providers from the internet every time.

Honestly, this was never really a problem. A provider is a couple MB. The download takes a few seconds. Not something worth losing sleep over.

But last week the backlog was clear. So it was time to poke around.

The Setup

The pod template has two main containers: terraform and tflint. Both run inside a Kubernetes pod defined in a YAML file. A PVC named terraform-provider-cache is mounted at /cache in both containers.

volumes:
  - name: tf-cache
    persistentVolumeClaim:
      claimName: terraform-provider-cache

containers:
  - name: terraform
    image: our-registry/terraform:$TF_VERSION
    env:
      - name: TF_CLI_CONFIG_FILE
        value: /cache/.terraformrc
      - name: TF_PLUGIN_CACHE_DIR
        value: /cache/terraform/plugins
    volumeMounts:
      - name: tf-cache
        mountPath: /cache

  - name: tflint
    image: our-registry/terraform:tflint-v0.60.0
    env:
      - name: TFLINT_PLUGIN_DIR
        value: /cache/tflint/plugins
    volumeMounts:
      - name: tf-cache
        mountPath: /cache

The Built-in Cache

Terraform has a built-in provider caching mechanism. Set plugin_cache_dir in .terraformrc, or export TF_PLUGIN_CACHE_DIR. When terraform downloads a provider, it stores a copy in that directory. Next time terraform init runs and the same version is requested, it reads from the cache instead of downloading again.

# /cache/.terraformrc
plugin_cache_dir = "/cache/terraform/plugins"
disable_checkpoint = true

The pod template sets TF_CLI_CONFIG_FILE to point to it. No extra pipeline logic needed.

The First Attempt

The PVC was added to the pod template and mounted at /cache. The env vars point terraform to the right subdirectory. The .terraformrc file and cache directories are created in the first pipeline stage.

First run: terraform downloads everything. The provider binaries land on the PVC.

Then we checked the second run. Most providers loaded from cache instantly. But the tencentcloud provider kept re-downloading from the internet every time. The binary was sitting on the PVC, but terraform ignored it.

The Problem

Without the lock file, terraform queries the registry every run to find the latest compatible version. It downloads fresh copies instead of using the cached ones. The tencentcloud provider was the most noticeable because it is the largest (~200 MB).

The .terraform.lock.hcl file is generated by terraform init on the first run. It pins the exact provider versions. Our repo only contains the Terraform code – the lock file was never committed. And since the workspace is ephemeral, the lock file was gone after every run.

The Fix

We added a per-project cache for the lock file on the same PVC. The pattern: restore before init, save after init.

LOCK_CACHE_DIR="/cache/terraform/lock-files/${JOB_NAME}"
mkdir -p "${LOCK_CACHE_DIR}"

# Restore before init
if [ -f "${LOCK_CACHE_DIR}/.terraform.lock.hcl" ]; then
    cp "${LOCK_CACHE_DIR}/.terraform.lock.hcl" .terraform.lock.hcl
    echo "[CACHE] Lock file restored"
fi

export TF_PLUGIN_CACHE_DIR=/cache/terraform/plugins
export TF_CLI_CONFIG_FILE=/cache/.terraformrc
terraform init -input=false

# Save after init
if [ -f .terraform.lock.hcl ]; then
    cp .terraform.lock.hcl "${LOCK_CACHE_DIR}/"
    echo "[CACHE] Lock file saved"
fi

First run generates the lock file. It gets saved to PVC. On the next run, it is restored before init. With the lock file present, terraform knows exactly which provider versions to use and reads them straight from the cache. Every provider, including tencentcloud.

Gotchas

Init Container to Pipeline Stage

The .terraformrc file needs to exist before terraform runs. We used to have an init container in the pod template to create it. Later the setup was moved into the first pipeline stage. The init container was commented out entirely.

if [ ! -f /cache/.terraformrc ]; then
    echo 'plugin_cache_dir = "/cache/terraform/plugins"' > /cache/.terraformrc
    echo 'disable_checkpoint = true' >> /cache/.terraformrc
fi

Keeping it in the pipeline avoids maintaining a separate init container image and keeps everything visible in one stage.

The tencentcloud Debugging

The tencentcloud provider took the longest to debug. The binary was on the PVC. The cache path was correct. But terraform kept downloading it. The lock file was the missing link the whole time, but it was not obvious at first because hashicorp providers (random, time, helm, kubernetes) seemed to cache fine without it. They did not – the lock file fixed those too, we just did not notice because they are smaller downloads.

Some Notes on the PVC

The PVC is backed by Tencent Cloud File Storage (CFS) with ReadWriteMany. All pods can mount and write to it concurrently. But Terraform's docs say TF_PLUGIN_CACHE_DIR is not concurrency-safe for concurrent writers.

In practice, this is rarely an issue. Running the same pipeline multiple times at the same time is unusual. And even if it happens, the failure mode is graceful. A partially written cache just means terraform re-downloads from the internet. The next successful run writes a clean copy.

The Results

Here is the init output before caching. Pay attention to the timestamps:

14:48:10  Initializing provider plugins...
14:48:10  - Finding hashicorp/helm versions matching "~> 2.12"...
14:48:10  - Finding gavinbunney/kubectl versions matching ">= 1.14.0"...
14:48:10  - Finding tencentcloudstack/tencentcloud versions matching "> 1.18.1"...
14:48:10  - Finding hashicorp/kubernetes versions matching "~> 2.25"...
14:48:10  - Finding hashicorp/random versions matching ">= 3.0.0"...
14:48:10  - Finding hashicorp/time versions matching ">= 0.9.0"...
14:48:12  - Installing tencentcloudstack/tencentcloud v1.83.3...
14:48:16  - Installed tencentcloudstack/tencentcloud v1.83.3 (signed by a HashiCorp partner, key ID 84F69E1C1BECF459)
14:48:16  - Installing hashicorp/kubernetes v2.38.0...
14:48:17  - Installed hashicorp/kubernetes v2.38.0 (signed by HashiCorp)
14:48:17  - Installing hashicorp/random v3.9.0...
14:48:17  - Installed hashicorp/random v3.9.0 (signed by HashiCorp)
14:48:17  - Installing hashicorp/time v0.14.0...
14:48:18  - Installed hashicorp/time v0.14.0 (signed by HashiCorp)
14:48:18  - Installing hashicorp/helm v2.17.0...
14:48:18  - Installed hashicorp/helm v2.17.0 (signed by HashiCorp)
14:48:19  - Installing gavinbunney/kubectl v1.19.0...
14:48:21  - Installed gavinbunney/kubectl v1.19.0 (self-signed, key ID 1E1CE42504F5FBB2)

Start: 14:48:10. End: 14:48:21. That is 11 seconds to download 6 providers.

And after caching:

16:59:29  Initializing provider plugins...
16:59:29  - Reusing previous version of hashicorp/random from the dependency lock file
16:59:29  - Reusing previous version of hashicorp/time from the dependency lock file
16:59:29  - Reusing previous version of hashicorp/helm from the dependency lock file
16:59:30  - Reusing previous version of gavinbunney/kubectl from the dependency lock file
16:59:30  - Reusing previous version of tencentcloudstack/tencentcloud from the dependency lock file
16:59:30  - Reusing previous version of hashicorp/kubernetes from the dependency lock file
16:59:31  - Using hashicorp/time v0.14.0 from the shared cache directory
16:59:32  - Using hashicorp/helm v2.17.0 from the shared cache directory
16:59:32  - Using gavinbunney/kubectl v1.19.0 from the shared cache directory
16:59:36  - Using tencentcloudstack/tencentcloud v1.83.8 from the shared cache directory
16:59:36  - Using hashicorp/kubernetes v2.38.0 from the shared cache directory
16:59:37  - Using hashicorp/random v3.9.0 from the shared cache directory

Start: 16:59:29. End: 16:59:37. That is 8 seconds. From 11 seconds down to 8 seconds. A 27% reduction for the provider init phase.