A single trailing comma crashed our production pods

It was midnight. I got a mention in the group chat. A developer was asking why the production pods would not start. The error logs showed something about OpenTelemetry instrumentation in the stack trace, so the assumption was that the OTEL auto-injection was breaking things.

Here is the error the pod kept printing every time it tried to come up:

2026-07-16T23:43:45: PM2 log: App [starter:0] online
ERROR
{'event':'start up','error':'Error: unable to parse config file
    at /usr/src/app/node_modules/fwsp-config/index.js:64:18
    at /otel-auto-instrumentation/...'}

After a few seconds the pod would exit, Kubernetes would restart it, and the cycle repeated. CrashLoopBackOff.

The OTEL red herring

Look at the stack trace again. Every other line mentions otel-auto-instrumentation:

at /otel-auto-instrumentation/node_modules/@opentelemetry/instrumentation-fs/build/src/instrumentation.js:171:31
at /otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/...

It was easy to point at OTEL. The Tencent Cloud TKE platform injects OTEL auto-instrumentation via the annotation cloud.tencent.com/inject-nodejs. Maybe the instrumentation was wrapping fs.readFile and breaking something? Maybe the OTEL sidecar was corrupting the file?

I started by checking what actually changed in this deployment. I went through the source code commit history and the GitOps repo history. No OTEL configuration changes. No new sidecar versions. Nothing that would explain why OTEL suddenly started breaking things.

The stack trace was a red herring. OTEL was just the messenger, not the problem.

What I ruled out early

The error came from fwsp-config, a package that reads a JSON config file at startup and passes it to hydra-express. The message "unable to parse config file" was clear enough. Something was wrong with the JSON.

I checked the commit that was deployed. There were config file changes in it. But the UAT environment was running without issues, and the Docker image built successfully. What gives?

Then I actually tried to parse the file locally:

node -e "JSON.parse(require('fs').readFileSync('./config/config.prod.json','utf8'))"

It failed. Expected double-quoted property name in JSON at position 62248 (line 1661 column 3). That was the smoking gun.

The investigation

The error pointed to line 1661 in a 1700-line config file. I opened it and found the rolist endpoint, the last entry in the "apis" object:

  "apis": {
    "masherytoken": { ... },
    "apigatetoken": { ... },
    ... (60+ more endpoints) ...
    "rolist": {
      "url": "https://<redacted>/api/v1/endpoint?msisdn=${msisdn}",
      "method": "GET",
      "headers": {
        "Content-Type": "application/json",
        "Accept": "application/json"
      },
      "circuitBreakerSleepWindowInMilliseconds": 60000,
      "circuitBreakerRequestVolumeThreshold": 5,
      "circuitBreakerForceOpened": false,
      "circuitBreakerForceClosed": false,
      "circuitBreakerErrorThresholdPercentage": 60,
      "statisticalWindowLength": 60000,
      "statisticalWindowNumberOfBuckets": 10,
      "requestVolumeRejectionThreshold": 100,
      "timeout": 31000
    },
  },
  "services": {

See the problem? Look at the closing brace of rolist. It has a comma after it.

In JSON, a trailing comma after the last property of an object is not allowed. The parser reads "timeout": 31000, then }, then ,. The comma tells the parser: "there is another property coming in the apis object." But the next thing is } closing the apis object, not a property name. The parser says: "I expected a double-quoted property name but found }."

The fix is simple. Remove the comma:

      "timeout": 31000
    }
  },
  "services": {

But here is the thing. I checked our other config files. Six out of eight files had the same trailing comma bug. Only config.test.json and config.uat.json were fine.

Why UAT was fine

This confused me at first. UAT was running the same config file without issues. The logs even showed it loaded its config file successfully.

The answer was that the UAT Docker image was built from a different commit. The UAT branch had the correct JSON (no trailing comma). Sometime after the UAT image was built, someone edited the config files in the main branch and introduced the trailing comma. The UAT container had the old, correct version baked into its image. Production was built from the newer, broken version.

Why the build did not catch it

The Docker build does not validate JSON files. The Dockerfile does COPY . . and npm ci, neither of which checks whether your config files are valid JSON. The error only shows up at runtime when the app starts and fwsp-config calls JSON.parse().

This is a gap. If your app depends on JSON config files, the build pipeline should validate them. Otherwise you ship a broken image and only find out when the pod tries to start.

Adding config validation to the pipeline

I added a validation step to our CI/CD pipeline. Before the Docker build runs, it checks every JSON file under the config/ directory using jq:

function validate_config_json() {
  local context=${1:-}

  if [ ! -d "config" ]; then
    echo "No config directory found, skipping validation"
    return 0
  fi

  local has_error=0
  local file_count=0

  if [ -n "$context" ]; then
    local config_file="config/config.${context}.json"
    if [ -f "$config_file" ]; then
      file_count=1
      if jq . "$config_file" > /dev/null 2>&1; then
        echo "  [PASS] Valid: $config_file"
      else
        echo "  [FAIL] Invalid: $config_file"
        has_error=1
      fi
    else
      echo "  - File not found: $config_file"
    fi
  else
    for file in config/*.json; do
      [ -f "$file" ] || continue
      file_count=$((file_count + 1))
      if jq . "$file" > /dev/null 2>&1; then
        echo "  [PASS] Valid: $file"
      else
        echo "  [FAIL] Invalid: $file"
        has_error=1
      fi
    done
  fi

  echo "---"
  if [ "$file_count" -eq 0 ]; then
    echo "No JSON files found under config/ to validate"
    return 0
  fi

  if [ $has_error -ne 0 ]; then
    echo "RESULT: $file_count file(s) checked, some are INVALID"
    return 1
  fi

  echo "RESULT: $file_count file(s) checked, all valid"
  return 0
}

jq . is a quick way to validate JSON. If the file is valid, it prints the formatted output and exits 0. If the file is invalid, it prints an error and exits non-zero. The function checks all files under config/ and fails the pipeline if any of them are broken.

The pipeline now runs validate_config_json before the Docker build step. If the validation fails, the pipeline stops. No broken image gets built.

Closing

The real mistake was not the trailing comma itself. Anyone can make a typo in a 1700-line JSON file. The mistake was that we had no automated check for it. We relied on the app to crash at runtime to tell us the config was broken, and by then it was already deployed.

A one-line jq . check in the pipeline would have caught this before the image was built. It took five minutes to write the script and will save hours of debugging the next time someone accidentally adds a trailing comma.

I also learned that checking the UAT environment is not enough. UAT and production can be running different versions of the same config file if the images are built from different commits. The only way to be sure is to validate the files at build time, every time.