Debugging SSL EOF Errors in Kubernetes with Nginx and SonarQube

The symptom

Our GitLab CI pipeline started failing with a weird error. Every time it tried to assign a permission template to a user in SonarQube, the API call would hang for about 22 seconds, then fail:

curl: (56) OpenSSL SSL_read: OpenSSL/3.5.5: error:0A000126:SSL routines::unexpected eof while reading, errno 0

The pipeline was calling the SonarQube API to set up a new project. The first API call, creating the permission template, worked fine every time. The second call, adding a user to the template, failed with that SSL EOF error. Every time.

The project was already failing for other reasons (missing test reports), so nobody noticed this specific error for a while. But once the other issues were fixed, this one blocked the pipeline completely.

What I ruled out first

My first instinct was to blame the SonarQube server itself. Maybe it was overloaded. Maybe the API was slow. But the first API call always worked, so the server was reachable. It was just the second call that died.

Next I checked the curl command. It was a simple POST with basic auth, nothing unusual. The token was valid. The URL was correct.

I tried adding --retry 3 --retry-delay 5 to the curl command. This helped sometimes, but the error still happened. Not a real fix.

At this point I knew the problem was somewhere in the network path between the CI runner and the SonarQube server. The SSL EOF error meant the TLS connection was being torn down from the server side. But where?

The network path

Our SonarQube runs on Kubernetes, behind quite a few layers:

CI Runner (outside the cluster)
  → TLS
    → Cloud Load Balancer (Tencent CLB)
      → HTTP
        → NGINX Ingress Controller
          → HTTP
            → nginx sidecar container
              → HTTP
                → SonarQube (port 9000)

The TLS terminates at the cloud load balancer. The ingress has tls: [] (empty), so everything after the LB is plain HTTP. The SSL EOF error meant the cloud LB was the one closing the TLS connection.

But why was the LB closing the connection?

The investigation

I looked at the ingress annotations first. The timeouts looked fine:

nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"

Five minutes for read and send. The API call was failing after 22 seconds. So the ingress controller was not the one timing out.

Then I checked the nginx sidecar config. There is a sidecar container running nginx that sits between the service and SonarQube itself. It handles some header forwarding. Here is what the config looked like:

server {
    listen 8080;
    client_max_body_size 100m;

    location / {
        proxy_pass http://localhost:9000;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $http_host;
    }
}

No proxy timeouts configured. This means nginx uses its defaults. proxy_read_timeout defaults to 60 seconds. That should be fine for a 22-second API call.

But here is the thing I missed at first. The sidecar also had no proxy_buffering setting. And the default for proxy_buffering is on.

The root cause

Here is what actually happened, step by step:

1. The CI runner sends a POST to add_user_to_template on the SonarQube API.
2. The request goes through the cloud LB, the ingress controller, and the nginx sidecar to SonarQube.
3. SonarQube starts processing. This particular API call takes about 20 seconds (permission recalculation, maybe ES updates, I am not sure).
4. While SonarQube is processing, no response data is flowing back. The nginx sidecar has proxy_buffering on, so it is waiting to buffer the entire response before forwarding it. The ingress controller also has proxy_buffering on (the default), so it is waiting too.
5. The cloud LB sees zero bytes on the connection for 20+ seconds. Its idle timeout kicks in (probably 15-20 seconds, the default on Tencent CLB).
6. The LB closes the TCP connection to the CI runner. The TLS session is torn down.
7. The CI runner's OpenSSL tries to read more data and gets a TCP FIN. SSL_read: unexpected eof while reading.

The key insight is that the cloud LB does not care about application-level timeouts. It only cares about whether data is physically flowing on the TCP connection. With buffering enabled at both nginx layers, the connection was completely silent for the entire duration of SonarQube's processing.

The fix

Two changes, both in the nginx sidecar ConfigMap and the ingress annotations.

First, the sidecar. I added explicit proxy timeouts and disabled buffering:

location / {
    proxy_pass http://localhost:9000;
    proxy_read_timeout 300s;
    proxy_connect_timeout 30s;
    proxy_send_timeout 300s;
    proxy_buffering off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    # ... existing headers ...
}

Second, the ingress annotation:

nginx.ingress.kubernetes.io/proxy-buffering: "off"

Why this works

With proxy_buffering off, the moment SonarQube starts sending the HTTP response (even just the status line and headers), both nginx layers immediately forward those bytes upstream. The cloud LB sees data flowing, resets its idle timer, and keeps the TLS connection alive.

Before the fix, the "no data" window was SonarQube processing time plus nginx buffering time. After the fix, it is just the SonarQube processing time. As soon as SonarQube responds, data starts flowing.

The proxy_read_timeout 300s on the sidecar ensures it waits up to five minutes for SonarQube to respond, matching the ingress proxy-read-timeout. The proxy_http_version 1.1 and empty Connection header enable HTTP keepalive between the sidecar and SonarQube, so connections do not get torn down after each request.