Upgrading SonarQube: from 10.6.0 to 26.7.0

The plan

I needed to upgrade SonarQube. The old version was sonarqube:10.6.0-community, running on a Kubernetes cluster with PostgreSQL on Tencent Cloud. The database was about 11 GB.

How I ended up doing this upgrade is not a grand story. My backlog was clear that day and I had nothing to do. So I proposed upgrading the SonarQube version to my manager. He approved it. Just like that.

We were not really concerned about security issues or feature compatibility. Honestly, I just thought a newer version would have fewer vulnerabilities. I confirmed this later with a Docker Scout image scan. The old image had a lot of CVEs. The new one had fewer.

The old version, 10.6, was released on June 25, 2024. That means this SonarQube instance had not been maintained for almost two years. Everyone was probably too busy with their own tasks. I was confused because I had nothing to do that day, so I took the risk and decided to pay this tech debt myself.

The plan was simple. Clone the database, update the image tag in the Helm chart, push to GitOps, and let the migration run. Then upgrade again to the latest version.

It did not go as planned. I hit four different issues across four different upgrades and downgrades. Here is what happened.

First upgrade: sonarqube:10.6.0-community to sonarqube:24.12.0.100206-community

Database clone

I cloned the existing database to a new one. The clone command was simple:

CREATE DATABASE sonarqubedevops_24_12
WITH TEMPLATE sonarqubedevops
OWNER user_sonarqube;

It completed without errors. I updated the SonarQube JDBC config to point to the new database name. Nothing else changed.

jdbcUrl: "jdbc:postgresql://<host>/sonarqubedevops_24_12?sslmode=require&socketTimeout=1500"
jdbcUsername: "user_sonarqube"
jdbcPassword: "<redacted>"

I deployed the new pod with the sonarqube:24.12.0.100206-community image. I expected it to just work.

But here's the catch. It did not.

Error 1: "Database connection cannot be established"

I opened the setup page and clicked "Setup Migration." The page showed:

Database connection cannot be established. Please check database status and JDBC settings.

I checked everything. The JDBC URL was correct. The username and password were correct. I tested the connection from the command line and it worked. I checked pg_stat_activity and saw the SonarQube pod was connecting successfully.

But the setup page said it was not. I was stuck.

Error 2: Migration fails during ALTER TABLE

I took a different approach. I deleted the pod and PVC, deployed a fresh one, and checked the logs directly. The migration started:

Database needs to be migrated
Executing 43 DB migrations...

Then it failed at step #107004:

org.postgresql.util.PSQLException:
ERROR: must be owner of table github_perms_mapping

The SQL was equivalent to:

ALTER TABLE github_perms_mapping
RENAME TO devops_perms_mapping;

Finding the root cause

I checked table ownership in the cloned database:

SELECT schemaname, tablename, tableowner
FROM pg_tables
WHERE schemaname='public';

Almost every table was owned by root, not by user_sonarqube. I checked the original database. Same thing. The original SonarQube had been running for years with user_sonarqube connecting to tables owned by root.

The difference is that normal database operations (SELECT, INSERT, UPDATE, DELETE) only need proper privileges. But database migrations run DDL statements like ALTER TABLE and RENAME, which require table ownership. The original SonarQube never needed to run DDL, so the ownership mismatch was never exposed.

The fix

I changed ownership of all SonarQube objects from root to user_sonarqube:

DO $$DECLARE r record;
BEGIN
  FOR r IN SELECT schemaname, tablename FROM pg_tables WHERE schemaname='public' AND tableowner='root'
  LOOP
    EXECUTE format('ALTER TABLE %I.%I OWNER TO user_sonarqube;', r.schemaname, r.tablename);
  END LOOP;

  FOR r IN SELECT sequence_schema, sequence_name FROM information_schema.sequences WHERE sequence_schema='public'
  LOOP
    EXECUTE format('ALTER SEQUENCE %I.%I OWNER TO user_sonarqube;', r.sequence_schema, r.sequence_name);
  END LOOP;

  FOR r IN SELECT table_schema, table_name FROM information_schema.views WHERE table_schema='public'
  LOOP
    EXECUTE format('ALTER VIEW %I.%I OWNER TO user_sonarqube;', r.table_schema, r.table_name);
  END LOOP;

  FOR r IN SELECT n.nspname, p.proname, p.oid
    FROM pg_proc p
    JOIN pg_namespace n ON n.oid = p.pronamespace
    WHERE n.nspname = 'public'
  LOOP
    EXECUTE format('ALTER FUNCTION %I.%I(%s) OWNER TO user_sonarqube;',
      r.nspname, r.proname,
      (SELECT string_agg(format_type(t.oid, NULL), ',' ORDER BY ORDINALITY)
       FROM unnest((SELECT proargtypes FROM pg_proc WHERE oid = r.oid)) WITH ORDINALITY AS t(oid, ORDINALITY)));
  END LOOP;
END$$;

After running the script, I recreated the pod. The migration ran to completion. All 43 migrations passed.

Two observations

The database got smaller. The original was 11 GB. The new database after migration was 5.7 GB. Almost half the size. I am not sure exactly what caused it. Maybe the migration cleaned up old data, or the new version stores things more efficiently. Fun fact: this was a nice surprise I did not expect.

Elasticsearch also reindexed automatically after the migration finished. I did not need to trigger anything. It ran in the background. The UI was usable during the process, but search results might not be complete until it finishes.

Second upgrade: sonarqube:24.12.0.100206-community to sonarqube:26.1.0.118079-community

The first upgrade was successful. I decided to push further and upgrade to sonarqube:26.1.0.118079-community.

The database migration went fine. Elasticsearch reindexed itself. Everything looked good.

Then I tried to log in with SAML.

The SAML error

After the SAML redirect from Azure AD, I got this page:

You're not authorized to access this page. Please contact the administrator.

The SonarQube logs showed the real error:

WARN  org.sonar.server.authentication.AuthenticationError - Fail to callback authentication with 'saml'
org.springframework.security.saml2.provider.service.authentication.Saml2AuthenticationException:
Did not decrypt response [...] since it is not signed

The stack trace pointed to OpenSamlAuthenticationProvider. This is the Spring Security SAML2 library.

I checked the SAML configuration in the database:

sonar.auth.saml.sp.certificate.secured -> (SP encryption certificate)
sonar.auth.saml.sp.privateKey.secured  -> (SP private key)
sonar.auth.saml.signature.enabled      -> true

The Azure AD Enterprise Application had this setting:

Signing Option: Sign SAML assertion
Signing Algorithm: SHA-256

The Azure AD SAML response was encrypted using the SP certificate, but only the assertion inside the response was signed. The response envelope itself was not signed.

Root cause

In sonarqube:24.12.0.100206-community, the OpenSAML library (version 3.x) allowed decryption of encrypted responses even if the response was not signed. In sonarqube:26.1.0.118079-community, the Spring Security SAML2 library uses OpenSAML 4.x. OpenSAML 4.x enforces a stricter security policy: if the response is encrypted, it must also be signed. Otherwise, it refuses to decrypt.

This is actually documented in the SonarQube 2025.1 release upgrade notes under SAML configuration update required:

When configuring SAML on your SonarQube Server instance with assertion encryption, the response signature must be enforced. If you use SAML with Microsoft Entra, make sure you sign the response by selecting Sign SAML response or Sign SAML response and assertion as the sign-in response.

I should have read the release notes before upgrading. But honestly, I did not. I just assumed the upgrade would be smooth since the first one was fine.

What I tried

I tried removing the SP certificate and private key from the SAML configuration. My thinking was that without them, SonarQube would stop advertising encryption support, and Azure AD would send the response unencrypted.

I cleared both fields in the SonarQube UI under Administration > Authentication > SAML > Service provider section. I saved the configuration. I deleted the properties from the database directly. I restarted the pod.

The error persisted. The same "Did not decrypt response since it is not signed" error.

This probably means Azure AD caches the SP metadata and still encrypts the response. Or the encryption is configured on the Azure AD side independent of the SP metadata. I could not verify this because I am not the Azure AD admin.

The real fix

The real fix was exactly what the release notes said. I asked the Azure AD admin to change the SAML signing option.

In the Azure AD Enterprise Application, the admin went to SAML-based Sign-on > SAML Signing Certificate > Edit, and changed:

Signing Option: Sign SAML assertion

to:

Signing Option: Sign SAML response and assertion

This makes Azure AD sign the response envelope, which satisfies OpenSAML 4.x security policy.

But I could not do this myself. I needed to ask the Azure AD admin, and it was a Friday evening. The admin would not be available until Monday.

So I decided to downgrade back to sonarqube:24.12.0.100206-community so the team could work over the weekend.

Downgrade: sonarqube:26.1.0.118079-community back to sonarqube:24.12.0.100206-community

I redeployed the sonarqube:24.12.0.100206-community image using the same PVC. The pod went into CrashLoopBackOff.

The log showed:

Could not load codec 'Lucene912'.
Did you forget to add lucene-backward-codecs.jar?

The Elasticsearch data directory (/opt/sonarqube/data/es8) still contained index data created by the newer version. The older Lucene library bundled with sonarqube:24.12.0.100206-community could not read the Lucene912 codec format used by the newer version.

The fix was simple. Delete the Elasticsearch index directory and let SonarQube rebuild it:

rm -rf /opt/sonarqube/data/es8

After deleting the directory and restarting the pod, SonarQube started successfully. Elasticsearch recreated the index from the database automatically.

SAML fix: back to sonarqube:26.1.0.118079-community

On Monday, the Azure AD admin changed the signing option to "Sign SAML response and assertion." After the change, I recreated the SonarQube pod with the sonarqube:26.1.0.118079-community image. The SAML login worked immediately. The second upgrade was finally successful.

The fix was on the Azure AD side, not the SonarQube side. The SP certificate and private key needed to stay in the SonarQube SAML configuration. The only change was the Azure AD signing option.

At this point, the upgrade path was: sonarqube:10.6.0-community -> sonarqube:24.12.0.100206-community -> sonarqube:26.1.0.118079-community. Three issues, one downgrade, and a weekend of running an older version. But it got done.

Third upgrade: sonarqube:26.1.0.118079-community to sonarqube:26.7.0.124771-community

After the SAML encryption issue was resolved and sonarqube:26.1.0.118079-community was running smoothly, I decided to upgrade again to sonarqube:26.7.0.124771-community.

I deployed the new image. The pod went into Running state. But the ingress at the SonarQube URL returned 503 Service Temporarily Unavailable.

I checked the pod events:

Readiness probe failed: sh: 5: wget: not found
Liveness probe failed: sh: 2: wget: not found

The Helm chart's liveness and readiness probes were using wget for health checks. The new sonarqube:26.7.0.124771-community image no longer includes wget.

The startup probe used httpGet (kubelet-native) and was not affected. That is why the pod appeared Running even though the probes were failing. But because the readiness probe never passed, the Service had no ready endpoints, and the ingress could not route traffic to the pod.

The fix

I replaced wget with curl in the probe commands across the Helm chart templates.

In templates/sonarqube-sts.yaml and templates/deployment.yaml:

# Liveness probe
- wget --no-proxy --quiet -O /dev/null --timeout=... --header=... "...api/system/liveness"
+ curl -s -f --max-time ... --header ... "...api/system/liveness" > /dev/null

# Readiness probe
- wget --no-proxy -qO- ...api/system/status | grep -q -e '"status":"UP"' ...
+ curl -s --max-time ... "...api/system/status" | grep -q -e '"status":"UP"' ...

After applying the changes and syncing via GitOps, the probes started passing, the Service had ready endpoints, and the ingress returned 200.

This was the simplest issue so far. No database changes, no Azure AD changes, no Elasticsearch crashes. Just a missing binary in the container image. So yeah, that was the fourth issue.

What I learned

The real lesson is that this SonarQube instance sat untouched for almost two years. Nobody noticed. Nobody cared. Until one day I had a clear backlog and nothing to do. I hope someone in the future does not let it drift that long again. Maintaining versions is boring. But paying two years of tech debt over two weeks is worse.