> ## Content Index > Fetch the complete content index at: https://anantafatur.dev/llms.txt > Use this file to discover other available public pages before exploring further. # GitLab Sidekiq Failing After Redis Migration to Tencent Cloud - URL: https://anantafatur.dev/gitlab-sidekiq-tencent-redis-resp3/ - Published: 2026-09-05T12:55:57.000Z - Updated: 2026-09-05T12:55:57.000Z - Description: A walkthrough of debugging GitLab Sidekiq crashing silently after migrating Redis from AWS ElastiCache to Tencent Cloud, caused by Tencent's proxy not supporting the RESP3 protocol required by GitLab's redis-client gem. - Author: Ananta We were migrating our company's GitLab instance from AWS to Tencent Cloud. Since Redis does not hold persistent data in a typical GitLab setup, we decided to migrate it first as a low-risk warm-up before touching the main instance. That turned out to be wrong. ## The environment GitLab was running on an EC2 instance on AWS. Redis was on AWS ElastiCache, running Redis OSS 7.0.7\. The two lived in the same VPC. Everything was working fine before we touched anything. The plan was to repoint GitLab's Redis to a new Tencent Cloud Redis instance hosted at `:6379`. The Tencent instance was in a different VPC and a different cloud entirely, connected over a cross-cloud network link. We provisioned it via Terraform using the `tencentcloud_redis_instance` resource with `type_id = 17`. In Tencent's Terraform provider, `type_id` is an integer that maps to a specific Redis engine version and architecture. `17` means Redis 7.0 Memory Edition, Standard Architecture \[1\]. The full list of values is in the [Tencent provider docs](https://registry.terraform.io/providers/tencentcloudstack/tencentcloud/latest/docs/resources/redis%5Finstance#type%5Fid-1) \[1\], but the ones you will most likely encounter are: - `8`: Redis 5.0, standard architecture - `15`: Redis 6.2, standard architecture - `16`: Redis 6.2, cluster architecture - `17`: Redis 7.0, standard architecture - `18`: Redis 7.0, cluster architecture The number tells you the engine version and whether it runs in standard (single primary with replicas) or cluster (sharded) mode. We picked `17` because we wanted to match the source as closely as possible. ElastiCache was on Redis OSS 7.0.7, so Redis 7.0 on Tencent was the nearest available option. Same major version, same standard architecture, one shard, one replica. --- ## The symptom After repointing and restarting GitLab, everything seemed fine at first. Login worked. Repo access worked. Push and pull worked. But when I opened the GitLab admin panel and went to `/admin/background_jobs`, I got a 500 error. Just the generic "We're sorry, something went wrong on our end." Sidekiq, the background job processor, was crashing on startup. ## Ruling out the obvious My first thought was network. GitLab is on AWS, Redis is on Tencent Cloud. Different VPCs, different clouds. That's a lot of surface area for a firewall rule to go wrong. So I tested connectivity from the GitLab EC2 instance directly: ```bash telnet 6379 ``` It connected. Port was open, no firewall blocking it. So it was not a network issue. ## Finding the root cause I tailed the Sidekiq logs and grepped for the error: ```bash sudo grep -B 50 "requires Redis 6+" /var/log/gitlab/sidekiq/current ``` This is what I found: ``` {"severity":"INFO","time":"2026-06-03T16:30:36.085Z","message":"Sidekiq 7.3.9 connecting to Redis with options {:size=>25, :pool_name=>"internal", :host=>"", :port=>6379, :ssl=>false, :password=>"REDACTED",...}"} redis-client requires Redis 6+ with HELLO command available (redis://:6379) ``` The error is clear. GitLab's `redis-client` gem sends the `HELLO` command when it connects \[2\]. `HELLO` was introduced in Redis 6.0.0 as part of the RESP3 protocol \[3\]. If Redis does not respond to it, the gem refuses to connect \[2\]. To confirm, I tested directly from the EC2 instance: ```bash redis-cli -h -p 6379 -a '' HELLO 3 ``` ``` (error) ERR unknown command 'HELLO' ``` And for comparison, against the old ElastiCache endpoint: ```bash redis-cli -h .cache.amazonaws.com HELLO 3 ``` ``` Error: Protocol error, got "%" as reply type byte ``` Both failed the `HELLO 3` test differently, but GitLab was working fine with ElastiCache before. That confused me for a moment. The difference is that ElastiCache does support enough of the protocol for GitLab's gem to negotiate correctly at the application level. The CLI error there is a client-side quirk, not a hard rejection. Tencent's Redis returns `ERR unknown command 'HELLO'`, which is a hard failure that kills the connection before Sidekiq can even initialize. ## Why Tencent Redis rejects HELLO The actual Redis version on Tencent is 7.1.4, which absolutely supports RESP3 and the `HELLO` command \[3\]. But GitLab does not connect to Redis directly. The connection goes through Tencent's distributed cache proxy layer first \[4\]: ``` GitLab | Tencent Proxy (5.8.25) | Redis 7.1.4 ``` That proxy does not support RESP3\. I confirmed this with the Tencent team the next day. Their exact response: > Tencent Cloud Redis Currently Does Not Support the RESP3 Protocol. The Proxy of Tencent Cloud's distributed cache database does not currently support the RESP3 protocol; executing HELLO 3 will return an error. For both Standard and Cluster editions, the HELLO command currently only supports HELLO 2, and it can only be executed successfully after the user has performed AUTH authentication. They also confirmed this applies across all versions on their platform, including Redis 5.0, 6.2, 7.0, and even Valkey 8.0\. The limitation is in the proxy, not the Redis version itself. The only potential workaround they mentioned was a "Direct Connection Mode" that bypasses the proxy. But as of when we hit this, it was still in canary. They said it would be generally available sometime in August. On top of that, we checked whether Valkey edition could be provisioned through Terraform as an alternative. It cannot, at least not through the official Tencent provider at the time of writing. The `tencentcloud_redis_instance` resource does not have Valkey-specific options yet \[1\]. ## The fix We rolled back. I repointed GitLab's Redis config back to AWS ElastiCache, restarted GitLab, and the background jobs page came back to normal immediately. ```bash sudo gitlab-ctl restart ``` That was the end of it for that environment. ## What happened after the full migration When we eventually migrated the entire GitLab instance to Tencent Cloud, we switched to GitLab's built-in Redis instead. That means Redis runs on the same VM as GitLab itself \[5\]. To enable it, I commented out the external Redis host config in `/etc/gitlab/gitlab.rb` and set: ```ruby redis['enable'] = true ``` Then ran `gitlab-ctl reconfigure`. The trade-off is that built-in Redis consumes RAM on the GitLab VM. I did not do a controlled comparison because the AWS and Tencent environments have different instance types and virtualization layers, so it would not be a fair benchmark anyway. What I can say is that it works, and Sidekiq is healthy. --- ## Looking back The real mistake was not testing Redis compatibility before the migration. We assumed that because both ElastiCache and Tencent Redis were "Redis 7," they would behave the same. They do not. The version of the backend engine matters less than what sits in front of it. The `HELLO` command compatibility check would have been a two-minute test before any change. I did not do it. That is on me. If you are planning to migrate GitLab to any managed Redis service, run this from your GitLab host before you repoint anything: ```bash redis-cli -h -p 6379 -a '' HELLO 2 ``` If you get back a map response with server info, you are probably fine. If you get `ERR unknown command`, stop and do not proceed. ## References \[1\] TencentCloud, "tencentcloud\_redis\_instance," Terraform Registry. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[2\] redis-rb, "redis-client: Simple low-level client for Redis 6+," GitHub. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[3\] Redis Ltd., "HELLO," Redis Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[4\] Tencent Cloud, "Proxy Features," Tencent Cloud Distributed Cache Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[5\] GitLab, "Configuring Redis for scaling," GitLab Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\].