SSH Failing on Tencent Cloud CLB After GitLab Migration

We had just finished migrating a GitLab instance from AWS to Tencent Cloud. The server was up, the web UI loaded fine over HTTPS, and from the internal jumphost I could SSH into the CVM and clone repos without issues. Everything looked good.

Then I tried cloning a repo from my local machine using SSH. Nothing. Connection refused.

I ran a quick netcat check against the public CLB:

nc -v <public-clb-ip> 22
# connect to <public-clb-ip> port 22 (tcp) failed: Connection refused

Port 443 and 80 were fine. Port 22 was dead. The web UI worked, but git clone git@<gitlab.example.com>:org/repo.git was completely broken.

Ruling out the server

The first thing I did was rule out the server itself. I jumped onto the internal jumphost and tested SSH directly to the CVM's private IP. It worked fine, so the GitLab service and SSH daemon on the CVM were running correctly.

nc -zv <cvm-private-ip> 22
# Connection to <cvm-private-ip> 22 port [tcp/ssh] succeeded!

The problem was somewhere between the public internet and the CVM. That narrowed it down to the load balancers.

The architecture I was working with

Before I explain the fix, here is the setup. We run a multi-account Tencent Cloud environment with a dedicated network account that owns the public-facing resources. The devops account holds the internal workloads. The two accounts talk to each other through a CCN (Cloud Connect Network).

The flow for a user hitting the GitLab server looks like this:

So for HTTPS traffic, the path is: user hits the public CLB on port 443, TLS is terminated there with an SSL certificate, and the request is forwarded as plain HTTP to the internal CLB on port 80. The internal CLB then routes it to the CVM based on the host header.

For SSH, the path is the same structure but simpler: the public CLB receives TCP on port 22, forwards it to the internal CLB on port 22, which forwards it to the CVM on port 22. No TLS to worry about, just a raw TCP pass-through.

The first rehearsal was wrong

This was actually the second rehearsal of this migration. In the first one, I made a mistake that cost me some time.

I had created the CVM in a public subnet with a public IP, and I tested SSH by connecting directly to that public IP. It worked, so I thought the setup was correct. But the real architecture has the CVM behind a CLB in a private subnet, so my test was not testing the actual path users would take.

Looking back, this was a waste of time. I was testing a configuration that did not match what we would ship. The second rehearsal, I made sure the CVM was in a private subnet and all traffic went through the CLBs.

The missing listener

So here is what happened. The existing public CLB, which we used for other devops services, only had HTTP (80) and HTTPS (443) listeners configured. No TCP listener on port 22.

When I tried to SSH, the traffic hit the public CLB, found no listener on port 22, and dropped the connection. The internal CLB and the CVM were both fine. The problem was that the public CLB simply did not know what to do with TCP traffic on port 22.

The fix was to create a dedicated public CLB just for the GitLab server, with a TCP listener on port 22 added alongside the existing HTTP and HTTPS listeners. This way, SSH traffic would be forwarded to the internal CLB, which would then pass it to the CVM.

Here is the traffic flow after the fix:

HTTPS (web access and git clone over HTTPS):

SSH (git clone and push):

The key difference is that the new public CLB now has a TCP listener on port 22. Without it, the connection never reaches the internal CLB, let alone the CVM.

Validating the fix

After setting up the new CLB with the TCP listener, I ran connectivity tests from multiple points to make sure everything was wired correctly.

Port-by-port connectivity

First, I tested the old CLB to confirm the original problem:

for port in 22 80 443; do
  nc -zv <old-clb-hostname> $port
done

Port 22 timed out as expected. Ports 80 and 443 succeeded. This CLB was never meant to handle SSH, and it was not.

Then I tested the new dedicated CLB:

for port in 22 80 443; do
  nc -zv <new-clb-hostname> $port
done

All three ports succeeded. Port 22 was now being forwarded through the internal CLB to the CVM.

I also tested the CVM directly from the jumphost to confirm the baseline:

for port in 22 80 443; do
  nc -zv <cvm-private-ip> $port
done

Port 22 and 80 succeeded. Port 443 was refused, which is expected. The CVM serves GitLab over plain HTTP on port 80. TLS termination happens at the public CLB, so the CVM itself does not listen on 443.

End-to-end test from my machine

To test the full path without waiting for DNS propagation, I added an entry to /etc/hosts pointing the GitLab domain to the new public CLB's IP. I got the IP from a DNS lookup on the CLB hostname:

dig +short <new-clb-hostname>
# -> <public-clb-ip>

Then edited /etc/hosts:

<public-clb-ip>   <gitlab.example.com>

I ran a ping first. It showed 100% packet loss, but that is expected. ICMP is blocked by the security group and does not affect anything.

Then I tested HTTPS with curl:

curl -v https://<gitlab.example.com>

The TLS handshake succeeded (TLSv1.2, the certificate CN matched the domain and was valid). I got a 302 Found redirect to /users/sign_in. The server header was nginx, and GitLab-specific headers like X-Gitlab-Meta and X-Request-Id were present. The web path was working.

Finally, SSH:

ssh -T git@<gitlab.example.com>
# -> Welcome to GitLab, @<gitlab-username>!

The host key had changed from the old server, so I had to run ssh-keygen -R <gitlab.example.com> first. After that, the connection went through.

To confirm the data was actually coming from the migrated server and not the old production instance, I cloned a repo and checked git log:

git clone git@<gitlab.example.com>:org/repo.git
# -> Cloning into 'repo'... done (1781 objects, 563 KiB)

The latest commit was from July 16, while the production EC2 instance had commits up to July 19. That three-day gap confirmed the /etc/hosts override was routing to the migrated Tencent CVM with slightly stale data, not the production EC2 instance. The clone was hitting the right target.

Closing

The root cause was simple: the CLB lacked a TCP listener on port 22. But the real mistake was rushing into the first rehearsal without thinking through the architecture.

I tested the CVM directly with its public IP instead of testing through the CLB. That test passed, but it was testing the wrong thing. It gave me false confidence and wasted a whole rehearsal cycle. If I had mapped out the actual traffic path first, I would have caught the missing listener before the first test.