GitLab Backup to Tencent COS After EC2 Migration

We recently moved our self-hosted GitLab instance from AWS EC2 to a Tencent Cloud CVM using go2tencentcloud, Tencent's lift-and-shift migration tool [1]. The migration itself went fine. The backup upload config, though, did not survive the move. Here is what broke and how I fixed it.

Moving from EC2 to CVM

On EC2, GitLab uploaded backups to an S3 bucket. We used an IAM role attached to the instance, so there were no access keys to manage. The config in gitlab.rb looked like this:

gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'ap-southeast-1',
  'use_iam_profile' => true
}
gitlab_rails['backup_upload_remote_directory'] = '<s3-bucket-name>'

Simple. No credentials in config files, no rotation headaches. The EC2 metadata service handled everything.

After moving to CVM, this stopped working. Not because the config was wrong, but because there is no EC2 metadata service on a CVM. Tencent Cloud has its own instance metadata service, but COS (Cloud Object Storage) does not support IAM roles the way S3 does in this context. I had to switch to access key and secret key authentication.

First try with COS credentials

I created a COS bucket, generated an access key pair, and updated the config:

gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'auto',
  'aws_access_key_id' => '<redacted>',
  'aws_secret_access_key' => '<redacted>',
  'endpoint' => 'https://<bucket-name>.cos.<region>.myqcloud.com',
  'use_iam_profile' => false
}
gitlab_rails['backup_upload_remote_directory'] = '<bucket-name>'

COS exposes an S3-compatible API, so provider => 'AWS' is correct. GitLab uses the fog-aws gem under the hood for S3 uploads, and fog-aws can talk to any S3-compatible endpoint.

I ran gitlab-ctl reconfigure and tried a backup. It failed.

The error was around AWS SDK Ruby's Expect: 100-continue handling. Something about "1xx responses not supported." The backup uploader was choking on the HTTP interaction between GitLab's patched Net::HTTP and the COS endpoint.

Testing the connection in isolation

Before digging into GitLab internals, I wanted to confirm the credentials and endpoint actually worked. I used GitLab's embedded Rails runner to create an AWS SDK client directly and test a PutObject:

sudo gitlab-rails runner '
c = Gitlab.config.backup.upload.connection.to_h

s3 = Aws::S3::Client.new(
  region: c["region"],
  endpoint: c["endpoint"],
  access_key_id: c["aws_access_key_id"],
  secret_access_key: c["aws_secret_access_key"],
  force_path_style: c["path_style"].to_s == "true",
  http_continue_timeout: 0
)

s3.put_object(
  bucket: Gitlab.config.backup.upload.remote_directory,
  key: "gitlab-cos-connection-test.txt",
  body: "GitLab COS connection test"
)

puts "SUCCESS: COS upload works"
'

It worked. The object showed up in the COS bucket. So credentials, endpoint, and permissions were all fine. The problem was somewhere in how GitLab's native backup uploader constructs the HTTP requests.

The http_continue_timeout: 0 in that test disables the Expect: 100-continue header in the AWS SDK. But this is an SDK client option, not something you can set in gitlab.rb.

Two settings that made it work

While searching for solutions, I found Ajay Furber's post about getting GitLab backups to work with Cloudflare R2 [2]. Same problem, different S3-compatible provider. The fix came down to two settings in the fog-aws configuration:

path_style: S3-compatible services often use path-style addressing (bucket-name.endpoint/bucket-name/key) instead of virtual-hosted style (bucket-name.endpoint/key). Some services don't support virtual-hosted style at all. Setting this to true forces path-style requests.

enable_signature_v4_streaming: By default, fog-aws sets this to true. When enabled, it uses aws-chunked content encoding with individually signed chunks. From what I have seen, S3-compatible services like Cloudflare R2 and Tencent COS do not implement this. Setting it to false disables chunked transfer and uses a single signed request instead.

Neither setting was obvious from GitLab's own documentation. The path_style option is at least mentioned in some places, but enable_signature_v4_streaming is not. I only learned about it from Ajay's post, which references a fog-aws PR where it was added.

So the final config looks like this:

gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'auto',
  'aws_access_key_id' => '<redacted>',
  'aws_secret_access_key' => '<redacted>',
  'endpoint' => 'https://<bucket-name>.cos.<region>.myqcloud.com',
  'path_style' => 'true',
  'enable_signature_v4_streaming' => 'false',
  'use_iam_profile' => false
}
gitlab_rails['backup_upload_remote_directory'] = '<bucket-name>'

Note that 'true' and 'false' are strings here, not booleans. That is how GitLab Omnibus passes them through to fog-aws.

After updating gitlab.rb, I ran sudo gitlab-ctl reconfigure to apply the changes.

Backup worked

Then I ran sudo gitlab-backup create. The backup completed and the archive was uploaded directly to COS. No additional configuration was needed beyond the two settings above.

The Expect: 100-continue error turned out to be a symptom, not the root cause. The real issue was that COS does not support v4 chunked streaming encoding, which fog-aws enables by default. Setting enable_signature_v4_streaming to false fixed the HTTP interaction entirely, and the http_continue_timeout: 0 SDK workaround was not needed for the native backup uploader.

Closing

The fix was two settings in gitlab.rb that are not mentioned in GitLab's own backup documentation. I found the solution through Ajay's writeup about Cloudflare R2 [2], which is essentially the same problem with a different S3-compatible provider. If you are using any S3-compatible object storage that is not AWS S3 itself, path_style and enable_signature_v4_streaming are probably the first things to check.

References

[1] Tencent Cloud, "Online Migration Overview," Tencent Cloud Documentation, Aug. 13, 2026. [Online]. Available: https://www.tencentcloud.com/document/product/213/35639. [Accessed: Sep. 12, 2026].

[2] A. Furber, "How to get fog-aws / Gitlab Backups to work with Cloudflare R2," ajf.me.uk, Jul. 6, 2024. [Online]. Available: https://www.ajf.me.uk/how-to-get-fog-aws-gitlab-backups-to-work-with-cloudflare-r2/. [Accessed: Sep. 10, 2026].