# Muhammad Ananta Faturrahman - DevOps Engineer > Hi, I’m Ananta! DevOps enthusiast who learns by building and breaking things in my homelab so your production doesn’t have to 👻 Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages _No public content available._ ## Posts ### When Docmost says "Error uploading file to drive" URL: https://anantafatur.dev/docmost-error-uploading-file-to-drive/ Last updated: 2026-09-18T19:00:55.000Z I was trying to upload a file attachment to a Docmost page and got hit with a generic error: "Error uploading file to drive." No stack trace, no permissions hint, no file path. Just a plain 400 Bad Request. The UI was equally unhelpful, showing the same message without any clue about what went wrong. This post walks through how I traced the real cause and how I fixed it permanently. ## The symptoms A file upload to a Docmost page returned a 400 error. Here is the full log: ``` "context":"AttachmentService","msg":"Error uploading file to drive:" "context":"AttachmentController","err":{"type":"BadRequestException","message":"Error uploading file to drive",...} ``` Two things stood out immediately. First, the `AttachmentService` log ends with a colon and nothing after it. That suggests the error object was logged but its string representation was empty. Second, the controller throws a `BadRequestException` with the same static message, which means whatever the real error was, it got replaced with a canned string. The error is clearly being swallowed. Somewhere between the storage driver throwing an exception and the controller returning a response, the real error message is lost. --- ## The environment I run Docmost in Docker using `docker compose`. The setup is simple: a Postgres database, a Redis instance, and the Docmost app itself. File storage uses the default `local` driver, meaning uploaded files go to the container filesystem at `/app/data/storage`. That path is mounted as a bind mount from a directory on the host. Docmost version is 0.96.0, pulled from the official Docker image. The container runs as user `node` (uid 1000), which is the default in the upstream image. I have a custom Dockerfile on top of the official image that applies a few npm package patches. The `docker-compose.yml` had the build section commented out, so it was using the vanilla image directly. ## Tracing the error I grabbed the error log and sent it to DeepSeek V4 Flash. I am not a Node.js developer and I did not want to spend time reading compiled JavaScript inside a container. The AI had the full picture from the log alone. It started by reading the compiled source from inside the running container. This is what it found in the `uploadToDrive` method inside `attachment.service.js`: ```javascript async uploadToDrive(filePath, fileContent) { try { await this.storageService.upload(filePath, fileContent); } catch (err) { this.logger.error('Error uploading file to drive:', err); throw new BadRequestException('Error uploading file to drive'); } } ``` The catch block logs `err` directly. If `err` is not a standard Error object, the NestJS Logger might serialize it as an empty string, which explains the colon with nothing after it in the log. Then it throws a new `BadRequestException` with a hardcoded message, completely burying the real cause. The AI went deeper. It checked the `storage.service.js` file and found that `upload` delegates to a storage driver. Since no `STORAGE_DRIVER` env var was set, it defaults to `local`. The local driver's upload method looked like this: ```javascript async upload(filePath, file) { try { const fullPath = this._fullPath(filePath); if (file instanceof Buffer) { await fs.outputFile(fullPath, file); } else { await fs.mkdir(dirname(fullPath), { recursive: true }); await pipeline(file, createWriteStream(fullPath)); } } catch (err) { throw new Error(`Failed to upload file: ${err.message}`); } } ``` For file uploads, the code creates the directory with `mkdir` and then pipes the stream to a write stream. The chain was clear: `mkdir` fails -> local driver wraps the error -> `storageService.upload` propagates it -> `uploadToDrive` logs and re-throws -> controller returns 400. That pointed to a filesystem permission problem. The AI ran a quick check inside the container: ```bash $ docker exec docmost-app ls -la /app/data/storage/ total 12 drwxr-xr-x 3 root root 4096 Apr 30 04:08 . drwxr-xr-x 3 node node 4096 Sep 8 15:49 .. drwxr-xr-x 3 root root 4096 Apr 30 04:08 ``` The storage directory and its subdirectories were owned by `root` with permissions `755`. The app runs as `node` (uid 1000), so it could read and traverse directories but could not create new files or directories inside them. The `mkdir` call was hitting `EACCES`. ## Why were the permissions wrong? The container had been running with a different configuration at some point -- perhaps before I added `USER node` to the custom Dockerfile, or perhaps an earlier version of the upstream image ran as root. When files were first written to the bind-mounted volume, they were created with root ownership. After switching to the `node` user, the app could no longer write to those directories. Fun fact: existing files from earlier uploads were still readable because the `r-x` permission on directories was enough for reading. Only new uploads broke. This also explains why the error was hard to debug. The app had been working fine for previous uploads. The breakage only showed up on new file writes, and the error message gave no hint about permissions. --- ## The fix The AI proposed both an immediate fix and a permanent one. ### Immediate fix Change the ownership of the storage directory to match the user the app runs as: ```bash docker exec -u root docmost-app chown -R node:node /app/data/storage ``` This runs `chown` as root inside the container and changes ownership of the entire storage tree to `node:node`. I ran this and uploads started working again. But this is a manual fix. If the container is recreated or if the volume is mounted on a new host, the permissions would be wrong again. ### Permanent fix The AI created a custom entrypoint script that runs `chown` at container startup before dropping privileges and running the application. Here is the entrypoint script (`entrypoint.sh`): ```bash #!/bin/sh set -e # Ensure the storage directory is writable by the node user chown -R node:node /app/data/storage # Drop privileges to node user and run the original entrypoint exec su -s /bin/sh -c "exec /docker-entrypoint.sh $*" node ``` The script runs as root because `USER node` was removed from the end of the Dockerfile. It fixes the permissions, then uses `su` to switch to `node` and execute the original entrypoint that ships with the Docmost image. The AI updated the Dockerfile to copy this script and set it as the entrypoint: ```dockerfile COPY entrypoint.sh /entrypoint-wrapper.sh RUN chmod +x /entrypoint-wrapper.sh ENTRYPOINT ["/entrypoint-wrapper.sh"] CMD ["pnpm", "start"] ``` It also uncommented the `build` section in `docker-compose.yml` so the container builds from the custom Dockerfile instead of pulling the vanilla image. After rebuilding and restarting, the permissions are now fixed automatically on every container start. I did not have to think about any of this – the AI handled the code changes, I just ran the rebuild command. ### go2tencentcloud Blocked by Security Group DROP Rules URL: https://anantafatur.dev/go2tencentcloud-blocked-security-group-drop/ Last updated: 2026-09-18T05:02:16.000Z ## Setting up a green environment We run GitLab on a Tencent Cloud CVM instance. Besides hosting repositories, this GitLab also stores Spring Cloud Config (SCC) files that are consumed by production applications. Messing up an upgrade here would break config resolution for a lot of services, not just Git access. I needed to upgrade GitLab to a newer version, and I wanted to do it with a blue-green approach. Spin up a green instance with the new version, migrate the data over, test everything, then cut over. If something goes wrong, the blue instance is still running untouched. The plan was simple. Spin up a new CVM with the same disk layout, same OS, same everything, and keep it completely isolated from the production instance. No traffic should flow between blue and green unless we explicitly allow it. So I did what any reasonable person would do. I configured the security groups to DROP all traffic between the two instances. Here is what the security group rules looked like. The production instance, call it `gitlab`, had this egress rule: ```hcl { action = "DROP" protocol = "ALL" port = "ALL" cidr_block = "/32" rule_description = "gitlab-green - Tencent CVM" } ``` And the green instance, `gitlab-green`, had the mirror ingress rule: ```hcl { action = "DROP" protocol = "ALL" port = "ALL" cidr_block = "/32" rule_description = "gitlab - Tencent CVM" } ``` Clean, simple, and completely isolated. Or so I thought. --- ## Attempting the migration To clone the production instance to the green one, I used go2tencentcloud. If you are not familiar with it, go2tencentcloud is Tencent Cloud's online migration tool. It migrates systems and applications from a source server directly to a destination CVM without needing to create or upload images \[1\]. You run a client on the source server, the tool connects to the destination, and it copies everything over. The destination was a fresh CVM with a bare OS. Same system disk size, same data disk size, but no pre-built image. Just an empty instance waiting for the migration to land. I kicked off the migration and waited. ## Error one: host-to-host connection Almost immediately, the migration failed. ``` Error code: ERROR_HOST_TO_HOST_CONNECTION Failure cause: The migration source cannot connect to the target instance. Please verify network connectivity between the source and target, and ensure port 80 is open in the target security group's inbound rules. ``` Right. The security group rules I was so proud of were blocking the migration tool. The tool tries to connect from the source to the target, and my DROP rules said no. Fair enough. The error message specifically said port 80\. The documentation also says the destination needs port 80, 443, and 3389 open \[2\]. But the error only mentioned port 80, so I followed the advice. I added a temporary ACCEPT rule for TCP port 80 from the production IP on the green instance's ingress, and a matching ACCEPT rule for TCP port 80 to the green IP on the production instance's egress. Both placed before the DROP rules. Here is what the green instance's ingress looked like after the fix: ```hcl ingress_rules = [ { # TEMPORARY: Allow port 80 from production for migration action = "ACCEPT" protocol = "TCP" port = "80" cidr_block = "/32" rule_description = "TEMP - gitlab - Allow migration port 80" }, { action = "DROP" protocol = "ALL" port = "ALL" cidr_block = "/32" rule_description = "gitlab - Tencent CVM" }, # ... other rules ] ``` I restarted the migration. ## Error two: data transmission It failed again. ``` Error code: ERROR_DATA_TRANSMISSION Failure cause: Data transmission failed. Please check the network connectivity between the source machine and the target instance, or contact customer service. ``` This one was frustrating. The host-to-host connection error was gone, which meant port 80 was open and the initial handshake worked. But the actual data transfer was failing. The error message was vague, no mention of which port or protocol was failing. The guide linked from the error was not helpful either. At this point I realized my mistake. I had no idea what ports go2tencentcloud actually uses during the data transfer phase. I only knew it needed port 80 for the initial connection because the first error told me so. The documentation mentions ports 80, 443, and 3389 \[2\], but the error message only called out port 80\. That mismatch between the docs and the error message tripped me up. I decided to stop guessing and just open everything. I changed both temporary rules from TCP port 80 to ALL protocols and ALL ports. ```hcl ingress_rules = [ { # TEMPORARY: Allow all ports from production for migration action = "ACCEPT" protocol = "ALL" port = "ALL" cidr_block = "/32" rule_description = "TEMP - gitlab - Allow migration" }, { action = "DROP" protocol = "ALL" port = "ALL" cidr_block = "/32" rule_description = "gitlab - Tencent CVM" }, # ... other rules ] ``` Same change on the production instance's egress side. ALL protocols, ALL ports, both directions. I restarted the migration one more time. It worked. The migration completed without issues. --- ## Closing The root cause was simple. I configured complete network isolation between the two instances with DROP rules, then tried to run a tool that needs direct connectivity between them. The DROP rules were doing exactly what I asked them to do. The interesting part is why the first fix, opening only port 80, was not enough. In Tencent Cloud security groups, rules are evaluated in order from top to bottom. The first rule that matches the traffic wins \[3\]. My DROP rules were the first ones in the list, and they matched everything. When I added the ACCEPT rule for port 80, it went before the DROP rule, so port 80 traffic was allowed. The initial connection worked, but the subsequent data transfer phase apparently uses additional ports or protocols beyond what the error message suggested. I am not sure exactly which ports go2tencentcloud uses during data transfer. The tool uses rsync-like behavior under the hood based on what I saw, but I did not dig into it further. The important thing is that trusting only the error message was the wrong call. The documentation listed three ports (80, 443, 3389) \[2\], and the actual tool behavior seems to need more than what the error described. ## References \[1\] Tencent Cloud, "Online Migration Overview," Cloud Virtual Machine documentation, 2026\. \[Online\]. Available: . \[Accessed: Sep. 18, 2026\]. \[2\] Tencent Cloud, "Server Migration via a Migration Tool," Cloud Virtual Machine documentation, 2026\. \[Online\]. Available: . \[Accessed: Sep. 18, 2026\]. \[3\] Tencent Cloud, "Security Group Overview," Cloud Virtual Machine documentation, 2025\. \[Online\]. Available: . \[Accessed: Sep. 18, 2026\]. ### GitLab Backup to Tencent COS After EC2 Migration URL: https://anantafatur.dev/gitlab-backup-tencent-cos-cvm/ Last updated: 2026-09-11T21:42:13.000Z We recently moved our self-hosted GitLab instance from AWS EC2 to a Tencent Cloud CVM using go2tencentcloud, Tencent's lift-and-shift migration tool \[1\]. The migration itself went fine. The backup upload config, though, did not survive the move. Here is what broke and how I fixed it. ## Moving from EC2 to CVM On EC2, GitLab uploaded backups to an S3 bucket. We used an IAM role attached to the instance, so there were no access keys to manage. The config in `gitlab.rb` looked like this: ```ruby gitlab_rails['backup_upload_connection'] = { 'provider' => 'AWS', 'region' => 'ap-southeast-1', 'use_iam_profile' => true } gitlab_rails['backup_upload_remote_directory'] = '' ``` Simple. No credentials in config files, no rotation headaches. The EC2 metadata service handled everything. After moving to CVM, this stopped working. Not because the config was wrong, but because there is no EC2 metadata service on a CVM. Tencent Cloud has its own instance metadata service, but COS (Cloud Object Storage) does not support IAM roles the way S3 does in this context. I had to switch to access key and secret key authentication. --- ## First try with COS credentials I created a COS bucket, generated an access key pair, and updated the config: ```ruby gitlab_rails['backup_upload_connection'] = { 'provider' => 'AWS', 'region' => 'auto', 'aws_access_key_id' => '', 'aws_secret_access_key' => '', 'endpoint' => 'https://.cos..myqcloud.com', 'use_iam_profile' => false } gitlab_rails['backup_upload_remote_directory'] = '' ``` COS exposes an S3-compatible API, so `provider => 'AWS'` is correct. GitLab uses the `fog-aws` gem under the hood for S3 uploads, and fog-aws can talk to any S3-compatible endpoint. I ran `gitlab-ctl reconfigure` and tried a backup. It failed. The error was around AWS SDK Ruby's `Expect: 100-continue` handling. Something about "1xx responses not supported." The backup uploader was choking on the HTTP interaction between GitLab's patched Net::HTTP and the COS endpoint. ## Testing the connection in isolation Before digging into GitLab internals, I wanted to confirm the credentials and endpoint actually worked. I used GitLab's embedded Rails runner to create an AWS SDK client directly and test a `PutObject`: ```ruby sudo gitlab-rails runner ' c = Gitlab.config.backup.upload.connection.to_h s3 = Aws::S3::Client.new( region: c["region"], endpoint: c["endpoint"], access_key_id: c["aws_access_key_id"], secret_access_key: c["aws_secret_access_key"], force_path_style: c["path_style"].to_s == "true", http_continue_timeout: 0 ) s3.put_object( bucket: Gitlab.config.backup.upload.remote_directory, key: "gitlab-cos-connection-test.txt", body: "GitLab COS connection test" ) puts "SUCCESS: COS upload works" ' ``` It worked. The object showed up in the COS bucket. So credentials, endpoint, and permissions were all fine. The problem was somewhere in how GitLab's native backup uploader constructs the HTTP requests. The `http_continue_timeout: 0` in that test disables the `Expect: 100-continue` header in the AWS SDK. But this is an SDK client option, not something you can set in `gitlab.rb`. ## Two settings that made it work While searching for solutions, I found Ajay Furber's post about getting GitLab backups to work with Cloudflare R2 \[2\]. Same problem, different S3-compatible provider. The fix came down to two settings in the fog-aws configuration: `**path_style**`: S3-compatible services often use path-style addressing (`bucket-name.endpoint/bucket-name/key`) instead of virtual-hosted style (`bucket-name.endpoint/key`). Some services don't support virtual-hosted style at all. Setting this to `true` forces path-style requests. `**enable_signature_v4_streaming**`: By default, fog-aws sets this to `true`. When enabled, it uses `aws-chunked` content encoding with individually signed chunks. From what I have seen, S3-compatible services like Cloudflare R2 and Tencent COS do not implement this. Setting it to `false` disables chunked transfer and uses a single signed request instead. Neither setting was obvious from GitLab's own documentation. The `path_style` option is at least mentioned in some places, but `enable_signature_v4_streaming` is not. I only learned about it from Ajay's post, which references a fog-aws PR where it was added. So the final config looks like this: ```ruby gitlab_rails['backup_upload_connection'] = { 'provider' => 'AWS', 'region' => 'auto', 'aws_access_key_id' => '', 'aws_secret_access_key' => '', 'endpoint' => 'https://.cos..myqcloud.com', 'path_style' => 'true', 'enable_signature_v4_streaming' => 'false', 'use_iam_profile' => false } gitlab_rails['backup_upload_remote_directory'] = '' ``` Note that `'true'` and `'false'` are strings here, not booleans. That is how GitLab Omnibus passes them through to fog-aws. After updating `gitlab.rb`, I ran `sudo gitlab-ctl reconfigure` to apply the changes. ## Backup worked Then I ran `sudo gitlab-backup create`. The backup completed and the archive was uploaded directly to COS. No additional configuration was needed beyond the two settings above. The `Expect: 100-continue` error turned out to be a symptom, not the root cause. The real issue was that COS does not support v4 chunked streaming encoding, which fog-aws enables by default. Setting `enable_signature_v4_streaming` to `false` fixed the HTTP interaction entirely, and the `http_continue_timeout: 0` SDK workaround was not needed for the native backup uploader. --- ## Closing The fix was two settings in `gitlab.rb` that are not mentioned in GitLab's own backup documentation. I found the solution through Ajay's writeup about Cloudflare R2 \[2\], which is essentially the same problem with a different S3-compatible provider. If you are using any S3-compatible object storage that is not AWS S3 itself, `path_style` and `enable_signature_v4_streaming` are probably the first things to check. ## References \[1\] Tencent Cloud, "Online Migration Overview," Tencent Cloud Documentation, Aug. 13, 2026\. \[Online\]. Available: . \[Accessed: Sep. 12, 2026\]. \[2\] A. Furber, "How to get fog-aws / Gitlab Backups to work with Cloudflare R2," ajf.me.uk, Jul. 6, 2024\. \[Online\]. Available: . \[Accessed: Sep. 10, 2026\]. ### Recovering Jenkins When Azure AD Client Secret Expires URL: https://anantafatur.dev/recovering-jenkins-azure-ad-secret-expired/ Last updated: 2026-09-09T13:39:31.000Z ## The problem started with a permission issue A colleague reported they could not approve a pipeline in Jenkins. I figured it was just a missing permission, so I opened Jenkins to approve it myself. That is when I noticed something was wrong. My account was also read-only. I am a DevOps engineer at a telecom company, managing the CI/CD pipeline. Our team has admin access in Jenkins. Seeing my own account reduced to read-only meant something deeper was broken. I tried logging out and logging back in through Azure AD. The login page loaded, but after authenticating with Entra, Jenkins showed an error. The authentication was failing silently. No useful error message, just a redirect loop back to the login page. That is when I realized the Azure AD client secret had probably expired. Microsoft Entra client secrets have a maximum lifetime of 24 months, and Microsoft recommends setting an expiration of less than 12 months \[3\]. Ours had been set a long time ago and nobody had rotated it. --- ## The environment Jenkins is running on an EC2 instance, installed directly on the VM (not in a container). I had SSH access to the instance with sudo. The Jenkins version is 2.492.2\. Authentication is handled by the Azure AD plugin \[1\], with the security realm set to `AzureSecurityRealm` and authorization set to `AzureAdMatrixAuthorizationStrategy` \[1\]. I did not know exactly how Jenkins was installed on the VM. It has a systemd service, so it is likely the Debian/Ubuntu package. But the exact install method did not matter for the recovery. As long as I had OS-level access and could stop and start Jenkins, I could fix this. ## What I ruled out Before SSHing into the instance, I checked a few things from the Entra side. The app registration for Jenkins still existed. The redirect URIs were correct. The client ID and tenant ID had not changed. The only thing that could have expired was the client secret. Microsoft Entra client secrets are created under "Certificates and secrets" in the app registration and have a configurable expiration date \[3\]. I also ruled out network issues. The EC2 security group allowed my IP. The issue was purely on the authentication layer inside Jenkins. ## Getting into the EC2 I SSH'd into the instance and checked that Jenkins was running: ```bash sudo systemctl status jenkins ``` It was running. No crashes, no obvious errors in the systemd output. The problem was not that Jenkins was down. It was running fine, just refusing to authenticate anyone. I found the Jenkins home directory: ```bash sudo systemctl cat jenkins | grep JENKINS_HOME ``` It was `/var/lib/jenkins`, the default location. Before touching anything, I made a backup of the main configuration: ```bash sudo cp /var/lib/jenkins/config.xml \ /var/lib/jenkins/config.xml.bak.$(date +%Y%m%d-%H%M%S) ``` This is important. If I messed up the config, I could always restore from this backup. I was not going to lose the existing job definitions, plugin configurations, or credentials. ## Temporarily disabling security The plan was simple. Disable Jenkins security temporarily, log in without authentication, update the Entra client secret through the UI, then re-enable security. The Jenkins documentation has an official procedure for this exact scenario \[2\]. The plugin's FAQ also references the same approach \[1\]. First, I stopped Jenkins: ```bash sudo systemctl stop jenkins ``` Then I edited the config to disable security. Following the Jenkins docs \[2\], I looked for the `useSecurity` element and changed it from `true` to `false`: ```xml false ``` I also changed the authorization strategy to `Unsecured` so there would be no permission checks at all: ```xml ``` Then I started Jenkins again: ```bash sudo systemctl start jenkins ``` At this point, Jenkins was wide open. Anyone who could reach the instance could do anything. The Jenkins docs warn about this \[2\]. I made sure the security group only allowed access from my IP before doing this. ## Creating a new Entra client secret I opened Jenkins in the browser. No login page. I went straight to the dashboard. I navigated to Manage Jenkins, then Security, then the Security Realm section. The Entra ID configuration was still there. The client ID and tenant ID were intact. But the client secret field was showing the old, expired secret. I went to the Azure portal, found the app registration for Jenkins, and created a new client secret. The Microsoft Entra documentation describes this under "Certificates and secrets" where you select "New client secret," set a description and expiration, and copy the value \[3\]. I copied the value (not the Secret ID, the actual value). Then I pasted it into the Jenkins UI and saved. The plugin encrypted the new secret and wrote it to `config.xml`. Now the file had `useSecurity` set to `false`, an `Unsecured` authorization strategy, but the correct `AzureSecurityRealm` with the new client secret. At this point, I could have just re-enabled security and called it done. But I wanted to restore the original authorization strategy, which had all the team permissions configured through Azure AD groups and users. The plugin uses `AzureAdMatrixAuthorizationStrategy` to match permissions based on the Object ID of users and groups \[1\]. ## Restoring the original authorization The original authorization was `AzureAdMatrixAuthorizationStrategy`. It had permissions assigned to specific Entra groups and users. I did not want to recreate those manually. I needed to restore them from the backup. The backup had the full authorization block. Something like: ```xml GROUP:com.cloudbees.plugins.credentials.CredentialsProvider.Create: USER:com.cloudbees.plugins.credentials.CredentialsProvider.Create: ... ``` I needed to extract this block from the backup and insert it into the current config, while keeping the new `securityRealm` with the updated client secret. I also needed to change `useSecurity` back to `true`. ## A gotcha with self-closing XML Here is where I made a mistake. I tried to use a simple regex approach to replace the authorization block. But the current config had a self-closing tag: ```xml ``` The regex was looking for an opening and closing tag pair with content in between. It did not match the self-closing element. The replacement silently failed. No damage was done. The backup was still intact. But I wasted some time wondering why the authorization block did not change. I fixed it with a Python script that handles both cases. It matches either a self-closing tag or a tag pair with content, and replaces it with the authorization block from the backup: ```python import re current = "/var/lib/jenkins/config.xml" backup = "/var/lib/jenkins/config.xml.bak.20260905-102328" with open(current, "r") as f: cur = f.read() with open(backup, "r") as f: bak = f.read() new = re.search( r']*>.*?', bak, re.DOTALL ) if not new: raise SystemExit("ERROR: Authorization block not found in backup") pattern = r']*/>|]*>.*?' old = re.search(pattern, cur, re.DOTALL) if not old: raise SystemExit("ERROR: Current authorizationStrategy not found") cur = cur[:old.start()] + new.group(0) + cur[old.end():] with open(current, "w") as f: f.write(cur) print("Authorization strategy restored successfully.") ``` This script extracts the full authorization block from the backup and replaces whatever is currently in the config, whether it is self-closing or not. After running it, I verified the result: ```bash sudo sed -n '10,25p' /var/lib/jenkins/config.xml ``` The output showed the correct authorization strategy with all the original permissions intact. I also validated the XML: ```bash sudo python3 -c 'import xml.etree.ElementTree as ET; ET.parse("/var/lib/jenkins/config.xml"); print("XML OK")' ``` It returned `XML OK`. Good. I also confirmed the security realm was still there with the new client secret. The `securityRealm` block looked correct: ```xml ... Secret ... ... ``` The new secret was there, encrypted by the plugin. The original authorization was restored. `useSecurity` was back to `true`. Everything looked right. ## Verifying and restarting I restarted Jenkins: ```bash sudo systemctl restart jenkins ``` Checked the status: ```bash sudo systemctl status jenkins --no-pager ``` Jenkins came up cleanly. I tested the login endpoint locally: ```bash curl -I http://127.0.0.1:8080/login ``` It returned a redirect to the Azure login page. That was the expected behavior. The security realm was active and pointing to Entra. I opened Jenkins in an incognito browser window and tested the full Azure AD login flow. I got the Microsoft login page, authenticated, and was redirected back to Jenkins with my full admin permissions. The colleague who originally reported the issue confirmed they could approve pipelines again. The final state was exactly what I wanted: - `useSecurity` set to `true` - `authorizationStrategy` set to `AzureAdMatrixAuthorizationStrategy` with all original permissions - `securityRealm` set to `AzureSecurityRealm` with the new client secret --- ## Closing The root cause was simple. The Entra client secret had an expiration date, and nobody remembered to rotate it before it expired. Microsoft Entra client secrets have a maximum lifetime of 24 months, with a recommendation of less than 12 months \[3\]. This is a classic operations problem. Secrets expire, and if you do not have monitoring or a rotation process in place, you find out when things break. The fix itself was not complicated. Stop Jenkins, edit a config file, create a new secret, start Jenkins. The tricky part was the self-closing XML tag. I assumed the authorization strategy would always be a tag pair, and that assumption wasted time. Next time I will check the actual structure of the config before writing a regex for it. I also learned that the Azure AD plugin's encryption is tied to the Jenkins instance. Even though I was editing `config.xml` directly, the new secret I entered through the UI was encrypted properly. I did not need to understand the encryption format. I just needed to use the plugin's UI to set the secret, and then restore the surrounding config from the backup. ## References \[1\] "Microsoft Entra ID (previously Azure AD)," Jenkins Plugins, version 711.v34046f788fd7\. \[Online\]. Available: . \[Accessed: Sep. 6, 2026\]. \[2\] "Disable Access Control," Jenkins User Documentation. \[Online\]. Available: . \[Accessed: Sep. 6, 2026\]. \[3\] "Register an application with the Microsoft identity platform," Microsoft Learn, last updated Jan. 22, 2026\. \[Online\]. Available: . \[Accessed: Sep. 6, 2026\]. ### GitLab Background Migration Stuck in Finalizing State URL: https://anantafatur.dev/gitlab-migration-stuck-finalizing/ Last updated: 2026-09-07T08:35:25.000Z ## The problem I was doing a rehearsal for a GitLab upgrade. The full path I need to get through is 18.6.5 to 18.8.11 to 18.11.11 to 19.2.5 to 19.3.1, several required stops along the way based on GitLab's own upgrade path docs \[2\]. This post covers the first hop, 18.6.5 to 18.8.11, which is a direct jump with no intermediate version needed in between. I wanted to test this whole path in a rehearsal before touching anything real, and this migration issue is what I hit right at the first hop. Before any GitLab upgrade, you're supposed to check that all background migrations are done \[1\]. So I ran the usual check: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" ``` And got this: ``` Database: main finalizing | BackfillUserDetailsFields,users,id,[] ``` One migration, `BackfillUserDetailsFields`, stuck in `finalizing`. Not `finished`, not `finalized`. GitLab won't let you upgrade past this. If you try, it throws an error like \[1\]: ``` Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active' ``` So this became a blocker before I could even continue the rehearsal. --- ## What I ruled out early First thing I checked, is GitLab actually still working? Yes. Batched background migrations run through Sidekiq in isolation, so the instance stays fully operational even with one of these stuck \[1\]. This is not an outage. It only becomes a problem when you try to upgrade. My setup uses an external PostgreSQL on TencentDB for PostgreSQL, not the bundled Omnibus database. I mention this because it changes a few things later, like how you check statement timeouts. Worth keeping in mind if you're following along with your own setup. ## Trying the documented fix first GitLab's docs give a command to manually finish a stuck migration \[1\]: ``` sudo gitlab-rake gitlab:background_migrations:finalize[BackfillUserDetailsFields,users,id,'[]'] ``` I ran it, and got this instead: ``` rake aborted! NameError: uninitialized constant Gitlab::BackgroundMigration::BackfillUserDetailsFields /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:235:in `job_class' ... ``` Honestly, my first thought was that something was broken in my install. But this is actually a known pattern. GitLab has a process where old migration code gets deleted from the codebase once it's considered safe to remove, sometimes called migration squashing \[3\]. The database still had a row saying "this migration is not done yet," but the actual Ruby class that does the work was already gone from the version I had installed. I confirmed this by searching the app directory directly instead of guessing: ``` find /opt/gitlab/embedded/service/gitlab-rails -iname "*queue_backfill_user_details_fields*" find /opt/gitlab/embedded/service/gitlab-rails -iname "backfill_user_details_fields.rb" ``` Both came back empty. So yeah, the code really was not there. This probably happened because I jumped across several versions during earlier upgrades, and this migration's row got left behind in an old state while the code for it kept moving forward and eventually got squashed out. ## Trying `mark_migration_complete` There's another rake task meant for cases where a regular Rails migration is missing: ``` sudo gitlab-rake gitlab:db:mark_migration_complete[20221019002459] ``` I got the version number `20221019002459` from GitLab's own merge request history for this migration \[4\], where the original migration output was pasted in the description. Not a guess, I traced it back to the actual source. Running it gave: ``` Migration version '20221019002459' is already marked complete on database main Migration version '20221019002459' is already marked complete on database ci ``` Okay, so that part was already done. But checking the background migration status again: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" Database: main finalizing | BackfillUserDetailsFields,users,id,[] ``` Still stuck. This is the part that confused me for a bit. Turns out `mark_migration_complete` only touches the `schema_migrations` table, which tracks the regular Rails migration that *queues* the background job. It has nothing to do with `batched_background_migrations`, which is the separate table tracking the actual job's own progress. Two different tracking systems, and I only fixed one of them. --- ## The actual fix GitLab documents a manual override for exactly this situation, a batched migration that cannot run because its code is gone, and it's considered safe to skip \[1\]. You run it through the Rails console: ``` sudo gitlab-rails console ``` Then paste this in: ```ruby connection = ApplicationRecord.connection Gitlab::Database::SharedModel.using_connection(connection) do migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( Gitlab::Database.gitlab_schemas_for_connection(connection), 'BackfillUserDetailsFields', :users, :id, [] ) migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value) migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) end ``` This does not run the migration logic at all. It just marks all the job records as succeeded and the migration itself as finished, directly in the database. Which is exactly what I needed, since the code to actually run it was gone anyway. The GitLab docs are upfront that this can cause data loss and leave your instance in a state that's hard to recover from, and they recommend contacting support first if you have a support plan \[1\]. I did not have a support contract for this, so I went in with a backup taken first and accepted the risk, since I'd already confirmed the code was missing and this specific migration only touches a small set of profile fields, not anything core like auth or repos. After running it: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" Database: main ``` Nothing left. Clear. ## Checking what I actually skipped This part mattered to me. Marking something "finished" without running it feels a bit like cheating, so I wanted to know what I was actually giving up. `BackfillUserDetailsFields` was supposed to copy six fields, `linkedin`, `twitter`, `skype`, `website_url`, `location`, `organization`, from the old `users` table columns into the newer `user_details` table \[4\]. This was a follow-up to an earlier migration \[5\] that set up syncing between the two. So I tried to check if there was a real gap: ```sql SELECT COUNT(*) FROM users u JOIN user_details ud ON ud.user_id = u.id WHERE (u.linkedin IS DISTINCT FROM '' AND u.linkedin IS DISTINCT FROM ud.linkedin) OR (u.twitter IS DISTINCT FROM '' AND u.twitter IS DISTINCT FROM ud.twitter) OR (u.skype IS DISTINCT FROM '' AND u.skype IS DISTINCT FROM ud.skype) OR (u.website_url IS DISTINCT FROM '' AND u.website_url IS DISTINCT FROM ud.website_url) OR (u.location IS DISTINCT FROM '' AND u.location IS DISTINCT FROM ud.location) OR (u.organization IS DISTINCT FROM '' AND u.organization IS DISTINCT FROM ud.organization); ``` Got this back: ``` pq: column u.linkedin does not exist ``` Which actually told me something useful. Those columns are already gone from the `users` table on my instance. The earlier migration \[5\] both synced the data over *and* truncated the old columns, and that had already happened, probably years before I ever hit this stuck migration. So there was nothing left in `users` for `BackfillUserDetailsFields` to copy even if it had run. The source data was already gone before I ever touched anything. That was a relief. It meant the workaround did not actually cost me anything. It was already moot by the time I got to it. --- ## References \[1\] GitLab, "Check migrations before upgrade," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[2\] GitLab, "Plan your upgrade path," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[3\] GitLab, "Batched background migrations," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[4\] B. Austin, "Backfill existing user\_details fields from users table," GitLab Merge Request !101830, Oct. 22, 2022\. \[Online\]. Available: (accessed Sep. 7, 2026). \[5\] B. Austin, "Sync profile fields to user\_details and truncate," GitLab Merge Request !95107, Aug. 10, 2022\. \[Online\]. Available: (accessed Sep. 7, 2026). ### GitLab Sidekiq Failing After Redis Migration to Tencent Cloud URL: https://anantafatur.dev/gitlab-sidekiq-tencent-redis-resp3/ Last updated: 2026-09-05T12:55:57.000Z We were migrating our company's GitLab instance from AWS to Tencent Cloud. Since Redis does not hold persistent data in a typical GitLab setup, we decided to migrate it first as a low-risk warm-up before touching the main instance. That turned out to be wrong. ## The environment GitLab was running on an EC2 instance on AWS. Redis was on AWS ElastiCache, running Redis OSS 7.0.7\. The two lived in the same VPC. Everything was working fine before we touched anything. The plan was to repoint GitLab's Redis to a new Tencent Cloud Redis instance hosted at `:6379`. The Tencent instance was in a different VPC and a different cloud entirely, connected over a cross-cloud network link. We provisioned it via Terraform using the `tencentcloud_redis_instance` resource with `type_id = 17`. In Tencent's Terraform provider, `type_id` is an integer that maps to a specific Redis engine version and architecture. `17` means Redis 7.0 Memory Edition, Standard Architecture \[1\]. The full list of values is in the [Tencent provider docs](https://registry.terraform.io/providers/tencentcloudstack/tencentcloud/latest/docs/resources/redis%5Finstance#type%5Fid-1) \[1\], but the ones you will most likely encounter are: - `8`: Redis 5.0, standard architecture - `15`: Redis 6.2, standard architecture - `16`: Redis 6.2, cluster architecture - `17`: Redis 7.0, standard architecture - `18`: Redis 7.0, cluster architecture The number tells you the engine version and whether it runs in standard (single primary with replicas) or cluster (sharded) mode. We picked `17` because we wanted to match the source as closely as possible. ElastiCache was on Redis OSS 7.0.7, so Redis 7.0 on Tencent was the nearest available option. Same major version, same standard architecture, one shard, one replica. --- ## The symptom After repointing and restarting GitLab, everything seemed fine at first. Login worked. Repo access worked. Push and pull worked. But when I opened the GitLab admin panel and went to `/admin/background_jobs`, I got a 500 error. Just the generic "We're sorry, something went wrong on our end." Sidekiq, the background job processor, was crashing on startup. ## Ruling out the obvious My first thought was network. GitLab is on AWS, Redis is on Tencent Cloud. Different VPCs, different clouds. That's a lot of surface area for a firewall rule to go wrong. So I tested connectivity from the GitLab EC2 instance directly: ```bash telnet 6379 ``` It connected. Port was open, no firewall blocking it. So it was not a network issue. ## Finding the root cause I tailed the Sidekiq logs and grepped for the error: ```bash sudo grep -B 50 "requires Redis 6+" /var/log/gitlab/sidekiq/current ``` This is what I found: ``` {"severity":"INFO","time":"2026-06-03T16:30:36.085Z","message":"Sidekiq 7.3.9 connecting to Redis with options {:size=>25, :pool_name=>"internal", :host=>"", :port=>6379, :ssl=>false, :password=>"REDACTED",...}"} redis-client requires Redis 6+ with HELLO command available (redis://:6379) ``` The error is clear. GitLab's `redis-client` gem sends the `HELLO` command when it connects \[2\]. `HELLO` was introduced in Redis 6.0.0 as part of the RESP3 protocol \[3\]. If Redis does not respond to it, the gem refuses to connect \[2\]. To confirm, I tested directly from the EC2 instance: ```bash redis-cli -h -p 6379 -a '' HELLO 3 ``` ``` (error) ERR unknown command 'HELLO' ``` And for comparison, against the old ElastiCache endpoint: ```bash redis-cli -h .cache.amazonaws.com HELLO 3 ``` ``` Error: Protocol error, got "%" as reply type byte ``` Both failed the `HELLO 3` test differently, but GitLab was working fine with ElastiCache before. That confused me for a moment. The difference is that ElastiCache does support enough of the protocol for GitLab's gem to negotiate correctly at the application level. The CLI error there is a client-side quirk, not a hard rejection. Tencent's Redis returns `ERR unknown command 'HELLO'`, which is a hard failure that kills the connection before Sidekiq can even initialize. ## Why Tencent Redis rejects HELLO The actual Redis version on Tencent is 7.1.4, which absolutely supports RESP3 and the `HELLO` command \[3\]. But GitLab does not connect to Redis directly. The connection goes through Tencent's distributed cache proxy layer first \[4\]: ``` GitLab | Tencent Proxy (5.8.25) | Redis 7.1.4 ``` That proxy does not support RESP3\. I confirmed this with the Tencent team the next day. Their exact response: > Tencent Cloud Redis Currently Does Not Support the RESP3 Protocol. The Proxy of Tencent Cloud's distributed cache database does not currently support the RESP3 protocol; executing HELLO 3 will return an error. For both Standard and Cluster editions, the HELLO command currently only supports HELLO 2, and it can only be executed successfully after the user has performed AUTH authentication. They also confirmed this applies across all versions on their platform, including Redis 5.0, 6.2, 7.0, and even Valkey 8.0\. The limitation is in the proxy, not the Redis version itself. The only potential workaround they mentioned was a "Direct Connection Mode" that bypasses the proxy. But as of when we hit this, it was still in canary. They said it would be generally available sometime in August. On top of that, we checked whether Valkey edition could be provisioned through Terraform as an alternative. It cannot, at least not through the official Tencent provider at the time of writing. The `tencentcloud_redis_instance` resource does not have Valkey-specific options yet \[1\]. ## The fix We rolled back. I repointed GitLab's Redis config back to AWS ElastiCache, restarted GitLab, and the background jobs page came back to normal immediately. ```bash sudo gitlab-ctl restart ``` That was the end of it for that environment. ## What happened after the full migration When we eventually migrated the entire GitLab instance to Tencent Cloud, we switched to GitLab's built-in Redis instead. That means Redis runs on the same VM as GitLab itself \[5\]. To enable it, I commented out the external Redis host config in `/etc/gitlab/gitlab.rb` and set: ```ruby redis['enable'] = true ``` Then ran `gitlab-ctl reconfigure`. The trade-off is that built-in Redis consumes RAM on the GitLab VM. I did not do a controlled comparison because the AWS and Tencent environments have different instance types and virtualization layers, so it would not be a fair benchmark anyway. What I can say is that it works, and Sidekiq is healthy. --- ## Looking back The real mistake was not testing Redis compatibility before the migration. We assumed that because both ElastiCache and Tencent Redis were "Redis 7," they would behave the same. They do not. The version of the backend engine matters less than what sits in front of it. The `HELLO` command compatibility check would have been a two-minute test before any change. I did not do it. That is on me. If you are planning to migrate GitLab to any managed Redis service, run this from your GitLab host before you repoint anything: ```bash redis-cli -h -p 6379 -a '' HELLO 2 ``` If you get back a map response with server info, you are probably fine. If you get `ERR unknown command`, stop and do not proceed. ## References \[1\] TencentCloud, "tencentcloud\_redis\_instance," Terraform Registry. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[2\] redis-rb, "redis-client: Simple low-level client for Redis 6+," GitHub. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[3\] Redis Ltd., "HELLO," Redis Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[4\] Tencent Cloud, "Proxy Features," Tencent Cloud Distributed Cache Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. \[5\] GitLab, "Configuring Redis for scaling," GitLab Documentation. \[Online\]. Available: . \[Accessed: Sep. 5, 2026\]. ### Checking IDX-listed companies for annual reports using Deno URL: https://anantafatur.dev/checking-idx-annual-reports-deno/ Last updated: 2026-09-03T03:20:29.000Z I had a simple question: do these 102 companies in the Basic Materials sector have annual financial reports on the IDX website? I needed to check for the years 2022 to 2025\. Not download the PDFs, not read the reports. Just a yes or no per company per year. ## Why I needed this I was working with a list of Indonesian stock market emiten. The list had 102 companies, all in the Basic Materials sector. Each one had a ticker code, a name, and a sector classification. Something like this: ``` 103;AVIA;Avian Tbk;Basic Materials 104;AYLS;Agro Yasa Lestari Tbk;Basic Materials 105;BAJA;Saranacentral Bajatama Tbk;Basic Materials ``` I wanted to know which of these companies have published their annual financial reports. The IDX website has a page for each company with financial reports going back years. But checking 102 companies manually, for 4 years each, would have taken forever. I needed to automate it. --- ## Finding an existing API wrapper I searched around and found a repo called [IDX-API](https://github.com/NeaByteLab/IDX-API) by NeaByteLab. It is a Deno project that wraps the IDX API endpoints. It handles session cookies, retries on failure, and stores data in a local SQLite database using Drizzle ORM. The repo already had a `CompanyModule` class with a method called `getFinancialReports`. It takes a company code, a year, and a period like `'audit'` (for annual reports) or `'TW1'` (quarter one). It calls the IDX API and returns the report metadata. I forked the repo and started looking at the code. ## How the API works The IDX API endpoint for financial reports looks like this: ``` https://www.idx.co.id/primary/ListedCompany/GetFinancialReport ?kodeEmiten=BBCA &year=2024 &periode=audit &pageSize=100 &reportType=rdf ``` The API returns JSON with a `Results` array. If the company has a report for that year, `ResultCount` is 1 and the `Results` array contains the attachment metadata. If not, `ResultCount` is 0. But here is the catch. The IDX website uses Cloudflare. You cannot just call the API directly. The existing wrapper handles this by first visiting the main IDX page to get a session cookie, then making a validation request, and only then calling the actual API. The `BaseClient` class in the repo does all of this. It stores the session cookie and adds it to every request. It also has retry logic with exponential backoff. ## Writing the check script Since the repo already had the API client, I did not need to write much. I just needed a script that reads my CSV file, loops through the companies, and calls `getFinancialReports` for each year. Here is the core of the script: ```typescript import IDXClient from '@app/index.ts' const client = new IDXClient() const years = [2022, 2023, 2024, 2025] for (const emiten of emitenList) { for (const year of years) { const reports = await client.company.getFinancialReports( emiten.code, year, 'audit' ) if (reports && reports.length > 0) { // YES, this company has a report for this year } else { // NO, no report found } } } ``` I wrote the results to a CSV file so I could stop the script at any time and still have the data. The output looked like this: ``` No;Code;Name;2022;2023;2024;2025;Total 1;AVIA;Avian Tbk;YES;YES;YES;YES;4/4 2;AYLS;Agro Yasa Lestari Tbk;YES;YES;YES;YES;4/4 3;BAJA;Saranacentral Bajatama Tbk;YES;YES;YES;NO;3/4 ``` ## Getting blocked by Cloudflare The first run failed after about 17 requests. Cloudflare started returning a block page instead of the API response. My script was too fast. The existing wrapper had retry logic for server errors, but it did not handle Cloudflare blocks. When Cloudflare returns an HTML page saying "Sorry, you have been blocked," the JSON parser fails and the function returns null. So my script marked those as "NO" even though the company might actually have a report. I needed to slow down the requests. A lot, apparently. I added random delays between each request. Between 1 and 2.5 seconds for each year check, and 3 to 5 seconds between companies. I also added a 15 second pause every 10 companies. And if the script detected a rate limit response (503, 403, or Cloudflare-specific errors), it would wait 30 seconds and retry once. ```typescript function randomDelay(min: number, max: number): Promise { const ms = Math.floor(Math.random() * (max - min + 1)) + min return new Promise((r) => setTimeout(r, ms)) } ``` With these delays, the script ran for about 15 minutes to check all 102 companies. Not fast, but it worked. No more Cloudflare blocks. --- ## The results Out of 102 companies in the Basic Materials sector: - 63 companies had all 4 years of reports (2022 through 2025). That is about 62 percent. - 32 companies had partial data. Some years missing, usually the earlier ones. - 7 companies had little to no data. Some like KBRI (Kertas Basuki Rachmat Indonesia) and SIMA (Siwani Makmur) had zero reports for the entire period. The full results are in the CSV file. Nothing surprising, but it confirmed what I needed for my work. ## The code You can find the script in my fork of the repo: [anantafaturdev/IDX-API](https://github.com/anantafaturdev/IDX-API). The check script is `check-emiten-reports.ts` and the results are in `emiten-report-check.csv`. ### Fixing Blue Ocean CSP Error After Plugin Upgrade URL: https://anantafatur.dev/fix-blue-ocean-csp-jenkins/ Last updated: 2026-08-28T10:33:11.000Z ## The problem I upgraded the Blue Ocean plugin on our Jenkins instance from version 1.27.25 to 1.27.26\. The old version worked fine, but Blue Ocean is deprecated. I wanted to grab the latest version while it's still available. After the restart, the Blue Ocean UI was broken. The browser console was full of CSP errors. This is the error I saw: ``` EvalError: Evaluating a string as JavaScript violates the following Content Security Policy directive because 'unsafe-eval' is not an allowed source of script: script-src 'report-sample' 'self'". at new Function () at optimizeLookup (blueocean-core-js.js:48503:13) ``` The old version (1.27.25) never had this problem. Something changed between 1.27.25 and 1.27.26 in the bundled JavaScript. The Blue Ocean JavaScript bundle was failing to initialize. The entire plugin was dead. --- ## The environment Our Jenkins runs inside a Docker container on a Tencent Cloud CVM, behind an nginx reverse proxy. The stack looks like this: - Jenkins 2.568.2 LTS (official Docker image) - Blue Ocean plugin (latest version, just upgraded) - CSP enabled via the Jenkins web UI checkbox - No custom CSP headers in nginx The Jenkins instance is internal, only accessible by the team. It's not public-facing. ## What I tried first The error message was pretty clear. The CSP `script-src` directive was blocking `unsafe-eval`. The Blue Ocean JavaScript bundle uses `new Function()` in its `optimizeLookup` function, and the browser treats that as `eval`. My first thought was to override the CSP by adding the `hudson.model.DirectoryBrowserSupport.CSP` system property. I added this to the `JAVA_OPTS` in the Docker Compose file: ```yaml JAVA_OPTS: "-Xms512m -Xmx4096m -Dhudson.model.DirectoryBrowserSupport.CSP=\"sandbox allow-scripts; default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline';\"" ``` I redeployed the container. Checked that the environment variable was set correctly with `docker inspect`: ``` JAVA_OPTS=-Xms512m -Xmx4096m -Dhudson.model.DirectoryBrowserSupport.CSP="sandbox allow-scripts; default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline';" ``` It was there. But the error still showed the default CSP: `script-src 'report-sample' 'self'`. My override was being ignored. ## What confused me At this point I was confused. The system property was set. The container had it. But Jenkins was still using the default CSP. Then I remembered: I had enabled CSP via the Jenkins web UI. In **Manage Jenkins > Configure Global Security**, there's a checkbox for "Content Security Policy". I had it checked. Here's the thing I didn't know: when you enable CSP via the web UI checkbox, Jenkins uses its own default CSP value. It does not give you a text field to customize the value. The checkbox just toggles between "default CSP on" and "CSP off". There's no way to add `unsafe-eval` through the UI. And the web UI setting takes precedence over the system property. So even though `JAVA_OPTS` had the correct value, Jenkins ignored it. ## The resource root URL dead end The Jenkins system settings page showed a warning: > The default Content-Security-Policy is currently overridden using the > hudson.model.DirectoryBrowserSupport.CSP system property, which is a > potential security issue when browsing untrusted files. As an alternative, > you can set up a resource root URL that Jenkins will use to serve some > static files without adding Content-Security-Policy headers. I tried setting up a Resource Root URL. The idea is that Jenkins serves Blue Ocean's static files from a separate URL without CSP headers, while the main Jenkins UI keeps its CSP protection. But Jenkins rejected it. The resource root URL must be on a different hostname than the Jenkins URL. I tried `/resourceRoot` and it said: > Cannot use the same host name for both Jenkins URL and resource root URL. Setting up a separate subdomain would mean creating a new DNS record, updating the SSL certificate, and adding nginx config. Too much work for a plugin upgrade. --- ## The actual fix I went back to the CSP checkbox in the web UI and just disabled it. That's it. Blue Ocean started working immediately. The fix is simple: go to **Manage Jenkins > Configure Global Security**, find the **Content Security Policy** checkbox, and uncheck it. Is this ideal? No. Disabling CSP removes a security layer. CSP helps prevent XSS attacks by restricting what scripts the browser can execute. For now, the trade-off is acceptable. We're planning to add Cloudflare Zone Lockdown to restrict access to the Jenkins instance, which will reduce the risk of XSS attacks. But until that's in place, there's a small gap. ### Checking if your GitLab instance is vulnerable to CVE-2026-19478 URL: https://anantafatur.dev/check-gitlab-cve-2026-19478/ Last updated: 2026-08-28T09:40:56.000Z A few days ago GitLab disclosed CVE-2026-19478, a CVSS 9.4 vulnerability in the GraphQL layer. No authentication needed, no user interaction. Just a single GraphQL query that can delete public projects or modify user data. I manage a self-hosted GitLab instance for our team. I wanted to check if we were affected. Here is how I did it, so you can follow the same steps on your own instance. ## What the vulnerability is in one paragraph The bug is in a GitLab GraphQL feature called `@gl_introduced`. This directive is used for version gating. When you query a field with `@gl_introduced(version: "999.0.0")`, GitLab creates a fallback field and calls Ruby's `public_send` on the database object using the field name as the method name. An attacker can call any zero-argument method on `Project` or `User` objects, like `destroy` or `deactivate`. The PoC is available at [davkharrr/cve-2026-19478-poc](https://github.com/davkharrr/cve-2026-19478-poc). It has a safe `check` mode that only calls `touch` (updates a timestamp, no damage). --- ## Step 1: Check your GitLab version First, check if your version falls in the affected range. The vulnerable versions are: - 18.x: 18.2.0 up to 18.11.11 - 19.0: up to 19.0.8 - 19.1: up to 19.1.6 - 19.2: up to 19.2.4 Our instance was on a version squarely in the affected range. If you are already on 18.11.11 or later, you are patched. ## Step 2: Run the PoC in check mode The PoC has a safe mode that just calls `touch`. It does not modify or delete anything. I ran it against our instance with a known username: ```bash python3 poc.py --url https:// --user --mode check ``` The output: ``` [*] Could not determine GitLab version (proceeding with exploit attempt). [*] Target object : user: [*] Method invoked: touch [*] Query: query { user(username: "") { username touch @gl_introduced(version: "999.0.0") } } [*] HTTP 200 [!] Parent object is null -> project/user not found or not visible. ``` The important parts: HTTP 200 means the GraphQL endpoint is reachable. But the parent object was null, meaning the user was not exposed to unauthenticated queries. That is a good sign. You can also try against a project path: ```bash python3 poc.py --url https:// --project group/project-name --mode check ``` If you get `"touch": true`, the instance is vulnerable. If you get `Field 'touch' doesn't exist on type 'Project'`, you are patched. If you get `Parent object is null`, the project is not publicly accessible. ## Step 3: Check if public projects are allowed This is the key check. The vulnerability only works on public projects (or public users). If you do not have public projects, the attack surface is closed. I checked our instance settings using the admin API: ```bash curl -s --header "PRIVATE-TOKEN: " \ "https:///api/v4/application/settings" | grep restricted_visibility_levels ``` The response: ```json "restricted_visibility_levels": ["internal", "public"] ``` This means both `internal` and `public` visibility levels are blocked on our instance. No one can create a public project. Every project is private by force. ## Step 4: Confirm no public projects exist Even with the setting above, I wanted to confirm there are no existing public projects: ```bash curl -s --header "PRIVATE-TOKEN: " \ "https:///api/v4/projects?visibility=public&per_page=100" ``` The response was an empty array: ```json [] ``` No public projects. If you run this command and get results, those projects are targets. ## What I found Our instance was not exploitable for unauthenticated attacks. The reasons: - The GraphQL endpoint is reachable, but that alone is not enough. - Public projects are blocked globally by `restricted_visibility_levels`. - User profiles are not exposed to unauthenticated GraphQL queries. - No public projects exist on the instance. The vulnerability exists in the code, but it is not reachable without authentication on our setup. --- ## What you should check on your own instance If you have a self-hosted GitLab instance, here is what I recommend: 1. Check your version. If you are below the fixed versions, plan an upgrade. 2. Run the PoC in check mode against a public project or user. 3. Check your `restricted_visibility_levels` setting. If `public` is not in the list, consider adding it. 4. Enumerate public projects. If any exist, they are attack targets. The `restricted_visibility_levels` setting is what saved us. In our case, `public` and `internal` were already blocked for other reasons. It turned out to be the best defense against this CVE. If you do have public projects and cannot patch immediately, consider a WAF rule that blocks requests containing `@gl_introduced` in the GraphQL body. The fix versions are 18.11.11, 19.0.8, 19.1.6, and 19.2.4. ### Fixing duplicate DKIM-Signature header with AWS SES and Rspamd URL: https://anantafatur.dev/aws-ses-duplicate-dkim-signature-rspamd/ Last updated: 2026-08-25T16:16:30.000Z ## The symptom I manage a self-hosted mail server for our AWS User Group Jakarta community. We use Billion Mail with AWS SES as the outbound relay. It had been running fine for months, then suddenly last month, emails started bouncing. No config changes. No updates. It just broke. The Postfix queue log showed this: ``` 5.0.0 1.6 0.63/0/0.71/0.27 bounced host email-smtp.ap-southeast-1.amazonaws.com[] said: 554 Transaction failed: Duplicate header 'DKIM-Signature'. (in reply to end of DATA command) ``` My first thought was, "Wait, what duplicate DKIM header? Nothing changed. This was working last month." I was wrong about the "nothing changed" part, something must have changed, I just could not see it yet. ## The environment Our mail stack runs on a single EC2 instance with these Docker containers: - **Postfix** (`billionmail/postfix:1.6`) as the MTA - **Rspamd** (`billionmail/rspamd:1.2`) as the milter for spam filtering and DKIM signing - **Dovecot** for IMAP and SASL authentication - **AWS SES** as the SMTP relay in the `ap-southeast-1` region - The SES relay is configured via the Billion Mail web console, which stores the relay mapping in PostgreSQL The relay config maps our domain to `email-smtp.ap-southeast-1.amazonaws.com:587` with SASL authentication. --- ## The investigation I started by checking the Postfix main.cf inside the container. The milter configuration looked like this: ```ini # dovecot rspamd smtpd_sasl_type = dovecot smtpd_sasl_authenticated_header = yes smtpd_sasl_path = inet:dovecot:10002 smtpd_sasl_auth_enable = yes virtual_transport = lmtp:inet:dovecot:26 smtpd_milters = inet:rspamd:11332 non_smtpd_milters = inet:rspamd:11332 smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination smtpd_use_tls = yes milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen} milter_protocol = 6 milter_default_action = accept ``` The `smtpd_milters` and `non_smtpd_milters` lines tell Postfix to pipe every email through Rspamd. That includes both incoming and outgoing mail. Then I checked the Rspamd DKIM signing config at `/etc/rspamd/local.d/dkim_signing.conf`: ```nginx sign_headers = "from:sender:reply-to:subject:date:message-id:to:cc:mime-version:content-type:content-transfer-encoding:content-language:resent-to:resent-cc:resent-from:resent-sender:resent-message-id:in-reply-to:references:list-id:list-help:list-owner:list-unsubscribe:list-subscribe:list-post:list-unsubscribe-post:disposition-notification-to:disposition-notification-options:original-recipient:openpgp:autocrypt"; use_esld = false; domain { #BT_DOMAIN_DKIM_BEGIN #_DKIM_BEGIN { selectors [ { path: "/var/lib/rspamd/dkim//default.private"; selector: "default"; }, { path: "/var/lib/rspamd/dkim//short.private"; selector: "short"; } ] } #_DKIM_END #BT_DOMAIN_DKIM_END } ``` And there it was. Rspamd was configured to DKIM-sign every outgoing email from our domain with two selectors, `default` and `short`. ## The root cause Here is the flow of what was happening: 1. A user sends an email through Roundcube (or an MUA) 2. Postfix receives it and pushes it through the Rspamd milter 3. Rspamd adds a `DKIM-Signature` header (or two, since there are two selectors) for our domain 4. Postfix then forwards the email to AWS SES via the relay config 5. SES sees the email already has a `DKIM-Signature` header and rejects it The error message is a bit misleading. SES does not actually add its own DKIM signature when sending via SMTP credentials (it only does that with Easy DKIM via the API). But SES still validates the email structure and refuses to relay messages that have pre-existing DKIM-Signature headers. The exact reason is not fully documented by AWS, but the behavior is consistent, other people have hit this with Postfix and SES too. The core problem is that both Rspamd and SES "want" to be the one signing your emails. When Rspamd signs first, SES sees a duplicate and bounces the message. Honestly, I still do not know why this worked before and suddenly broke. The config had not changed. My best guess is that AWS SES tightened its header validation on their side, maybe in a silent update. But I could not find any announcement about it. If you know what actually triggered this, I would love to hear about it. --- ## The fix I had two options. **Option A**: Remove the DKIM domain block from Rspamd's config and let SES handle DKIM signing via Easy DKIM. This is the cleaner approach because it preserves Rspamd's spam filtering while only disabling the signing part. **Option B**: Completely disable the Rspamd milter in Postfix. This is the nuclear option, it turns off everything Rspamd does, including spam filtering. But it is simple and it works immediately. I went with Option B because I did not want to set up Easy DKIM in SES right now. Here is the change I made to the Postfix main.cf: ```ini # dovecot (milter disabled to stop DKIM conflict with SES) smtpd_sasl_type = dovecot smtpd_sasl_authenticated_header = yes smtpd_sasl_path = inet:dovecot:10002 smtpd_sasl_auth_enable = yes virtual_transport = lmtp:inet:dovecot:26 # Milters disabled #smtpd_milters = inet:rspamd:11332 #non_smtpd_milters = inet:rspamd:11332 smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination smtpd_use_tls = yes # Milter params commented out since milter is off #milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen} #milter_protocol = 6 #milter_default_action = accept ``` I applied this through the Billion Mail web console, under Settings, Service, Edit config of the Postfix container. Then I reloaded Postfix and tried sending again. The email went through immediately. ![](https://anantafatur.dev/content/images/2026/08/image-3.png) This workaround came from a [GitHub issue on the BillionMail repository](https://github.com/Billionmail/BillionMail/issues/289), where someone else hit the same problem with SES and Rspamd. --- ## What I gave up By disabling the milter entirely, I also turned off Rspamd's spam filtering, virus scanning, and reputation checks. For now that is fine because our community mail server has a small number of users and we do not receive much spam. But if you have multiple users or public-facing addresses, you should probably go with Option A instead, just remove the DKIM domain block from `dkim_signing.conf` and restart Rspamd. ### Fixing MariaDB column_stats upgrade error URL: https://anantafatur.dev/fixing-mariadb-column-stats-upgrade-error/ Last updated: 2026-08-25T08:13:32.000Z I updated my Ghost CMS Docker Compose stack from MariaDB 12.3.2 to 12.3.3\. A patch bump. Nothing to worry about, right? After restarting the containers, everything seemed fine. The blog loaded. No downtime. But when I checked the logs, I saw this: ``` ghost-mariadb | 2026-08-25 7:54:19 3 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to be nullable. ``` The container was running. Ghost was serving pages. But an error was sitting in the logs, and I did not like it. --- ## The environment My setup is a simple Docker Compose file with two services: - **ghost-app** — Ghost 6.59.0 Alpine - **ghost-mariadb** — MariaDB 12.3.3 Noble The MariaDB data lives on a named Docker volume (`ghost-mariadb`). I was upgrading from `mariadb:12.3.2-noble` to `mariadb:12.3.3-noble`. A patch bump, so I honestly did not expect any issues. That was the key detail. ## What I ruled out first At first I thought it might be a Ghost compatibility issue. But Ghost was running fine. Pages loaded. The API worked. Whatever was wrong, it was on the MariaDB side. I also checked if the volume was corrupted. It was not. The data was intact, the tables were accessible. The error was specifically about a schema definition mismatch, not about data corruption. So I dug into the logs. ## What was going on I looked at the full startup log more carefully. Two lines stood out: ``` [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting ``` The MariaDB entrypoint script knew the data directory needed an upgrade. But it skipped it. The `MARIADB_AUTO_UPGRADE` environment variable was not set in my compose file, and for whatever reason, the entrypoint chose to skip rather than run the upgrade automatically. So the server started with system tables from 12.3.2\. The `mysql.column_stats` table had a `hist_type` column defined as `NOT NULL`. But MariaDB 12.3.3 expected it to be nullable. The column\_stats table stores histogram statistics for the query optimizer. A mismatch here means the optimizer cannot use column histograms correctly. The server would still run. Queries would still work. But the error was there, and I did not want to leave it. --- ## The fix What worked for me was running `mariadb-upgrade` inside the container. It checks and repairs system tables, views, and privileges. The MariaDB docs have more detail on this, but the command itself is simple. ```bash docker exec ghost-mariadb mariadb-upgrade -u root -p"" ``` The output was long but clean. Phase 1 checked the `mysql` database and fixed the `column_stats` table immediately: ``` Phase 1/8: Checking and upgrading mysql database Processing databases mysql mysql.column_stats OK ``` The rest of the phases ran through views, privilege tables, and all the Ghost application tables. Everything came back OK. No drama. ## Preventing it next time After the upgrade, I added `MARIADB_AUTO_UPGRADE: 1` to the `ghost-mariadb` environment in my `docker-compose.yaml`: ```yaml ghost-mariadb: image: mariadb:12.3.3-noble environment: MARIADB_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} MARIADB_DATABASE: ${MYSQL_DATABASE} MARIADB_USER: ${MYSQL_USER} MARIADB_PASSWORD: ${MYSQL_PASSWORD} MARIADB_AUTO_UPGRADE: 1 ``` This tells the MariaDB entrypoint to run `mariadb-upgrade` automatically when it detects the data directory is from an older version. No more skipped upgrades, no more silent schema mismatches. ### Jenkins Azure AD login redirects to root instead of original URL URL: https://anantafatur.dev/jenkins-azure-ad-redirect-root/ Last updated: 2026-08-22T19:01:12.000Z We run Jenkins behind nginx, with Azure AD for authentication. A user reported that when they tried to access a deep URL, like `/job////`, they got bounced to Azure AD to log in. So far, normal. But after logging in, they landed on the Jenkins root URL instead of the page they originally asked for. Every time. This kind of thing is annoying. You click a link someone sent you, you authenticate, and then you have to navigate back to where you wanted to go. Multiply that by a team of engineers and it adds up fast. ## The setup Jenkins is running in a Docker container behind an nginx reverse proxy, both on the same VM. The HTTPS certificate is terminated by a cloud load balancer, which forwards HTTP to nginx. The DNS for `` is a CNAME pointing to the load balancer's public DNS name. The load balancer's public IP is something like ``. Jenkins is configured with its URL set to `https:///`. The Azure AD app registration has the Reply URL set to `https:///securityRealm/finishLogin`. Both look correct. So what was going wrong? --- ## First clue: the NullPointerException The Jenkins logs had this error, repeated every time someone tried to log in: ``` Error while serving https:///securityRealm/commenceLogin java.lang.NullPointerException: Cannot invoke "Object.toString()" because the return value of "jakarta.servlet.http.HttpSession.getAttribute(String)" is null at PluginClassLoader for azure-ad//com.microsoft.jenkins.azuread.oauth.StateCache.generateValue(StateCache.java:18) at PluginClassLoader for azure-ad//com.microsoft.jenkins.azuread.AzureSecurityRealm.doCommenceLogin(AzureSecurityRealm.java:388) ``` Two things jumped out at me. First, the URL in the error message was `https:///securityRealm/commenceLogin`. Not ``. The IP address. That meant Jenkins was seeing the request as coming from the load balancer's IP, not the domain name. Second, the `StateCache.generateValue` method was failing because a session attribute was null. The `StateCache` is what stores the original URL the user was trying to access. If it can't find the session attribute, the original URL is lost. ## Why the load balancer IP was showing up The request flow is: ``` Browser (DNS) → CNAME → Load balancer → nginx → Jenkins ``` The load balancer terminates HTTPS and forwards HTTP to nginx. But when it forwards the request, it sends the `Host` header as its own IP address, not the original domain. My nginx config had: ```nginx proxy_set_header Host $host; ``` The `$host` variable in nginx evaluates to whatever `Host` header it received from upstream. In this case, the load balancer was sending `Host: `. So nginx forwarded that to Jenkins. Jenkins then used the IP address to construct its redirect URLs. When Jenkins redirected the browser to `/securityRealm/commenceLogin`, the browser got a URL with the IP address. The browser went there, but the JSESSIONID cookie was set for the domain ``. The browser didn't send the cookie to the IP address. Jenkins saw a brand new session with no saved attributes. The `StateCache` crashed. ## Fix 1: hardcode the Host header in nginx The fix was simple. Instead of forwarding whatever `Host` header the load balancer sent, I hardcoded it to the domain: ```nginx proxy_set_header Host ; proxy_set_header X-Forwarded-Host ; ``` I also replaced `proxy_redirect off` with proper redirect rewriting, just as a safety net: ```nginx proxy_redirect http://:8080/ https:///; proxy_redirect http://:8080 https:///; ``` After restarting nginx, the `NullPointerException` was gone. The `doCommenceLogin` method was completing without errors. But the redirect after login was still going to the root URL. ## Second clue: the updateAvatar error The new logs showed a different error: ``` WARNING ... AzureSecurityRealm updateAvatar Failed to get profile photo for com.microsoft.graph.http.GraphServiceException: 401 : Unauthorized GET https://graph.microsoft.com/v1.0/users//photos/48x48 ``` The `doFinishLogin` method was now being called successfully. It authenticated the user. Then it called `updateAvatar`, which tried to fetch the user's profile photo from Microsoft Graph. That call failed with a 401. I looked at the Azure AD app permissions. The app had `User.Read` (Delegated) for Microsoft Graph. But looking at the plugin source code, the `getAzureClient()` method uses client credentials (application context) to call the Graph API. The `getAccessToken()` method requests the `.default` scope, which only resolves application permissions. `User.Read` is a delegated permission, not an application permission. So the token didn't have the right scope. But here's the thing. Looking at the `updateAvatar` method in the plugin source, it catches the `ApiException` and logs it. It doesn't re-throw it. So the `doFinishLogin` method should continue to the redirect code. Yet the redirect was still going to the root URL. I was about to try disabling the avatar fetching with a system property when I noticed something in the plugin changelog. ## The real fix: plugin version 668 The Azure AD plugin we had installed was version **667**. Version **668** had a single fix listed: > "Restore support for redirecting to non root path" (PR #800) That was it. The redirect to the root URL was a known bug in version 667, fixed in 668\. The `updateAvatar` error was a red herring. It was logged as a warning, but the real reason the redirect was failing was the plugin bug. Upgrading to version 668 fixed the issue immediately. After the upgrade, users who logged in through Azure AD were redirected to the original URL they requested, not the root. --- ## Closing The nginx `Host` header fix was still necessary. The load balancer sending the IP as the `Host` header was causing the `StateCache` NullPointerException, which was a separate issue. If I had only upgraded the plugin, the login would still have failed with the first error. The `updateAvatar` error was a distraction. It was logged as a warning but didn't actually block the redirect. I spent time looking at Graph API permissions when the real fix was a one-line plugin upgrade. After all changes, here's what the nginx config looks like: ```nginx server { listen 80; server_name ; location / { proxy_pass http://:8080; proxy_http_version 1.1; proxy_set_header Host ; proxy_set_header X-Forwarded-Host ; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Port 443; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_redirect http://:8080/ https:///; proxy_redirect http://:8080 https:///; } } ``` And the Azure AD plugin is now on version 668. ### Bypassing Cloudflare DNS to Speed Up Git Clone in Jenkins URL: https://anantafatur.dev/bypass-cloudflare-dns-git-clone-jenkins/ Last updated: 2026-08-22T18:11:04.000Z ## The problem: a 12-minute git clone I maintain a Jenkins pipeline that runs terraform operations on Tencent Kubernetes Engine (TKE). Jenkins itself runs as a Docker container on a CVM, version 2.568.2 LTS, with the Kubernetes executor spawning runner pods on TKE. Every run starts with a git clone from our self-hosted GitLab at ``. Looking at the pipeline logs, the checkout stage was taking forever. Here is a typical run before I made any changes: ``` 19:05:29 [ Run Log ] [ Stage ] [ Checkout SCM ] START [Pipeline] checkout 19:07:14 The recommended git tool is: NONE 19:17:33 using credential 19:17:33 Cloning the remote Git repository 19:17:33 Using shallow clone with depth 1 19:18:01 Cloning repository git@:/.git 19:18:09 > git init /home/jenkins/agent/workspace// # timeout=10 19:18:12 Fetching upstream changes from git@:/.git 19:18:12 > git --version # timeout=10 19:18:12 > git --version # 'git version 2.47.3' 19:18:12 using GIT_SSH to set credentials 19:18:14 Verifying host key using known hosts file, will automatically accept unseen keys 19:18:14 > git fetch --tags --force --progress --depth=1 -- git@:/.git +refs/heads/*:refs/remotes/origin/* # timeout=10 19:20:11 Avoid second fetch 19:20:12 Checking out Revision (origin/master) ``` Look at the gaps. 1 minute 45 seconds between START and "The recommended git tool is: NONE." Another 10 minutes before "using credential." The actual git fetch took nearly 2 minutes. Total: almost 15 minutes for a shallow clone. After the DNS changes, the same pipeline looked like this: ``` 00:34:19 [ Run Log ] [ Stage ] [ Checkout SCM ] START [Pipeline] checkout 00:34:19 The recommended git tool is: NONE 00:34:21 using credential 00:34:21 Cloning the remote Git repository 00:34:21 Using shallow clone with depth 1 00:34:21 Cloning repository git@:/.git 00:34:21 > git init /home/jenkins/agent/workspace// # timeout=10 00:34:21 Fetching upstream changes from git@:/.git 00:34:21 > git --version # timeout=10 00:34:21 > git --version # 'git version 2.47.3' 00:34:21 using GIT_SSH to set credentials 00:34:21 Verifying host key using known hosts file, will automatically accept unseen keys 00:34:21 > git fetch --tags --force --progress --depth=1 -- git@:/.git +refs/heads/*:refs/remotes/origin/* # timeout=10 00:34:24 Avoid second fetch 00:34:24 Checking out Revision (origin/master) ``` 5 seconds. Total. But here is the catch: I cannot say for sure that the DNS changes caused this improvement. I still do not know the root cause. I had no idea what was causing the slowness. So I did the first thing I could think of: I looked at the DNS resolution path. Maybe the multi-hop DNS chain was adding latency to every git operation. --- ## The DNS resolution chain Our GitLab instance lives on an EC2 instance in AWS. It sits behind an Application Load Balancer (ALB). The domain `` is registered on Cloudflare in DNS-only mode, which means Cloudflare just returns the ALB's IPs. Every time the Jenkins runner pod on TKE resolves ``, this chain runs: ![DNS resolution chain before optimization: Jenkins CVM to TKE runner to gitlab-domain through Cloudflare DNS to AWS ALB to EC2 GitLab VM](https://anantafatur.dev/content/images/2026/08/dns-before.png) That's a DNS lookup through Cloudflare and a hop through the ALB before the git client can even start transferring data. For a shallow clone, the DNS overhead might be a significant fraction of the total time. But here is the thing. Our TKE cluster has private network connectivity to AWS. The GitLab VM has a private IP address. So why are we going through the public internet at all? ## Verifying the private path I hopped into a debug pod in the same TKE namespace and started testing. First, I checked the DNS resolution from the pod: ``` debug-pod:~# nslookup has address has address ``` As expected, the domain resolves to the public ALB IPs. Then I tested the GitLab VM's private IP directly: ``` debug-pod:~# for port in 22 80 443 8080 8443 2443; do timeout 2 nc -zv $port 2>&1 done Connection to 22 port [tcp/ssh] succeeded! Connection to 80 port [tcp/http] succeeded! nc: connect to port 443 (tcp) failed: Connection refused nc: connect to port 8080 (tcp) failed: Connection refused ``` Port 22 (SSH) and port 80 (HTTP) were open. Port 443 (HTTPS) was closed. This made sense because the ALB was terminating TLS and forwarding HTTP to the VM on port 80. My pipeline uses SSH for git clone, which hits port 22\. So if I could make the pod resolve `` to the VM's private IP, the SSH clone would work fine. But before going all-in on DNS, I wanted to rule out the basics. I ran a git clone from a jumphost to confirm that GitLab connectivity was fine from our Tencent network: ``` [root@ ~]# time git clone --depth 1 git@:/.git /tmp/speed-test Cloning into '/tmp/speed-test'... remote: Enumerating objects: 90, done. remote: Counting objects: 100% (90/90), done. remote: Compressing objects: 100% (78/78), done. remote: Total 90 (delta 11), reused 57 (delta 9), pack-reused 0 (from 0) Receiving objects: 100% (90/90), 93.84 KiB | 1000.00 KiB/s, done. Resolving deltas: 100% (11/11), done. real 0m2.564s user 0m0.032s sys 0m0.030s ``` 2.5 seconds. GitLab was reachable. The Tencent-to-AWS network path was fine. The problem was specific to how Jenkins was doing the clone. But there was a catch. The pipeline also uses the GitLab API over HTTPS with `GITLAB_HOST = 'https://'` for things like MR change detection and commit status updates. Those would hit port 443, which is closed on the VM. --- ## The fix: hostAliases plus HTTP The Kubernetes pod spec has a field called `hostAliases` (see the [Kubernetes docs](https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/)). It injects entries directly into `/etc/hosts` in all containers in the pod. The OS resolves the domain from `/etc/hosts` before ever hitting DNS. No Cloudflare, no ALB. I added this to the pod template YAML: ```yaml spec: hostAliases: - ip: "" hostnames: - "" ``` I could have used the ALB's public IPs from the `dig` output, but I wasn't sure if those IPs were fixed. AWS ALB IPs can change. The EC2 private IP is static, so I went with that directly. Safer. Now when the pod resolves ``, it goes straight to the VM's private IP over the internal network. For the GitLab API calls, I changed `GITLAB_HOST` from `https://` to `http://` in every pipeline file that referenced it. Since port 80 is open on the VM and the traffic stays within the private network, this felt safe enough for our setup. ```groovy // Uses HTTP because the pod resolves to the // GitLab VM's private IP via hostAliases, bypassing the ALB. // Port 80 is open on the VM over the private network. GITLAB_HOST = 'http://' ``` ## The new resolution path After the change, the DNS resolution chain for the pipeline pod is effectively gone: ![DNS resolution after optimization: gitlab-domain resolves via /etc/hosts hostAliases directly to VM private IP on ports 22 and 80](https://anantafatur.dev/content/images/2026/08/dns-after.png) No Cloudflare DNS. No ALB. No internet egress. Every git operation resolves in microseconds from the local `/etc/hosts` file. --- ## Closing I still do not know exactly what was making the git clone take 12 minutes. There might be something else going on with the network or the GitLab server itself. The DNS optimization was the first thing I thought to try, and honestly, it was worth a shot. The `hostAliases` approach is simple, but it has a tradeoff. If the GitLab VM's private IP ever changes, the pipeline will break until someone updates the YAML template. In my case, the VM has a fixed private IP, so this is unlikely. But it is worth documenting. While I was at it, I also updated the Jenkins URL in the cloud configuration to use the CVM's private IP instead of the public domain. This way the TKE runner connects to the Jenkins master via JNLP over the private network too. Same idea: keep internal traffic off the public DNS path. For now, the git clone issue is fixed. But I still do not know what actually fixed it. Before I tried the DNS approach, I restarted the Jenkins and nginx containers and the issue went away. However, on another occasion, restarting the containers did nothing. So I tried this DNS approach. Because the error is intermittent, it is hard to reproduce. If it comes back, my next move is to upgrade the Jenkins Git plugin version. Maybe there is a bug there. ### Monitoring Docker Images with Diun and SMTP Notifications URL: https://anantafatur.dev/diun-docker-image-monitoring/ Last updated: 2026-08-22T14:29:40.000Z I run a few Docker containers on my server: Ghost, MariaDB, Beszel, Docmost, Vaultwarden, and Smokeping. I want to keep them updated. Not for new features. I just want to reduce the CVE risk. A stale container image is an open invitation for problems, and I do not want to check Docker Hub manually every week just to see if there is a new patch. So I needed a tool that watches my Docker images and tells me when a new version is available. I settled on Diun, a lightweight container image update notifier by crazy-max. It does one thing and does it well: it checks registries and notifies you when your images have updates. Here is how I set it up, what broke, and how I fixed it. --- ## The setup I run everything with Docker Compose. Diun itself runs as its own container, mounts the Docker socket, and watches a list of images I define in a config file. No labels on my containers. Just a simple `diun.yml` that lists exactly what I care about: ```yaml - name: ghost:alpine - name: mariadb:lts-noble - name: henrygd/beszel:latest - name: docmost/docmost:latest - name: vaultwarden/server:alpine - name: linuxserver/smokeping:latest ``` I use floating tags like `ghost:alpine`, `mariadb:lts-noble`, and `vaultwarden/server:alpine`. These track the latest version within a major release or variant. The digest changes whenever the upstream maintainer pushes a new build, and Diun catches it. This is exactly what I want. I get notified about updates without worrying about pinned versions going stale. ## Installing Diun Diun has a simple compose file. I use the File provider so it reads my `diun.yml` directly: ```yaml name: diun services: diun: image: crazymax/diun:4.33 command: serve volumes: - "./data:/data" - "./diun.yml:/data/diun.yml" - "/var/run/docker.sock:/var/run/docker.sock" environment: - "TZ=Asia/Jakarta" - "DIUN_WATCH_WORKERS=20" - "DIUN_WATCH_SCHEDULE=0 */6 * * *" - "DIUN_WATCH_JITTER=30s" - "DIUN_PROVIDERS_FILE_FILENAME=/data/diun.yml" labels: - "diun.enable=true" restart: always ``` The File provider reads `diun.yml` and watches exactly those images. I also keep the `diun.enable=true` label on Diun itself so it monitors its own image. That way I know when a new Diun release is available too. ## Adding SMTP email notifications Diun supports many notification backends: Discord, Telegram, Slack, webhooks, and even plain SMTP email. I wanted email because I do not need real-time alerts. I just want to see an update notification in my inbox when a new image is available. I use SMTP2Go for my SMTP. It is a free SMTP relay service that gives you 1,000 emails per month. I [already set it up](https://anantafatur.dev/how-to-get-a-free-smtp-for-your-personal-homelab-using-smtp2go/) for my homelab before, so I had the credentials ready. I added them to a `.env` file so they are not hardcoded in the compose file: ``` ## SMTP DIUN_NOTIF_MAIL_HOST=smtp.example.com DIUN_NOTIF_MAIL_PORT=587 DIUN_NOTIF_MAIL_SSL=false DIUN_NOTIF_MAIL_INSECURESKIPVERIFY=false DIUN_NOTIF_MAIL_USERNAME=diun@example.com DIUN_NOTIF_MAIL_PASSWORD=changeme DIUN_NOTIF_MAIL_FROM= DIUN_NOTIF_MAIL_TO= ``` Then referenced those variables in the compose file: ```yaml environment: - "DIUN_NOTIF_MAIL_HOST=${DIUN_NOTIF_MAIL_HOST}" - "DIUN_NOTIF_MAIL_PORT=${DIUN_NOTIF_MAIL_PORT}" - "DIUN_NOTIF_MAIL_SSL=${DIUN_NOTIF_MAIL_SSL}" - "DIUN_NOTIF_MAIL_INSECURESKIPVERIFY=${DIUN_NOTIF_MAIL_INSECURESKIPVERIFY}" - "DIUN_NOTIF_MAIL_USERNAME=${DIUN_NOTIF_MAIL_USERNAME}" - "DIUN_NOTIF_MAIL_PASSWORD=${DIUN_NOTIF_MAIL_PASSWORD}" - "DIUN_NOTIF_MAIL_FROM=${DIUN_NOTIF_MAIL_FROM}" - "DIUN_NOTIF_MAIL_TO=${DIUN_NOTIF_MAIL_TO}" ``` Docker Compose picks up the `.env` file automatically from the same directory. No extra wiring needed. ## The gotchas Of course, things did not work on the first try. Here is what I hit. ### The FROM field only accepts a plain email I tried to use a display name format: ``` DIUN_NOTIF_MAIL_FROM=My Name ``` Diun rejected it: ``` Key: 'Config.Notif.Mail.From' Error:Field validation for 'From' failed on the 'email' tag ``` The `from` field only accepts a plain email address, not a display name. I changed it to just a plain email address and it worked. ### The watchRepo incident Here is the big one. I initially tried Diun's Docker provider with labels on my containers. But I learned that with pinned tags, Diun only checks the digest of that specific tag. If the maintainer rebuilds the same tag, I get notified. But if a new version comes out, Diun does not see it because it only watches the tag I am pinned to. So I thought: let me enable `watchRepo`, which scans all tags in the repository: ```yaml - "DIUN_DEFAULTS_WATCHREPO=true" ``` Bad idea. Very bad idea. I restarted Diun and my inbox exploded. Diun scanned every single tag in the repositories, found every version that was "new" relative to my database, and sent me an email for each one. I got dozens of emails in seconds. I immediately took the container down. Do not enable `watchRepo` unless you also configure `max_tags` and `include_tags` filters. Otherwise it will scan everything. --- ## Why I switched to the File provider After the `watchRepo` disaster, I switched to the File provider. Instead of adding labels to every container and relying on the Docker provider, I just list the images I want to watch in a `diun.yml` file. This is cleaner. I can see all the images I am watching in one place. And I use floating tags like `ghost:alpine` and `mariadb:lts-noble` that track the latest version within a major release, so Diun catches new builds without needing `watchRepo`. The File provider and the Docker provider are different things. The Docker provider scans running containers with labels. The File provider reads a config file. I use the File provider because it gives me explicit control over what I watch, and I do not have to remember to add labels to every new container. I just add a line to `diun.yml`. ## Testing the notification Diun has a built-in test command for notifications: ```sh docker compose -f diun/docker-compose.yaml exec -T diun diun notif test ``` This sends a test email using your configured SMTP settings. It confirmed that my SMTP config was working without needing to wait for an actual image update. When I ran the full check, the logs showed: ``` Found 6 image(s) to analyze Jobs completed added=0 failed=0 skipped=0 unchanged=6 updated=0 ``` All six images were up to date. No emails. Exactly what I wanted. ## Final config Here is the complete Diun compose file I ended up with: ```yaml name: diun services: diun: image: crazymax/diun:4.33 command: serve volumes: - "./data:/data" - "./diun.yml:/data/diun.yml" - "/var/run/docker.sock:/var/run/docker.sock" environment: - "TZ=Asia/Jakarta" - "DIUN_WATCH_WORKERS=20" - "DIUN_WATCH_SCHEDULE=0 */6 * * *" - "DIUN_WATCH_JITTER=30s" - "DIUN_PROVIDERS_FILE_FILENAME=/data/diun.yml" - "DIUN_NOTIF_MAIL_HOST=${DIUN_NOTIF_MAIL_HOST}" - "DIUN_NOTIF_MAIL_PORT=${DIUN_NOTIF_MAIL_PORT}" - "DIUN_NOTIF_MAIL_SSL=${DIUN_NOTIF_MAIL_SSL}" - "DIUN_NOTIF_MAIL_INSECURESKIPVERIFY=${DIUN_NOTIF_MAIL_INSECURESKIPVERIFY}" - "DIUN_NOTIF_MAIL_USERNAME=${DIUN_NOTIF_MAIL_USERNAME}" - "DIUN_NOTIF_MAIL_PASSWORD=${DIUN_NOTIF_MAIL_PASSWORD}" - "DIUN_NOTIF_MAIL_FROM=${DIUN_NOTIF_MAIL_FROM}" - "DIUN_NOTIF_MAIL_TO=${DIUN_NOTIF_MAIL_TO}" healthcheck: test: ["CMD", "diun", "healthcheck"] interval: 30s timeout: 5s retries: 3 start_period: 60s labels: - "diun.enable=true" restart: always ``` And the `diun.yml`: ```yaml - name: ghost:alpine - name: mariadb:lts-noble - name: henrygd/beszel:latest - name: docmost/docmost:latest - name: vaultwarden/server:alpine - name: linuxserver/smokeping:latest ``` --- ## Closing Diun is a simple tool that does one thing. It watches images and tells you when they change. The setup took maybe 30 minutes, and most of that was the `watchRepo` incident. It is also incredibly lightweight. Here is the resource usage on my server after letting it run for a while: ``` CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O 89ded4f5a824 0.04% 12.69MiB / 7.619GiB 0.16% 928kB / 446kB 1.79MB / 1.03MB ``` 12 MiB of RAM and barely any CPU. The image itself is only 82 MB on disk. It just sits there quietly, checks every 6 hours, and sends an email if something changes. Exactly what I wanted. For now, this works. My containers stay updated, my inbox stays quiet, and I sleep better knowing I will get an email when something changes. ### GitLab CI Pipelines Stuck After RDS Storage Filled Up URL: https://anantafatur.dev/gitlab-ci-pipelines-stuck-rds-storage-full/ Last updated: 2026-08-21T14:20:35.000Z We run GitLab self-hosted on an EC2 instance, with PostgreSQL on RDS and Redis on ElastiCache. About an hour before I started writing this, the EC2 root disk hit 100% and GitLab got very laggy. I resized the EBS volume, thought that was the fix, and moved on. Thirty minutes later, every CI pipeline was stuck at "created" and never moved to "pending" or "running." This is what actually happened --- ## The first symptom: a full disk and a laggy GitLab The EC2 instance's root volume filled up to 100%. GitLab got slow, as expected when a machine can't write logs or temp files. The obvious first move was to increase the EBS volume size, so I did that through the AWS console. After the resize, the volume status showed "optimizing," which is normal for EBS after a size change. It stays usable during that window, just with reduced IOPS/throughput until the background rebalancing finishes. I expected this to fix things. It didn't. ## What I ruled out early Thirty minutes after the EBS resize, every pipeline in GitLab was stuck at "created." Not "pending," not "running," just sitting there. No runner was picking anything up. First thing I checked: did the EBS resize actually apply at the OS level? Increasing an EBS volume's size in AWS does not automatically grow the filesystem, you usually need to run `growpart` and `resize2fs` (or `xfs_growfs`) on the instance itself. So I checked: ``` root@:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 437G 334G 103G 77% / ... /dev/nvme1n1 500G 435G 66G 87% /var/opt ``` Turned out the filesystem had already grown correctly. Root was at 77% with 103G free, and `/var/opt` (where GitLab's data lives) was at 87% with 66G free. So the local disk was not the bottleneck anymore. That ruled out the thing I originally suspected, which was a bit annoying because it meant the real cause was somewhere else. Pipelines being stuck at "created" (not even "pending") is actually a clue in itself. That status means GitLab has not finished processing the pipeline, which happens in a background job called `PipelineProcessWorker`, run by [Sidekiq](https://docs.gitlab.com/ee/administration/sidekiq/). It is not a runner problem at that stage, it is a Sidekiq/database problem. So I stopped looking at runners and started looking at Sidekiq. ## The investigation I checked if GitLab could even talk to the database: ``` root@:~# sudo gitlab-rails runner "puts ActiveRecord::Base.connection.active?" false ``` That's a bad sign. `false` means Rails cannot get an active connection to Postgres. Since we run Postgres on RDS and Redis on ElastiCache, this pointed away from the EC2 box and toward RDS itself. I checked RDS in the CloudWatch console and found the actual culprit: - `FreeStorageSpace` (minimum, last 3 hours): only **5GB** free, out of a 40GB allocated volume. - `OldestReplicationSlotLag`: climbed from about **5MB to 250MB** over the same window. - `FreeableMemory`: stable at 8 to 10GB the whole time, so memory was never the issue. ![](https://anantafatur.dev/content/images/2026/08/image-1.png) FreeStorageSpace (Minimum) ![](https://anantafatur.dev/content/images/2026/08/image-2.png) OldestReplicationSlotLag That `OldestReplicationSlotLag` number is the key clue. In Postgres, a [replication slot](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS) holds onto WAL (write-ahead log) files until whatever is supposed to consume that slot actually reads them. If nothing is consuming the slot, WAL just piles up on disk and never gets purged. That's exactly what was eating the RDS storage. The disk-full event on EC2 an hour earlier and the RDS storage pressure were probably two separate symptoms of the same underlying slowdown, not directly causing each other, but both draining resources around the same time. At 5GB free and actively climbing, RDS was close to going into a degraded state. I have read that Postgres on RDS can effectively stop accepting writes once storage is completely full, and I did not want to find out what that looks like in production. --- ## The fix First move: get some breathing room immediately. I bumped RDS allocated storage from 40GB to 60GB. This is a live, non-disruptive change (no failover, no reboot, for a [storage-only modification](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER%5FPIOPS.StorageTypes.html)): ``` aws rds modify-db-instance \ --db-instance-identifier \ --allocated-storage 60 \ --apply-immediately ``` I confirmed it landed cleanly by checking RDS events: ``` { "Message": "Applying modification to allocated storage", "Date": "2026-08-21T11:37:15+00:00" }, { "Message": "Finished applying modification to allocated storage", "Date": "2026-08-21T11:39:38+00:00" } ``` No storage-full event, no failure, just a clean resize. The instance briefly sat in "storage-optimization" status afterward, which is normal, the DB stays available and writable during that. After the resize, `FreeStorageSpace` went from 5GB minimum up to 25GB, and `OldestReplicationSlotLag` started coming back down instead of climbing. That was the real turning point. Before restarting anything on the GitLab side, I wanted to actually confirm Sidekiq needed a restart, instead of just restarting blindly. I tailed the Sidekiq log: ``` sudo gitlab-ctl tail sidekiq ``` and found jobs completing successfully, with real database writes: ``` "class":"CreatePipelineWorker", "job_status":"done", "duration_s":10.039718, "db_primary_write_count":2, "queue_duration_s":265.244705 ``` No `PG::ConnectionBad`, no `ActiveRecord::ConnectionNotEstablished`, nothing indicating a broken connection. The jobs were working, they had just been sitting in the queue for about 4 to 5 minutes (`queue_duration_s` around 265 to 271 seconds) before a worker picked them up. That is backlog, not breakage. GitLab had been effectively down for close to an hour, so a lot of jobs had queued up in Redis and Sidekiq was now working through all of it. So I did not restart Sidekiq or Puma. A restart at that point would have killed in-flight jobs and their retries, which would have made the backlog worse, not better. I just watched the queue duration and the pipeline list in the UI, and gave it time to clear the backlog. Pipelines started moving from "created" to "pending" to "running" on their own as the backlog cleared. --- ## Root cause The chain of events, as best I can piece it together: Something (I still don't know exactly what) drove a replication slot on RDS to stop being consumed, or to fall far behind. That slot held onto WAL files instead of letting Postgres purge them, which slowly ate away at `FreeStorageSpace`. RDS storage dropped to 5GB free, right around the same time the EC2 root disk also filled up and GitLab got laggy. With RDS under storage pressure, database writes from GitLab (including pipeline creation and processing) started failing or timing out. Jobs backed up in the Sidekiq/Redis queue instead of being processed. Even after I resized the EC2 EBS volume, none of that helped, because the actual problem was on the RDS side the whole time. Pipelines stayed stuck at "created" until RDS storage was increased. ### Debugging SSL EOF Errors in Kubernetes with Nginx and SonarQube URL: https://anantafatur.dev/debugging-ssl-eof-nginx-sonarqube/ Last updated: 2026-08-21T07:40:58.000Z ## The symptom Our GitLab CI pipeline started failing with a weird error. Every time it tried to assign a permission template to a user in SonarQube, the API call would hang for about 22 seconds, then fail: ``` curl: (56) OpenSSL SSL_read: OpenSSL/3.5.5: error:0A000126:SSL routines::unexpected eof while reading, errno 0 ``` The pipeline was calling the SonarQube API to set up a new project. The first API call, creating the permission template, worked fine every time. The second call, adding a user to the template, failed with that SSL EOF error. Every time. The project was already failing for other reasons (missing test reports), so nobody noticed this specific error for a while. But once the other issues were fixed, this one blocked the pipeline completely. --- ## What I ruled out first My first instinct was to blame the SonarQube server itself. Maybe it was overloaded. Maybe the API was slow. But the first API call always worked, so the server was reachable. It was just the second call that died. Next I checked the curl command. It was a simple POST with basic auth, nothing unusual. The token was valid. The URL was correct. I tried adding `--retry 3 --retry-delay 5` to the curl command. This helped sometimes, but the error still happened. Not a real fix. At this point I knew the problem was somewhere in the network path between the CI runner and the SonarQube server. The SSL EOF error meant the TLS connection was being torn down from the server side. But where? ## The network path Our SonarQube runs on Kubernetes, behind quite a few layers: ``` CI Runner (outside the cluster) → TLS → Cloud Load Balancer (Tencent CLB) → HTTP → NGINX Ingress Controller → HTTP → nginx sidecar container → HTTP → SonarQube (port 9000) ``` The TLS terminates at the cloud load balancer. The ingress has `tls: []` (empty), so everything after the LB is plain HTTP. The SSL EOF error meant the cloud LB was the one closing the TLS connection. But why was the LB closing the connection? ## The investigation I looked at the ingress annotations first. The timeouts looked fine: ```yaml nginx.ingress.kubernetes.io/proxy-connect-timeout: "30" nginx.ingress.kubernetes.io/proxy-read-timeout: "300" nginx.ingress.kubernetes.io/proxy-send-timeout: "300" ``` Five minutes for read and send. The API call was failing after 22 seconds. So the ingress controller was not the one timing out. Then I checked the nginx sidecar config. There is a sidecar container running nginx that sits between the service and SonarQube itself. It handles some header forwarding. Here is what the config looked like: ```nginx server { listen 8080; client_max_body_size 100m; location / { proxy_pass http://localhost:9000; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Port 443; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $http_host; } } ``` No proxy timeouts configured. This means nginx uses its defaults. `proxy_read_timeout` defaults to 60 seconds. That should be fine for a 22-second API call. But here is the thing I missed at first. The sidecar also had no `proxy_buffering` setting. And the default for `proxy_buffering` is `on`. ## The root cause Here is what actually happened, step by step: 1. The CI runner sends a POST to `add_user_to_template` on the SonarQube API. 2. The request goes through the cloud LB, the ingress controller, and the nginx sidecar to SonarQube. 3. SonarQube starts processing. This particular API call takes about 20 seconds (permission recalculation, maybe ES updates, I am not sure). 4. While SonarQube is processing, **no response data is flowing back**. The nginx sidecar has `proxy_buffering on`, so it is waiting to buffer the entire response before forwarding it. The ingress controller also has `proxy_buffering on` (the default), so it is waiting too. 5. The cloud LB sees **zero bytes** on the connection for 20+ seconds. Its idle timeout kicks in (probably 15-20 seconds, the default on Tencent CLB). 6. The LB closes the TCP connection to the CI runner. The TLS session is torn down. 7. The CI runner's OpenSSL tries to read more data and gets a TCP FIN. `SSL_read: unexpected eof while reading`. The key insight is that the cloud LB does not care about application-level timeouts. It only cares about whether data is physically flowing on the TCP connection. With buffering enabled at both nginx layers, the connection was completely silent for the entire duration of SonarQube's processing. --- ## The fix Two changes, both in the nginx sidecar ConfigMap and the ingress annotations. First, the sidecar. I added explicit proxy timeouts and disabled buffering: ```nginx location / { proxy_pass http://localhost:9000; proxy_read_timeout 300s; proxy_connect_timeout 30s; proxy_send_timeout 300s; proxy_buffering off; proxy_http_version 1.1; proxy_set_header Connection ""; # ... existing headers ... } ``` Second, the ingress annotation: ```yaml nginx.ingress.kubernetes.io/proxy-buffering: "off" ``` ## Why this works With `proxy_buffering off`, the moment SonarQube starts sending the HTTP response (even just the status line and headers), both nginx layers immediately forward those bytes upstream. The cloud LB sees data flowing, resets its idle timer, and keeps the TLS connection alive. Before the fix, the "no data" window was SonarQube processing time plus nginx buffering time. After the fix, it is just the SonarQube processing time. As soon as SonarQube responds, data starts flowing. The `proxy_read_timeout 300s` on the sidecar ensures it waits up to five minutes for SonarQube to respond, matching the ingress `proxy-read-timeout`. The `proxy_http_version 1.1` and empty `Connection` header enable HTTP keepalive between the sidecar and SonarQube, so connections do not get torn down after each request. ### Upgrading SonarQube from Community to Enterprise URL: https://anantafatur.dev/upgrading-sonarqube-community-to-enterprise/ Last updated: 2026-08-20T08:20:29.000Z Last week I upgraded our SonarQube from `26.7.0.124771-community` to `2026.4.1-enterprise`. The upgrade itself was simple, but I hit a few problems along the way. A server ID mismatch from a bad database clone decision, and a pod restart issue that took some debugging. Here is how it went. ## Cleaning up before the upgrade Our license covers 20 million lines of code. Before any upgrade, we needed to free up space. A coworker took care of this part, so I will describe the flow briefly. He exported the full project list from SonarQube, then filtered for projects that had not been analyzed in more than two years. From that list, he sorted by the largest lines of code. For each candidate, he verified the repository's last commit date and confirmed with the app owner that the project was safe to delete. After getting confirmation, he deleted the projects from SonarQube. This freed up enough lines of code for the new license. --- ## The server ID and license problem The SonarQube license is bound to the server ID. So before upgrading the edition, you need to share your server ID with the vendor, they generate the license, and you apply it. But here is the question. What happens if you upgrade the edition first, without the license applied? I tested this. The answer is that your pipeline stage still passes. The scanner reports `ANALYSIS SUCCESSFUL`. But the code is not actually analyzed. The SonarQube API returns this: ``` "errorMessage": "Analyses suspended. Please set a valid license for the Edition you installed." ``` It is not a pipeline blocker. It silently fails. That is the kind of thing you only notice when you go looking for the analysis results and find nothing. So the strategy was clear. Stay on the current Community edition, share the server ID, get the license, then upgrade to Enterprise and inject the license. Everything went fine at first. I got the license, updated the image tags, and moved on. ## The database clone mistake Here is where I messed up. I have written before about [upgrading SonarQube from 10.6.0 to 26.7.0](https://anantafatur.dev/upgrading-sonarqube-from-10-6-0-to-26-7-0/), and my usual pattern is to clone the database before any upgrade. So I did the same thing here. I created a new database: ``` CREATE DATABASE sonarqube_enterprise WITH TEMPLATE sonarqube_community; ``` Then I changed the JDBC URL in the Helm values to point to the new database, `sonarqube_enterprise`, and updated the image tags to the Enterprise version. The instance came up. Everything looked normal. But then I realized the problem. Changing the JDBC URL changed the server ID. The license I had was generated for the old server ID, the one tied to the `sonarqube_community` database name. This was a facepalm moment. I had created a new database with a new server ID, but the license was for the old one. The fix was simple enough. I reverted the JDBC URL back to `sonarqube_community`, kept the Enterprise image tags, and injected the license. The server ID matched, and everything worked. Looking back, the mistake was obvious. The database name is part of the JDBC URL, and the JDBC URL is part of what generates the server ID. I should have known that. ## What I learned about the upgrade flow If I were doing this again, here is the order I would follow. First, rename the existing database to `sonarqube_enterprise` while still on the Community edition. Then, get the server ID from this renamed instance and share it with the vendor. Once you have the license, create a backup database for safety, but do not change the JDBC URL. Then upgrade the image tags to Enterprise and apply the license. The key insight is simple. Get the server ID after the database is named what you want it to be named long-term. Do not change the JDBC URL after the license is generated. It is funny, really. The database is now named `sonarqube_community` but running the Enterprise edition. The name does not match the edition at all. But it works, and the server ID is stable. --- ## The pod restart mystery After the upgrade was done, I ran a test pipeline that uses this SonarQube instance for code scanning. The SonarQube scan stage passed. But then I noticed the SonarQube pod was restarting. Every time I applied a Helm change, the pod would shut down and restart. The logs showed it was receiving a stop signal. At first, I thought something was wrong with the application. Maybe the Enterprise edition was crashing under load. ### What I ruled out early I checked the application logs. No out-of-memory errors. No Java exceptions. The application was healthy. The shutdown was graceful, not a crash. The clue was in the exit code. The pod was exiting with code 130, which is a SIGINT. The Dockerfile for the SonarQube image sets `STOPSIGNAL SIGINT`. Kubernetes was sending this signal during `helm upgrade` rolling updates. The pod was not crashing. The StatefulSet controller was terminating the old pod to start a new one, every time I applied a Helm change. The logs I was seeing were just the graceful shutdown of the old pod. ### Resource tuning Even though the restart was not an application crash, the pod was clearly under-provisioned. The default resource limits were too tight for the Enterprise edition. The Community edition ran fine on the same defaults, but the Enterprise edition is heavier and needs more headroom. I adjusted the resources. Here is what I changed: ![Resource changes before and after the Helm upgrade](https://anantafatur.dev/content/images/2026/08/resource-changes.png) The final memory budget for the 16Gi pod looks like this: ![Memory budget breakdown for the 16Gi pod](https://anantafatur.dev/content/images/2026/08/memory-budget.png) After these changes, the pod stabilized. The CI/CD pipeline can now run project analyses without memory pressure. --- ## Closing The Enterprise edition of SonarQube is heavier than the Community edition. The resource requirements are real, and the default Helm values will not cut it. If you are upgrading, budget for at least 16Gi of memory and 8 CPU cores for the pod. The database clone mistake was entirely avoidable. I got lucky that reverting the JDBC URL was enough to fix it. If I had already deleted the old database, I would have needed to go back to the vendor for a new license. That would have been embarrassing. ### I Automated Google Index Status Checks for My Blog URL: https://anantafatur.dev/automate-google-index-status-check-sitemap/ Last updated: 2026-08-17T15:37:30.000Z I had a long weekend recently. You know those weekends where you tell yourself you will relax, but instead you end up writing five blog posts? Yeah, that. Normally, after I publish a post, I go to Google Search Console, paste the URL into the inspection tool, and click "Request Indexing." I do this every time. It takes maybe 30 seconds per post. Not a big deal when you publish once a week. But after a long weekend with multiple posts, it adds up. I am a DevOps engineer, not an SEO person. I had no idea if Google even had an API for this. I just knew the manual workflow was getting old. So I asked AI to write the code for me. --- ## I did not know the API existed Here is the thing. I use Google Search Console almost daily. I inspect URLs, I check performance reports, I look at crawl stats. I have been doing this for months. And somehow I never noticed that there is an API for all of it. The AI told me there are two APIs I needed: **Search Console URL Inspection API**. Checks whether a URL is indexed, when it was last crawled, and if there are any issues. This is the API version of the inspection tool I was using manually. **Indexing API**. Submits URLs for indexing. The `URL_UPDATED` notification type tells Google to re-crawl a page. This is the API version of the "Request Indexing" button. Honestly, I felt a bit dumb. I had been doing this by hand for months. But in my defense, when you are deep in infrastructure work, you do not think about automating the SEO side of things. You just click the button and move on. ## The script the AI wrote The AI built a Python CLI tool that does three things: 1. Fetches my sitemap (`https://anantafatur.dev/sitemap-posts.xml`) and extracts all article URLs 2. Sends each URL to the Search Console Inspection API to check its index status 3. Sends the unindexed URLs to the Indexing API to request indexing I gave it my sitemap URL and my Google service account key. It handled the rest. The code is on [GitHub](https://github.com/anantafaturdev/sitemap-index-automation) if you want to see it. The CLI has four commands: ``` python main.py check # just inspect, no submission python main.py submit # inspect first, then submit unindexed ones python main.py submit-all # submit everything regardless of status python main.py check-submit # same as submit ``` It also has a `--urls-only` flag to test the sitemap parsing without making any API calls. I used that first to make sure it was pulling the right URLs before burning quota. ## Setting up the Google Cloud side The AI gave me clear instructions for the setup. I needed to: 1. Enable both APIs in my Google Cloud project 2. Create a service account and download the JSON key 3. Add the service account as an owner of my property in Search Console The last step matters. The Indexing API requires the service account to be an owner, not just a user with full permissions. I learned that the hard way. I used the domain property format (`sc-domain:anantafatur.dev`) because it covers all protocols and subdomains. If you use a URL-prefix property, the API calls need to match that format exactly. ## The first run failed I ran the script for the first time and immediately got an error: ``` Google Search Console API has not been used in project before or it is disabled. ``` I had enabled the Indexing API, but I forgot to enable the Search Console API. They are two separate APIs. Two separate toggle switches in the Cloud Console. I only enabled one. Fixed that, waited a few minutes for propagation, and ran it again. ## The actual results Here is what came back after fixing the matching: So 45 out of 58 are indexed. 9 are not. 1 threw a server error. The unindexed ones are mostly older articles. I probably forgot to manually submit them when I published. The one with the server error is my most recent article, published just a day ago. Google might have tried to crawl it while my server was under load. I will re-check it later. Not great, but also not terrible. I was honestly expecting worse. ## Submitting the unindexed ones The `submit` command checks all URLs first, then submits only the ones that are not indexed. It uses the Indexing API's `URL_UPDATED` notification type. The Indexing API has a quota of 200 URLs per day. My 9 unindexed articles barely make a dent. But if I had 500 articles with 200 unindexed, I would need to spread the submissions across multiple days. The script handles this with a configurable delay between requests. I did not bother resubmitting the already-indexed articles. Some of them have been updated with better SEO titles, but Google will eventually re-crawl them on its own. For a small personal blog, natural re-crawling is fast enough. I would rather save the quota for new content. --- ## Closing The AI wrote the whole thing. I gave it my sitemap URL and my service account key, and it built a working CLI tool with four commands, rate limiting, CSV/JSON reports, and proper error handling. I just had to do the Google Cloud setup and catch one string matching bug. I will probably run this once a month to catch any articles that slipped through. The code is on [GitHub](https://github.com/anantafaturdev/sitemap-index-automation) if you want to use it. Just replace the sitemap URL and service account key, and you are good to go. If you are a DevOps person like me who has been clicking buttons in Search Console without thinking about automation, this is your sign. There is an API for that. Or you can just ask AI to write the code. Either way works. ### A single trailing comma crashed our production pods URL: https://anantafatur.dev/trailing-comma-json-crashloop/ Last updated: 2026-08-17T14:23:58.000Z It was midnight. I got a mention in the group chat. A developer was asking why the production pods would not start. The error logs showed something about OpenTelemetry instrumentation in the stack trace, so the assumption was that the OTEL auto-injection was breaking things. Here is the error the pod kept printing every time it tried to come up: ``` 2026-07-16T23:43:45: PM2 log: App [starter:0] online ERROR {'event':'start up','error':'Error: unable to parse config file at /usr/src/app/node_modules/fwsp-config/index.js:64:18 at /otel-auto-instrumentation/...'} ``` After a few seconds the pod would exit, Kubernetes would restart it, and the cycle repeated. CrashLoopBackOff. --- ## The OTEL red herring Look at the stack trace again. Every other line mentions `otel-auto-instrumentation`: ``` at /otel-auto-instrumentation/node_modules/@opentelemetry/instrumentation-fs/build/src/instrumentation.js:171:31 at /otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/... ``` It was easy to point at OTEL. The Tencent Cloud TKE platform injects OTEL auto-instrumentation via the annotation `cloud.tencent.com/inject-nodejs`. Maybe the instrumentation was wrapping `fs.readFile` and breaking something? Maybe the OTEL sidecar was corrupting the file? I started by checking what actually changed in this deployment. I went through the source code commit history and the GitOps repo history. No OTEL configuration changes. No new sidecar versions. Nothing that would explain why OTEL suddenly started breaking things. The stack trace was a red herring. OTEL was just the messenger, not the problem. ## What I ruled out early The error came from `fwsp-config`, a package that reads a JSON config file at startup and passes it to `hydra-express`. The message "unable to parse config file" was clear enough. Something was wrong with the JSON. I checked the commit that was deployed. There were config file changes in it. But the UAT environment was running without issues, and the Docker image built successfully. What gives? Then I actually tried to parse the file locally: ```bash node -e "JSON.parse(require('fs').readFileSync('./config/config.prod.json','utf8'))" ``` It failed. `Expected double-quoted property name in JSON at position 62248 (line 1661 column 3)`. That was the smoking gun. ## The investigation The error pointed to line 1661 in a 1700-line config file. I opened it and found the `rolist` endpoint, the last entry in the `"apis"` object: ```json "apis": { "masherytoken": { ... }, "apigatetoken": { ... }, ... (60+ more endpoints) ... "rolist": { "url": "https:///api/v1/endpoint?msisdn=${msisdn}", "method": "GET", "headers": { "Content-Type": "application/json", "Accept": "application/json" }, "circuitBreakerSleepWindowInMilliseconds": 60000, "circuitBreakerRequestVolumeThreshold": 5, "circuitBreakerForceOpened": false, "circuitBreakerForceClosed": false, "circuitBreakerErrorThresholdPercentage": 60, "statisticalWindowLength": 60000, "statisticalWindowNumberOfBuckets": 10, "requestVolumeRejectionThreshold": 100, "timeout": 31000 }, }, "services": { ``` See the problem? Look at the closing brace of `rolist`. It has a comma after it. In JSON, a trailing comma after the last property of an object is not allowed. The parser reads `"timeout": 31000`, then `}`, then `,`. The comma tells the parser: "there is another property coming in the `apis` object." But the next thing is `}` closing the `apis` object, not a property name. The parser says: "I expected a double-quoted property name but found `}`." The fix is simple. Remove the comma: ```json "timeout": 31000 } }, "services": { ``` But here is the thing. I checked our other config files. Six out of eight files had the same trailing comma bug. Only config.test.json and config.uat.json were fine. ## Why UAT was fine This confused me at first. UAT was running the same config file without issues. The logs even showed it loaded its config file successfully. The answer was that the UAT Docker image was built from a different commit. The UAT branch had the correct JSON (no trailing comma). Sometime after the UAT image was built, someone edited the config files in the main branch and introduced the trailing comma. The UAT container had the old, correct version baked into its image. Production was built from the newer, broken version. ## Why the build did not catch it The Docker build does not validate JSON files. The `Dockerfile` does `COPY . .` and `npm ci`, neither of which checks whether your config files are valid JSON. The error only shows up at runtime when the app starts and `fwsp-config` calls `JSON.parse()`. This is a gap. If your app depends on JSON config files, the build pipeline should validate them. Otherwise you ship a broken image and only find out when the pod tries to start. ## Adding config validation to the pipeline I added a validation step to our CI/CD pipeline. Before the Docker build runs, it checks every JSON file under the `config/` directory using `jq`: ```bash function validate_config_json() { local context=${1:-} if [ ! -d "config" ]; then echo "No config directory found, skipping validation" return 0 fi local has_error=0 local file_count=0 if [ -n "$context" ]; then local config_file="config/config.${context}.json" if [ -f "$config_file" ]; then file_count=1 if jq . "$config_file" > /dev/null 2>&1; then echo " [PASS] Valid: $config_file" else echo " [FAIL] Invalid: $config_file" has_error=1 fi else echo " - File not found: $config_file" fi else for file in config/*.json; do [ -f "$file" ] || continue file_count=$((file_count + 1)) if jq . "$file" > /dev/null 2>&1; then echo " [PASS] Valid: $file" else echo " [FAIL] Invalid: $file" has_error=1 fi done fi echo "---" if [ "$file_count" -eq 0 ]; then echo "No JSON files found under config/ to validate" return 0 fi if [ $has_error -ne 0 ]; then echo "RESULT: $file_count file(s) checked, some are INVALID" return 1 fi echo "RESULT: $file_count file(s) checked, all valid" return 0 } ``` `jq .` is a quick way to validate JSON. If the file is valid, it prints the formatted output and exits 0\. If the file is invalid, it prints an error and exits non-zero. The function checks all files under `config/` and fails the pipeline if any of them are broken. The pipeline now runs `validate_config_json` before the Docker build step. If the validation fails, the pipeline stops. No broken image gets built. --- ## Closing The real mistake was not the trailing comma itself. Anyone can make a typo in a 1700-line JSON file. The mistake was that we had no automated check for it. We relied on the app to crash at runtime to tell us the config was broken, and by then it was already deployed. A one-line `jq .` check in the pipeline would have caught this before the image was built. It took five minutes to write the script and will save hours of debugging the next time someone accidentally adds a trailing comma. I also learned that checking the UAT environment is not enough. UAT and production can be running different versions of the same config file if the images are built from different commits. The only way to be sure is to validate the files at build time, every time. ### SSH Failing on Tencent Cloud CLB After GitLab Migration URL: https://anantafatur.dev/ssh-failing-tencent-clb-gitlab-migration/ Last updated: 2026-08-16T07:51:34.000Z We had just finished migrating a GitLab instance from AWS to Tencent Cloud. The server was up, the web UI loaded fine over HTTPS, and from the internal jumphost I could SSH into the CVM and clone repos without issues. Everything looked good. Then I tried cloning a repo from my local machine using SSH. Nothing. Connection refused. I ran a quick netcat check against the public CLB: ```bash nc -v 22 # connect to port 22 (tcp) failed: Connection refused ``` Port 443 and 80 were fine. Port 22 was dead. The web UI worked, but `git clone git@:org/repo.git` was completely broken. --- ## Ruling out the server The first thing I did was rule out the server itself. I jumped onto the internal jumphost and tested SSH directly to the CVM's private IP. It worked fine, so the GitLab service and SSH daemon on the CVM were running correctly. ```bash nc -zv 22 # Connection to 22 port [tcp/ssh] succeeded! ``` The problem was somewhere between the public internet and the CVM. That narrowed it down to the load balancers. ## The architecture I was working with Before I explain the fix, here is the setup. We run a multi-account Tencent Cloud environment with a dedicated network account that owns the public-facing resources. The devops account holds the internal workloads. The two accounts talk to each other through a CCN (Cloud Connect Network). The flow for a user hitting the GitLab server looks like this: ![Architecture diagram showing the traffic flow: User -> public CLB (network-core) -> CCN -> internal CLB (devops) -> gitlab-server CVM, with HTTPS on 443 and SSH on 22](https://anantafatur.dev/content/images/2026/08/diagram.png) So for HTTPS traffic, the path is: user hits the public CLB on port 443, TLS is terminated there with an SSL certificate, and the request is forwarded as plain HTTP to the internal CLB on port 80\. The internal CLB then routes it to the CVM based on the host header. For SSH, the path is the same structure but simpler: the public CLB receives TCP on port 22, forwards it to the internal CLB on port 22, which forwards it to the CVM on port 22\. No TLS to worry about, just a raw TCP pass-through. ### The first rehearsal was wrong This was actually the second rehearsal of this migration. In the first one, I made a mistake that cost me some time. I had created the CVM in a public subnet with a public IP, and I tested SSH by connecting directly to that public IP. It worked, so I thought the setup was correct. But the real architecture has the CVM behind a CLB in a private subnet, so my test was not testing the actual path users would take. Looking back, this was a waste of time. I was testing a configuration that did not match what we would ship. The second rehearsal, I made sure the CVM was in a private subnet and all traffic went through the CLBs. ## The missing listener So here is what happened. The existing public CLB, which we used for other devops services, only had HTTP (80) and HTTPS (443) listeners configured. No TCP listener on port 22. When I tried to SSH, the traffic hit the public CLB, found no listener on port 22, and dropped the connection. The internal CLB and the CVM were both fine. The problem was that the public CLB simply did not know what to do with TCP traffic on port 22. The fix was to create a dedicated public CLB just for the GitLab server, with a TCP listener on port 22 added alongside the existing HTTP and HTTPS listeners. This way, SSH traffic would be forwarded to the internal CLB, which would then pass it to the CVM. Here is the traffic flow after the fix: **HTTPS (web access and git clone over HTTPS):** ![HTTPS traffic flow: User to Public CLB on HTTPS 443, TLS terminated, forwarded as HTTP 80 via CCN to Internal CLB, then host-based routing to GitLab CVM on port 80](https://anantafatur.dev/content/images/2026/08/https-flow.png) **SSH (git clone and push):** ![SSH traffic flow: User to Public CLB on TCP 22, forwarded via CCN to Internal CLB on TCP 22, then to GitLab CVM on port 22](https://anantafatur.dev/content/images/2026/08/ssh-flow.png) The key difference is that the new public CLB now has a TCP listener on port 22\. Without it, the connection never reaches the internal CLB, let alone the CVM. --- ## Validating the fix After setting up the new CLB with the TCP listener, I ran connectivity tests from multiple points to make sure everything was wired correctly. ### Port-by-port connectivity First, I tested the old CLB to confirm the original problem: ```bash for port in 22 80 443; do nc -zv $port done ``` Port 22 timed out as expected. Ports 80 and 443 succeeded. This CLB was never meant to handle SSH, and it was not. Then I tested the new dedicated CLB: ```bash for port in 22 80 443; do nc -zv $port done ``` All three ports succeeded. Port 22 was now being forwarded through the internal CLB to the CVM. I also tested the CVM directly from the jumphost to confirm the baseline: ```bash for port in 22 80 443; do nc -zv $port done ``` Port 22 and 80 succeeded. Port 443 was refused, which is expected. The CVM serves GitLab over plain HTTP on port 80\. TLS termination happens at the public CLB, so the CVM itself does not listen on 443. ### End-to-end test from my machine To test the full path without waiting for DNS propagation, I added an entry to `/etc/hosts` pointing the GitLab domain to the new public CLB's IP. I got the IP from a DNS lookup on the CLB hostname: ```bash dig +short # -> ``` Then edited `/etc/hosts`: ``` ``` I ran a ping first. It showed 100% packet loss, but that is expected. ICMP is blocked by the security group and does not affect anything. Then I tested HTTPS with curl: ```bash curl -v https:// ``` The TLS handshake succeeded (TLSv1.2, the certificate CN matched the domain and was valid). I got a `302 Found` redirect to `/users/sign_in`. The server header was `nginx`, and GitLab-specific headers like `X-Gitlab-Meta` and `X-Request-Id` were present. The web path was working. Finally, SSH: ```bash ssh -T git@ # -> Welcome to GitLab, @! ``` The host key had changed from the old server, so I had to run `ssh-keygen -R ` first. After that, the connection went through. To confirm the data was actually coming from the migrated server and not the old production instance, I cloned a repo and checked `git log`: ```bash git clone git@:org/repo.git # -> Cloning into 'repo'... done (1781 objects, 563 KiB) ``` The latest commit was from July 16, while the production EC2 instance had commits up to July 19\. That three-day gap confirmed the `/etc/hosts` override was routing to the migrated Tencent CVM with slightly stale data, not the production EC2 instance. The clone was hitting the right target. --- ## Closing The root cause was simple: the CLB lacked a TCP listener on port 22\. But the real mistake was rushing into the first rehearsal without thinking through the architecture. I tested the CVM directly with its public IP instead of testing through the CLB. That test passed, but it was testing the wrong thing. It gave me false confidence and wasted a whole rehearsal cycle. If I had mapped out the actual traffic path first, I would have caught the missing listener before the first test. ### MariaDB Galera Cluster on hybrid cloud benchmark URL: https://anantafatur.dev/mariadb-galera-cluster-hybrid-cloud/ Last updated: 2026-08-16T04:45:29.000Z I needed a database that could survive a node failure. Not just survive, but keep working without anyone noticing. That means synchronous replication, multi-master, active-active. The obvious choice was MariaDB Galera Cluster \[5\], \[6\], \[7\]. But I did not want to put everything in one place. If AWS us-east-1 goes down, I wanted a node on my own hardware that could keep things running. The problem? Nobody had published real benchmarks for running Galera across a hybrid cloud setup. Plenty of papers look at Galera in homogeneous environments \[5\], \[8\]. Plenty of papers talk about hybrid cloud in general \[9\], \[10\], \[11\], \[12\]. But nobody had put the two together. So I did. This blog post is adapted from the conference paper version I presented at [8th ISRITI 2025](https://isriti.utdi.ac.id/): same data, less formal. This was actually my skripsi, my final project during college. Is it a simple study? Yes. I really did not care about the challenge. I just wanted to finish my degree as fast as possible so I could start getting real work experience. This was also my first paper. Even though my advisor helped proofread it, I am sure it still has some consistency issues. Sorry about that. --- ## The setup I built a three-node cluster. Two nodes on AWS EC2 (t3.medium: 2 vCPUs, 4 GB RAM, 32 GB gp3 SSD). One node on my own Proxmox VE server with the same specs. All three nodes ran MariaDB 10.11.8 with Galera synchronous replication. To connect the AWS and Proxmox nodes, I used WireGuard VPN. This kept the traffic encrypted and put all three nodes on the same private network. I also deployed MariaDB MaxScale 24.02.5 on one of the AWS nodes to handle load balancing, read-write splitting, and automated failover \[8\]. Honestly, the WireGuard tunnel was the biggest unknown. I knew it would add latency. I just did not know how much it would hurt. For benchmarking, I used SysBench 1.0.20 with three OLTP workloads: read-only, write-only, and mixed read-write. Each test ran for 120 seconds, with thread counts from 1 to 192\. I measured transactions per second (TPS) and 95th percentile response time \[13\], \[14\], \[15\], \[16\]. I tested three configurations: hybrid (AWS + Proxmox), AWS-only, and Proxmox-only. The single-cloud tests gave me the baseline. The hybrid test told me what I was actually paying for the cross-cloud redundancy. ## Read-only: AWS dominates The read-only workload was the clearest story. AWS-only averaged 2,131.92 TPS. Proxmox-only averaged 1,303.27 TPS. That is a 63.5% gap. AWS peaked at 2,663.53 TPS at 16 threads. Proxmox peaked later, at 1,662.84 TPS at 144 threads, but never got close. The hybrid setup sat in the middle: 1,203.06 TPS average. That is 43.6% lower than AWS-only. The WireGuard overhead is real. Every read that hits the Proxmox node has to cross the VPN, and that shows up in the numbers. Latency told the same story. AWS-only had a minimum 95th percentile of 4.82 ms at 1 thread. Hybrid started at 19.90 ms. By 192 threads, AWS was at 116.80 ms, hybrid at 170.34 ms. ![Line chart comparing read-only TPS and 95th percentile latency for AWS-only, Proxmox-only, and hybrid configurations across thread counts from 1 to 192.](https://anantafatur.dev/content/images/2026/08/image1.png) **Figure 1: Read-only workload performance. AWS-only in blue, Proxmox-only in green, hybrid in orange. (a) TPS throughput, (b) 95th percentile latency.* ## Write-only: everyone suffers Write workloads are where synchronous replication really hurts. The Galera protocol requires every node to certify every write before it commits. That means the slowest node becomes the bottleneck. AWS-only averaged 104.42 TPS. Not great, but the best of the three. Proxmox-only averaged 76.29 TPS. The hybrid setup managed 91.74 TPS, which is only 12.1% lower than AWS-only. That surprised me. The write penalty for the hybrid setup was smaller than I expected. But here is the real headline: latency exploded for all three configurations. At 64 threads, the 95th percentile crossed 1,200 ms for every setup. At 192 threads, Proxmox-only hit 9,624.59 ms. AWS-only, the best case, still hit 6,247.39 ms. That is over 200 times higher than the minimum latency. This is not a cloud problem. It is a Galera problem. Synchronous replication just does not scale for writes. The consensus protocol caps your throughput no matter how good your hardware is. ![Line chart comparing write-only TPS and 95th percentile latency for AWS-only, Proxmox-only, and hybrid configurations across thread counts from 1 to 192.](https://anantafatur.dev/content/images/2026/08/image2.png) **Figure 2: Write-only workload performance. AWS-only in blue, Proxmox-only in green, hybrid in orange. (a) TPS throughput, (b) 95th percentile latency. Note the exponential latency growth beyond 64 threads across all configurations.* ## Mixed read-write: the practical workload The mixed workload is probably the most realistic test. It is a mix of reads and writes, similar to what a typical web application would generate. AWS-only averaged 579.99 TPS and peaked at 653.48 TPS. Proxmox-only averaged 439.72 TPS. The hybrid setup averaged 419.95 TPS. That is actually slightly lower than Proxmox-only on average throughput, but the hybrid had better latency consistency: 333.16 ms vs 359.11 ms for the 95th percentile. So the hybrid setup was not the fastest, but it was more predictable. For a workload that mixes reads and writes, the cross-cloud latency penalty is noticeable but not crippling. ![Line chart comparing mixed read-write TPS and 95th percentile latency for AWS-only, Proxmox-only, and hybrid configurations across thread counts from 1 to 192.](https://anantafatur.dev/content/images/2026/08/image3.png) **Figure 3: Mixed read-write workload performance. AWS-only in blue, Proxmox-only in green, hybrid in orange. (a) TPS throughput, (b) 95th percentile latency.* ## The write bottleneck is universal The single most important thing I learned from this whole experiment is the write scalability ceiling. It does not matter which configuration you pick. Once you push past 64 concurrent threads doing writes, the 95th percentile latency crosses 1.2 seconds. By 192 threads, it is measured in seconds, not milliseconds. This is the cost of synchronous replication. Galera uses a certification-based replication where every write must be certified by all nodes before it commits. More threads mean more contention. More contention means more waiting. The latency curve is exponential. If you are building a write-heavy application on Galera, this is the wall you will hit. It is not a cloud problem. It is a protocol problem. ## What this means If you want raw performance, put everything in AWS. The AWS-only setup gave 63.5% more read throughput and 36.9% more write throughput than Proxmox-only. The internal AWS networking is fast, and not having to cross a VPN helps a lot. But if you want high availability across multiple locations, the hybrid setup is a reasonable compromise. You lose 43.6% read throughput and 12.1% write throughput compared to AWS-only. But you gain a node that sits outside AWS. If the AWS region has an outage, your Proxmox node is still running. The hybrid model is best suited for read-heavy applications. Things like content delivery, e-commerce catalogs, or internal dashboards. The read penalty is significant but the writes are actually not that much worse than AWS-only. For write-heavy workloads like financial transactions or real-time analytics, I would not recommend hybrid. The write bottleneck is already bad enough without adding cross-cloud latency on top. --- ## What I would recommend (with caveats) This is not a universal truth. It is what I observed with one specific setup, one benchmark tool, and one database engine. But here is what I would say: If you need a Galera cluster and you care about performance above all else, go cloud-native. Put all your nodes in the same AWS region. You will get the best throughput and the lowest latency. If you need geographic redundancy and can tolerate a throughput hit, the hybrid model works. The read penalty is around 40%, the write penalty is around 10%. For many applications, that is an acceptable trade-off for having a node outside the cloud provider. If you are building a write-heavy application, think hard about whether synchronous replication is actually what you need. The write bottleneck is severe, and it applies to every configuration. You might be better off with asynchronous replication, or a different architecture entirely. ## What I did not test I did not measure RTT between the AWS and Proxmox nodes. I know the latency is there, but I cannot say exactly how much it contributes. I did not tune any Galera parameters for the hybrid network conditions. Maybe there are configs that reduce the cross-cloud penalty. I did not test semi-synchronous or asynchronous replication models, which could address the write scalability bottleneck. I only tested one instance type (t3.medium) on one cloud provider (AWS) with one on-premise setup (Proxmox VE). Different hardware would give different numbers. I did not test with real application workloads, only SysBench OLTP patterns. I did not test failover timing or measure how quickly MaxScale redirects traffic during a node failure, though I did confirm it works. --- ## References \[1\] P. Memarzia, H. Zhang, K. Ho, R. Grosman, and J. Wang, "GaussDB-Global: A Geographically Distributed Database System," in *2024 IEEE 40th International Conference on Data Engineering (ICDE)*, May 2024, pp. 5111–5118\. doi: 10.1109/ICDE60146.2024.00383. \[2\] S. Ferreira, J. Mendonca, B. Nogueira, W. Tiengo, and E. Andrade, "Benchmarking Consistency Levels of Cloud-Distributed NoSQL Databases Using YCSB," *IEEE Access*, vol. 13, pp. 63428–63438, 2025, doi: 10.1109/ACCESS.2025.3558923. \[3\] P. Aryan, R. Khatri, and V. Balakrishnan, "An Experimental Framework for Implementing Decentralized Autonomous Database Systems in Rust," Dec. 06, 2024, *arXiv*: arXiv:2412.05078\. doi: 10.48550/arXiv.2412.05078. \[4\] L. Zhang, T. Jia, M. Jia, and Y. Li, "LogDB: Multivariate Log-based Failure Diagnosis for Distributed Databases (Extended from MultiLog)," May 03, 2025, *arXiv*: arXiv:2505.01676\. doi: 10.48550/arXiv.2505.01676. \[5\] R. Shrestha and T. Tandel, "An Evaluation Method and Comparison of Modern Cluster-Based Highly Available Database Solutions," in *Proceedings of the 13th International Conference on Cloud Computing and Services Science*, Prague, Czech Republic: SCITEPRESS, 2023, pp. 131–138\. doi: 10.5220/0011714400003488. \[6\] A. A. E. Alflahi, M. A. Y. Mohammed, and A. Alsammani, "Enhancement of database access performance by improving data consistency in a non-relational database system (NoSQL)." \[7\] V. B. F. Gomes, M. Kleppmann, D. P. Mulligan, and A. R. Beresford, "Verifying Strong Eventual Consistency in Distributed Systems," *Proc. ACM Program. Lang.*, vol. 1, no. OOPSLA, pp. 1–28, Oct. 2017, doi: 10.1145/3133933. \[8\] R. Shrestha, "High Availability and Performance of Database in the Cloud - Traditional Master-slave Replication versus Modern Cluster-based Solutions," in *Proceedings of the 7th International Conference on Cloud Computing and Services Science*, Porto, Portugal: SCITEPRESS, 2017, pp. 413–420\. doi: 10.5220/0006294604130420. \[9\] F. Ullah, S. Dhingra, X. Xia, and M. A. Babar, "Evaluation of Distributed Data Processing Frameworks in Hybrid Clouds," Jan. 06, 2022, *arXiv*: arXiv:2201.01948\. doi: 10.48550/arXiv.2201.01948. \[10\] M. Waseem et al., "Containerization in Multi-Cloud Environment: Roles, Strategies, Challenges, and Solutions for Effective Implementation," July 08, 2025, *arXiv*: arXiv:2403.12980\. doi: 10.48550/arXiv.2403.12980. \[11\] M. M. Kondappan, "Cloud computing and its role in insurance platform integration," *World J. Adv. Res. Rev.*, vol. 26, no. 1, pp. 3700–3712, Apr. 2025, doi: 10.30574/wjarr.2025.26.1.1506. \[12\] A. Gurram, "The role of cloud architecture in modernizing enterprise platforms," *World J. Adv. Res. Rev.*, vol. 26, no. 1, pp. 3501–3511, Apr. 2025, doi: 10.30574/wjarr.2025.26.1.1436. \[13\] N. Husufa and I. Prihandi, "Optimizing JMeter on Performance Testing Using the Bulk Data Method," vol. 4, no. 2, 2022. \[14\] H. Li et al., "Online Timestamp-based Transactional Isolation Checking of Database Systems (Extended Version)," Apr. 02, 2025, *arXiv*: arXiv:2504.01477\. doi: 10.48550/arXiv.2504.01477. \[15\] D. Landau, J. Barbosa, and N. Saurabh, "eBPF-Based Instrumentation for Generalisable Diagnosis of Performance Degradation," May 19, 2025, *arXiv*: arXiv:2505.13160\. doi: 10.48550/arXiv.2505.13160. \[16\] D. Liyanage et al., "A Benchmark for Databases with Varying Value Lengths," Aug. 11, 2025, *arXiv*: arXiv:2508.07551\. doi: 10.48550/arXiv.2508.07551. ### ARM vs x86_64 on AWS: Cost-Performance of MariaDB OLTP URL: https://anantafatur.dev/arm-vs-x86-mariadb-oltp-aws/ Last updated: 2026-08-15T19:07:56.000Z I kept hearing the same claim: "ARM instances are just as fast as x86\_64 and 30% cheaper." AWS says it \[3\]. Cloud cost calculators repeat it. But cloud computing has changed how we deploy databases. The shift to on-demand infrastructure means architecture choices now have a direct dollar impact \[1\]. When I looked for hard numbers on databases, specifically MariaDB, I found almost nothing. Most benchmarks out there test raw compute or video encoding. Not a database where the CPU, the storage layer, and the transaction engine all interact in ways that are hard to predict. So I ran my own benchmarks. This is what I found. ## Why I did this I was working on a research project at Telkom University and we had a simple question: if you're deploying MariaDB for OLTP on AWS, should you go ARM (Graviton3) or x86\_64 (AMD EPYC)? This work was presented at [3rd ICoABCD 2025](https://icoabcd.asasi.or.id/). This blog post is adapted from the conference paper version: same data, less formal. The two big architectures on AWS are x86\_64 (historically Intel and AMD) and ARM (AWS Graviton) \[2\]. Graviton3 processors are built on ARMv8.4 and are marketed as cheaper and more efficient \[3\]. AMD EPYC represents the established x86\_64 family with proven compatibility and reliability \[4\]. For OLTP workloads, the database is usually the bottleneck. OLTP systems power things like fraud detection, e-commerce, and social media \[6\]. They need consistent throughput and low latency under lots of concurrent requests \[7\]. So the architecture you pick matters a lot. Honestly, I expected ARM to win. The marketing made it sound that way. And spoiler: it did win on cost. But the performance story was more nuanced than I expected. --- ## The setup I ran all tests in us-east-1\. Four EC2 instances total, all in the same VPC, subnet, and availability zone to keep network latency out of the picture. The database was MariaDB 10.11.13, installed from the default Ubuntu 24.04 LTS package repo. MariaDB is a widely adopted open-source relational database \[8\], and it supports both ARM and x86\_64 without any code changes \[9\]. That made it a clean platform for this comparison. No custom tuning. I wanted a baseline that any engineer could reproduce without spending days tweaking configs. The only changes I made were bumping max\_connections to 512 and binding the server to 0.0.0.0 so the benchmark client could reach it. The loader was a separate EC2 instance running SysBench 1.0.20 with LuaJIT 2.1.0-beta3\. It sat in the same availability zone as the database. All traffic went over private IPs. I used the standard SysBench OLTP workloads: read-only, write-only, and mixed read-write \[5\]. The dataset was 50 tables, each with 100,000 rows. I tested from 1 thread up to 192 threads, stepping by 16\. Each test ran for 120 seconds. ## The instances I compared I picked one comparison that reflects a real decision an architect would make: latest-generation compute-optimized instances from each architecture. **c7g.large (ARM):** AWS Graviton3, 2 vCPUs, 4 GiB RAM, $0.0725/hr \[10\]. **c7a.large (x86\_64):** AMD EPYC, 2 vCPUs, 4 GiB RAM, $0.10264/hr \[10\]. Same vCPUs, same RAM, same network bandwidth. The only real difference is the CPU architecture. And the price. The ARM instance is about 29.4% cheaper per hour. Storage was gp3 across the board: 8 GB, 3,000 IOPS, 125 MB/s throughput. --- ## Read-only: ARM has a real lead at low concurrency ARM had a clear advantage in single-threaded read performance. At 1 thread, ARM processed 50,080 transactions vs x86\_64's 40,929\. That's a 22.4% gap. But as concurrency went up, the gap disappeared. At 16 threads, both hit around 209,000 transactions. From 32 threads upward, the two architectures were basically tied. At 192 threads, x86\_64 actually pulled slightly ahead: 205,960 vs 203,785. Latency scaled predictably for both. At 1 thread, ARM had 3.89 ms (95th percentile) vs x86\_64's 4.57 ms. By 192 threads, both were around 290 ms. So for read-heavy workloads, ARM wins if you're running at low concurrency. But if you're saturating the instance, it doesn't matter. ## Write-only: nobody wins, writes are just slow Write workloads are harder on databases. Lock contention, disk I/O, transaction serialization. All of that shows up in the numbers. ARM had a small throughput advantage in the mid-range (32 to 80 threads), but nothing worth bragging about. At 1 thread, ARM did 7,001 transactions vs x86\_64's 6,428\. At higher concurrency, the gap was negligible. Latency for write workloads was much worse than read. At 1 thread, both were around 25 ms. By 48 threads, latency passed 1,200 ms. By 192 threads, it hit over 10 seconds for both. At this point, the CPU isn't the bottleneck anymore. The database engine itself is the limit, handling locks and disk writes. The architecture barely matters. ## Mixed read-write: same story as read-only The mixed workload looked a lot like the read-only pattern. ARM had a small advantage at low concurrency (20,838 vs 19,219 transactions at 1 thread). At higher concurrency, both architectures converged. Latency sat between the read-only and write-only ranges. At 1 thread, iaround 8 ms for both. At 192 threads, around 1,000 ms. ![Line chart comparing total transactions and 95th percentile latency for ARM (Graviton3) and x86_64 (AMD EPYC) across read-only, write-only, and mixed read-write workloads at varying concurrency levels.](https://anantafatur.dev/content/images/2026/08/image9.png) **Figure 1: Total transactions and 95th percentile latency for both architectures across all three workloads. ARM and x86\_64 show nearly identical throughput and latency trends at high concurrency.* So here is the raw performance summary: ARM wins at low concurrency, especially for reads. At high concurrency, both architectures deliver the same throughput. The CPU stops being the bottleneck. Memory bandwidth, storage I/O, and database locking take over. If you only look at raw performance, the choice doesn't matter much unless you're running at low thread counts. But that's only half the story. ## The cost angle: ARM wins, and it's not close Raw performance is one thing. But in the cloud, you pay by the hour. An instance that's 5% slower but 30% cheaper might still be the better deal. I calculated the cost per transaction (CPT). Take the price per second, divide by transactions per second. Lower CPT means you get more work done for the same money. The ARM instance costs $0.0725/hr. The x86\_64 instance costs $0.10264/hr. That's a 29.4% price difference. ![Line chart showing cost-per-transaction trend for the mixed read-write workload across concurrency levels. The ARM curve runs consistently below the x86_64 curve.](https://anantafatur.dev/content/images/2026/08/image10.png) **Figure 2: Cost-per-transaction ratio for the mixed read-write workload. ARM (Graviton3) is consistently cheaper per transaction across all concurrency levels.* ARM had a lower CPT at every concurrency level. At 1 thread, ARM was 34.8% cheaper per transaction. That's the compounding effect of better single-threaded performance plus lower hourly cost. Both advantages stack in the same direction. At higher concurrency, the CPT advantage stabilized at about 29.4%. This makes sense. At high concurrency, both architectures have similar throughput, so the cost advantage comes almost entirely from the lower hourly price. The 29.4% CPT gap matches the 29.4% price gap almost exactly. The same trend held for read-only and write-only workloads. I'm not showing those charts to keep things short, but the pattern is consistent. ARM gives you more transactions per dollar, period. ## Why ARM wins on cost but not raw performance This is the part that surprised me. ARM wins on cost, but the raw performance is basically a tie at high concurrency. Why? I think the answer is simple. The ARM instance is 29.4% cheaper. And at high concurrency, it delivers the same throughput. So the cost advantage translates directly into a CPT advantage. There's no magic here. It's just cheaper hardware doing the same work. The single-threaded performance advantage is more interesting. ARM does better at low concurrency across all three workloads. The Graviton3's architectural optimizations seem to help when there's less contention. But once the system is saturated, the bottleneck shifts away from the CPU. Both architectures hit the same wall. I can't say whether the single-threaded advantage comes from the ARM ISA itself, from Graviton3's specific microarchitecture, or from software optimization. The experiment wasn't designed to isolate that. But as a practical matter, it doesn't matter. If you're running MariaDB on AWS, ARM gives you more for less. --- ## What I'd recommend (with caveats) This is not a universal truth. It's what I observed in one region, on one database engine, with one benchmark tool. But here's what I'd say: If you're deploying MariaDB for OLTP on AWS, go with ARM (Graviton3). You get better single-threaded performance, the same multi-threaded performance, and roughly 29% more transactions per dollar. There's no downside I could find. The only reason to stick with x86\_64 is if you have software that genuinely doesn't run on ARM. But MariaDB supports both architectures natively \[9\], so that's unlikely to be an issue. ## What I didn't test I only tested MariaDB. PostgreSQL or MySQL might behave differently. I used default database configs except for max\_connections. Maybe there are tuning parameters that favor one architecture. I only used gp3 storage with modest IOPS. Higher IOPS or provisioned storage might shift the bottleneck. I didn't look at energy efficiency. ARM instances are supposed to be more power-efficient, but I didn't have access to that data. I only tested in us-east-1\. Different regions might have different underlying hardware. I didn't test reserved instance pricing or long-term sustained workloads. On-demand pricing is the simplest comparison, but real deployments often use reserved or savings plans. --- ## References \[1\] A. V. Indukuri, "Cloud-native transformation: Architectural principles and organizational strategies for infrastructure modernization," *World J. Adv. Res. Rev.*, vol. 26, no. 1, pp. 3914–3926, Apr. 2025, doi: 10.30574/wjarr.2025.26.1.1467. \[2\] J. Tharwani and A. A. Purkayastha, "Cost-Performance Evaluation of General Compute Instances: AWS, Azure, GCP, and OCI," Dec. 04, 2024, *arXiv*: arXiv:2412.03037\. doi: 10.48550/arXiv.2412.03037. \[3\] "ARM Processor - AWS Graviton Processor - AWS," Amazon Web Services, Inc. Accessed: Aug. 20, 2025\. \[Online\]. Available: \[4\] "5th Gen AMD EPYC Processor Architecture White Paper." \[5\] F. Nirwansyah, "Comparison Performance Analysis of HDD, SSD and NVME for OLTP Database Server," *J. Syntax Admiration*, vol. 5, no. 10, pp. 4379–4388, Oct. 2024, doi: 10.46799/jsa.v5i10.1730. \[6\] J. Wang et al., "PolarDB-IMCI: A Cloud-Native HTAP Database System at Alibaba," *Proc. ACM Manag. Data*, vol. 1, no. 2, pp. 1–25, June 2023, doi: 10.1145/3589785. \[7\] Y. Sheng, A. Tomasic, T. Zhang, and A. Pavlo, "Scheduling OLTP Transactions via Machine Learning," May 29, 2019, *arXiv*: arXiv:1903.02990\. doi: 10.48550/arXiv.1903.02990. \[8\] R. G. Kula, B. A. Reid, and C. Treude, "Open Source at a Crossroads: The Future of Licensing Driven by Monetization," June 01, 2025, *arXiv*: arXiv:2503.02817\. doi: 10.48550/arXiv.2503.02817. \[9\] S. Kaiser, Md. S. Haq, A. S. Tosun, and T. Korkmaz, "Container Technologies for ARM Architecture: A Comprehensive Survey of the State-of-the-Art," *IEEE Access*, vol. 10, pp. 84853–84881, 2022, doi: 10.1109/ACCESS.2022.3197151. \[10\] "EC2 On-Demand Instance Pricing," Amazon Web Services, Inc. Accessed: Aug. 21, 2025\. \[Online\]. Available: ### Umami recorder.js not ingesting data on Ghost CMS URL: https://anantafatur.dev/umami-recorder-heatmap-debug/ Last updated: 2026-08-15T18:52:25.000Z My blog runs on Ghost 6.57.1 (alpine3.23) and I use a self-hosted Umami v3.3.0 instance at `analytics.anantafatur.dev` for web analytics. The normal tracker worked fine. Pageviews, custom events, all good. But when I tried to enable replays and heatmaps, nothing showed up. The recorder script loaded with a 200 OK, but the dashboard stayed empty. ## The setup Both scripts are loaded through Ghost's Code Injector footer. I had them like this: ```html ``` I also had the Replays and Heatmaps toggles turned on in the Umami admin panel. I double checked the API endpoint to confirm: ```json { "enabled": true, "replayEnabled": true, "heatmapEnabled": true, "sampleRate": 1, "heatmapSampleRate": 1 } ``` Everything looked right. But no data. --- ## What I ruled out early First thing I checked was the script loading order. I thought maybe `recorder.js` needed to load before `script.js` to hook into the tracker. I tried swapping them. Did not help. The original order was correct: the recorder needs `window.umami` to exist first, so `script.js` should come before `recorder.js`. Next, I checked the Content Security Policy. My site is behind Cloudflare, and I have a CSP rule in place. Looking at the browser console, I saw two things missing. The `rrweb` library bundled inside `recorder.js` needs two things my CSP was not allowing. First, it uses `eval()` internally for DOM serialization, so `script-src` needed `'unsafe-eval'`. Second, it spawns blob Web Workers for canvas recording, which meant I needed a new `worker-src` directive. I added both. Final CSP: ``` default-src 'self' https: data: blob:; script-src 'self' https://cdn.jsdelivr.net https://cdnjs.cloudflare.com https://analytics.anantafatur.dev https://static.cloudflareinsights.com https://challenges.cloudflare.com 'unsafe-inline' 'unsafe-eval'; style-src 'self' https: 'unsafe-inline'; img-src 'self' https: data: blob:; connect-src 'self' https://analytics.anantafatur.dev https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://api.unsplash.com https://ghost.org https://challenges.cloudflare.com; frame-ancestors 'self' https://analytics.anantafatur.dev; worker-src 'self' blob: ``` There was also a `Content-Security-Policy-Report-Only` header with `connect-src 'none'` showing up in the console. It was report-only, so it was not actually blocking anything. I ignored it. CSP fixed. Still no data. --- ## The investigation At this point I was stuck. The recorder loaded fine, the API config was correct, the CSP was fixed. But the dashboard stayed empty. I needed to understand what the recorder was actually doing internally, and honestly, I was not about to read thousands of lines of minified JavaScript by myself. So I asked DeepSeek-v4-pro (the AI coding agent in my Zed editor) to help. It fetched both scripts from my Umami server, read through the minified source, and pointed out the critical piece. The recorder has a function that gates all data sending: ```javascript const P = () => e.umami?.getSession?.()?.cache; ``` And the function that sends data to the server checks this gate: ```javascript const W = (e, t, r) => { const s = P(); if (!s) return; // silently drops everything // ... fetch to /api/record }; ``` If `getSession().cache` is falsy, the recorder never sends a single byte. It also has a startup retry loop that checks the same gate every 100ms for 5 seconds. If the gate stays closed, it gives up silently. So I opened Chrome DevTools and checked what my tracker actually exposes: ```javascript window.umami // { track: f, identify: f } window.umami?.getSession?.() // undefined ``` There it was. The tracker only exposes `track` and `identify`. No `getSession` method. The recorder expects it, the tracker does not provide it, and the whole thing fails silently. ## The root cause DeepSeek-v4-pro also dug into the tracker source code. Here is how it sets up `window.umami`: ```javascript t.umami = { track: J, identify: P }; ``` That is it. Two methods. The session cache exists as a private variable `R` inside the tracker's closure, but it is never exposed. The recorder, on the other hand, was written expecting `getSession()` to return that cache. Both scripts are served from the same Umami v3.3.0 Docker image. This is a version mismatch bug inside the same release. The tracker and recorder shipped together but are incompatible with each other. ## The fix I could not modify the Umami source directly. It is served from the Docker container. But DeepSeek-v4-pro suggested a polyfill approach: add a script in my Ghost Code Injector footer that runs before the tracker scripts. The idea is simple. Use `Object.defineProperty` to intercept the `window.umami` assignment. When the tracker sets `window.umami`, inject `getSession` into it. The cache value needs to be the real session cache from the tracker's API responses, so the polyfill also monkey-patches `fetch` to capture it. DeepSeek-v4-pro wrote the polyfill and I dropped it into my Code Injector footer: ```html ``` Three things happen here. First, I patch `fetch` to intercept responses from `/api/send` and extract the real session cache. The tracker sends its first request, the server responds with a JWT cache, and my patch captures it. Second, I intercept the `window.umami` assignment. When the tracker sets it, I add `getSession()` that returns the captured cache. Third, I use `'1'` as a fallback for the initial request. The recorder checks `getSession().cache` for truthiness, and `'1'` is truthy. An empty string would not work because the recorder's gate is `if (!s) return`. The fallback is a bit hacky. The real fix would be for Umami to expose the method natively. But this polyfill gets the recorder running. ## Verification After deploying the polyfill, I checked the console: ```javascript window.umami.getSession() // { cache: 'eyJhbGciOiJIUzI1NiIs...', website: 'fdba9eea-5a40-4a24-927f-4d5672a9b7c8' } ``` The method exists. The cache is a real JWT. The recorder's gate opens. I browsed a few pages, waited a minute, and checked the Umami dashboard. Replays and heatmaps were both ingesting data. --- ## Closing The real mistake was assuming that if a script loads with 200 OK, it is working. The recorder fails silently. There is no console error, no network error, no warning. It just does nothing. I spent time on CSP and script ordering because those are the visible problems. The actual bug was hidden inside the minified source code. Having an AI coding agent that can fetch URLs, read source code, and trace execution paths saved me hours of tedious debugging. The CSP and script ordering fixes were things I could find on my own. But the real bug? That needed a second pair of eyes that was willing to read minified source. ### AWS Graviton vs Intel on Aurora PostgreSQL benchmark URL: https://anantafatur.dev/aws-graviton-vs-intel-on-aurora-postgresql-benchmark/ Last updated: 2026-08-14T19:37:22.000Z I kept hearing the same claim: "Graviton gives you up to 40% better price-performance than x86." AWS says it \[4\]. Blog posts repeat it. But when I looked for hard numbers on managed databases, specifically Aurora PostgreSQL \[6\], I found almost nothing. Most benchmarks out there test raw EC2 compute \[1\], \[4\] or serverless functions \[3\]. Not a managed database where the engine, the virtualization layer, and the architecture all interact in ways that are hard to predict. So I ran my own benchmarks. This is what I found. --- ## Why I did this I was working on a research project at Telkom University, and we had a simple question: if you're an architect picking an Aurora PostgreSQL instance today, should you go Graviton (ARM64) or Intel (x86\_64)? This work was presented at [14th ICoICT 2026](https://icoict.telkomuniversity.ac.id/). This blog post is adapted from the conference paper version with same data, less formal. The answer turns out to depend on which instance family you're looking at. Not just the architecture. The family and the generation matter just as much. Honestly, I expected Graviton to win across the board. The marketing made it sound that way. But the numbers told a different story. ## The setup I ran all tests in us-east-1 (Northern Virginia). The database was Aurora PostgreSQL 17.4, using the default parameter group. No custom tuning. I wanted a baseline that any engineer could reproduce without spending days tweaking configs. The loader was a t3.medium EC2 instance running Ubuntu 24.04 with pgbench 16.11\. It sat in the same availability zone as the database to keep network latency out of the picture. I initialized a single pgbench \[13\] database with scale factor 500\. That gives you about 8 GB of data, which is enough to overflow the buffer cache on smaller instances and force real I/O. Each test ran three times for 120 seconds with prepared statements. I took the mean of the three runs. The coefficient of variation stayed under 3% across all tests, which is consistent with what other cloud benchmarking studies have found \[14\]. I tested four concurrency levels: 16, 32, 64, and 128 clients. And three workload types: the default TPC-B-like mix, select-only, and write-intensive (simple-update). ## The instances I compared I picked two comparisons that reflect real decisions architects make. **General-purpose (M-series):** db.m7g.large (Graviton3) vs db.m7i.large (4th Gen Intel Xeon Scalable, Sapphire Rapids). Both have 2 vCPUs and 8 GiB RAM. Both sit on the AWS Nitro System. These are the latest generation from each architecture \[9\]. At the time of testing, db.m7i.large cost $0.178/hr and db.m7g.large cost $0.168/hr in us-east-1\. A 6% difference. **Burstable (T-series):** I compared db.t4g (Graviton2) against db.t3 (Intel) \[10\]. This is not a generation-matched comparison. The T4g instances are newer than the T3 instances they commonly replace \[8\]. But that's the point: if you're on T3 today, T4g is what AWS offers as the upgrade path. I tested both medium (2 vCPU, 4 GiB) and large (2 vCPU, 8 GiB) sizes. The T4g instances are 10-11% cheaper per hour than the equivalent T3 sizes. All T-series instances were launched with Unlimited mode enabled \[12\], so CPU credits were not a bottleneck during the benchmarks. I should note: AMD EPYC instances were not available in the M7 generation for Aurora PostgreSQL at the time, so the comparison is purely Intel vs Graviton. --- ## M-series: Intel wins, and not by a little The db.m7i.large (Intel) beat the db.m7g.large (Graviton3) in every workload, at every concurrency level. In the default mixed workload, the Intel instance averaged 2,624.93 TPS. The Graviton instance averaged 1,917.01 TPS. That's a 36.9% gap. The advantage ranged from 29.3% at 64 clients to 48.7% at 16 clients. The gap got even wider on the select-only workload. Intel averaged 34,493.43 TPS vs Graviton's 23,574.45 TPS. A 46.3% advantage. At 128 clients, Intel hit 68.2% more throughput. The Graviton instance actually started degrading at higher concurrency. It peaked at 25,351.1 TPS with 32 clients, then dropped to 20,215.3 TPS at 128 clients. The Intel instance just kept scaling. For the write-intensive workload, Intel maintained a 35.7% average advantage. Both instances scaled well here, but Intel was consistently ahead. But here's the catch: the Intel instance costs 6% more per hour. So does the higher throughput justify the higher cost? I calculated the cost-per-transaction (CPT) by dividing the price per second by the average TPS. Lower CPT is better. For the M-series default workload: - db.m7i.large (Intel): $18.84 × 10⁻⁹ per transaction - db.m7g.large (Graviton3): $24.35 × 10⁻⁹ per transaction Intel was 22.6% cheaper per transaction, despite the higher hourly rate. The throughput advantage more than compensated for the price difference. ![Line chart comparing M-series instance performance across three workloads (default, select-only, simple-update) at 16, 32, 64, and 128 clients. db.m7i.large (Intel, coral) consistently outperforms db.m7g.large (Graviton3, blue) across all workloads and concurrency levels. Select-only TPS scaled by 10x for visualization.](https://anantafatur.dev/content/images/2026/08/fig1.png) Figure 1: M-series performance across three workloads. Select-only TPS scaled by 10× for visualization. ## T-series: Graviton takes the lead The burstable tier told a different story. The Graviton2-based T4g instances consistently outperformed the older Intel T3 instances. For the medium instances, the db.t4g.medium averaged 990.12 TPS in the default workload vs db.t3.medium's 898.16 TPS. A 10.2% advantage. The gap grew with concurrency, peaking at 13.0% with 64 clients. ![Line chart comparing T-series medium instance performance across three workloads at 16, 32, 64, and 128 clients. db.t4g.medium (Graviton2, blue) consistently outperforms db.t3.medium (Intel, coral) across all workloads. Select-only TPS scaled by 10x for visualization.](https://anantafatur.dev/content/images/2026/08/fig2.png) Figure 2: T-series medium performance across three workloads. Select-only TPS scaled by 10× for visualization. The large instances showed an even bigger gap. db.t4g.large averaged 1,155.73 TPS vs db.t3.large's 922.13 TPS. That's a 25.3% advantage. At 32 clients, the gap hit 28.8%. The select-only workload on large instances showed the biggest difference. db.t4g.large hit 7,732.94 TPS mean vs db.t3.large's 6,254.90 TPS. A 23.6% advantage. ![Line chart comparing T-series large instance performance across three workloads at 16, 32, 64, and 128 clients. db.t4g.large (Graviton2, blue) shows a substantial throughput advantage over db.t3.large (Intel, coral) across all workloads and concurrency levels. Select-only TPS scaled by 10x for visualization.](https://anantafatur.dev/content/images/2026/08/fig3.png) Figure 3: T-series large performance across three workloads. Select-only TPS scaled by 10× for visualization. The cost-per-transaction numbers were just as clear: - db.t4g.medium: $18.24 × 10⁻⁹ (18.1% better than T3) - db.t3.medium: $22.27 × 10⁻⁹ - db.t4g.large: $31.01 × 10⁻⁹ (29.0% better than T3) - db.t3.large: $43.68 × 10⁻⁹ The T4g instances are both faster and cheaper. That's the best kind of win. ![Bar chart comparing cost-per-transaction in USD x 10 to the power of negative 9 across all six instances. db.m7i.large (Intel) has the lowest CPT among general-purpose at 18.84. db.t4g.medium (Graviton) has the lowest overall CPT at 18.24. ARM instances shown in blue, Intel in coral. Burstable instances shown with hatched pattern, general-purpose with solid fill.](https://anantafatur.dev/content/images/2026/08/fig4.png) Figure 4: Cost-per-transaction (CPT) comparison across all instances. Lower is better. ARM instances in blue, Intel in coral. Burstable instances shown with hatched pattern. ## Why the results flip between tiers This surprised me at first. Why would Graviton lose in M-series but win in T-series? I think there are two things going on. First, the microarchitecture. In the M-series, we're comparing Graviton3 against Intel Sapphire Rapids. Both are the latest from their respective vendors. And on this specific workload, Aurora PostgreSQL OLTP, Intel's microarchitecture simply performs better. A previous study by Loghin \[15\] found that ARM servers spend roughly twice as much time in Linux kernel system calls compared to Xeon servers, and show lower per-core performance due to less optimized software. That lines up with what I saw. Second, the T-series comparison is cross-generational. The T4g instances run Graviton2, which is a newer microarchitecture than the Intel Xeon processors in T3 instances (Skylake-SP or Cascade Lake, depending on what AWS assigns you). You're getting a generational leap in CPU design, a more modern memory subsystem, and better process node. All of that translates to higher throughput, and AWS prices it lower too. I can't separate how much of the T-series advantage comes from the ARM architecture vs just being a newer chip. The instance types available don't let me isolate that. But as a practical matter, it doesn't matter. If you're on T3 today, moving to T4g gives you more performance for less money. The reason is academic. ## What I'd recommend (with caveats) This is not a universal truth. It's what I observed in one region, on one database engine, with one benchmark tool. But here's what I'd say: If you're running a production Aurora PostgreSQL database on general-purpose instances, and performance matters, go with Intel M7i. The db.m7i.large gives you 29-68% more throughput for only 6% more cost. The cost-per-transaction is 22.6% better. That's a clear win. If you're on T3 instances today for dev, staging, or variable workloads, migrate to T4g. You'll get 10-25% more throughput and 18-29% better cost-per-transaction. There's no downside I could find. Always consider instance generation alongside architecture. Don't just ask "ARM or x86?" Ask "which specific instance, from which generation, for which workload?" ## What I didn't test Plenty. This study has real limits. I only used pgbench with the standard TPC-B-like workload. Something like TPC-C would be a better simulation of real-world OLTP. I didn't touch Aurora-specific parameter tuning. Maybe there are configs that favor Graviton more. I didn't test storage-optimized instance families like the I/O-optimized ones. I didn't monitor CPU telemetry like credit balances or thermal throttling on the burstable instances, so I can't confirm exactly what was happening under the hood. I only tested in us-east-1\. Different regions might have different underlying hardware and resource overcommitment practices, which can introduce variable CPU steal times \[16\]. I only tested Aurora PostgreSQL. Aurora MySQL might behave differently. I also didn't test AMD instances, since they weren't available in the M7 generation for Aurora at the time. If someone else picks this up, those are the gaps I'd fill first. I might do a follow-up with TPC-C if I get the time and budget for it. But for now, this is what I have. Take it as one data point, not the final word. --- ## References \[1\] J. Tharwani and A. A. Purkayastha, "Cost-Performance Evaluation of General Compute Instances: AWS, Azure, GCP, and OCI," 2024, arXiv. doi: 10.48550/ARXIV.2412.03037. \[2\] B. Varghese and R. Buyya, "Next Generation Cloud Computing: New Trends and Research Directions," 2017, arXiv. doi: 10.48550/ARXIV.1707.07452. \[3\] X. Chen, L.-H. Hung, R. Cordingly, and W. Lloyd, "X86 vs. ARM64: An Investigation of Factors Influencing Serverless Performance," in *Proceedings of the 9th International Workshop on Serverless Computing*, Bologna, Italy: ACM, Dec. 2023, pp. 7–12\. doi: 10.1145/3631295.3631394. \[4\] R. Mathá, D. Kimovski, A. Zabrovskiy, C. Timmerer, and R. Prodan, "Where to Encode: A Performance Analysis of x86 and Arm-based Amazon EC2 Instances," 2021, arXiv. doi: 10.48550/ARXIV.2106.06242. \[5\] M. D. Lima, L. Teylo, and L. Drummond, "An Analysis of Performance Variability in AWS Virtual Machines," in *Anais do XXV Simpósio em Sistemas Computacionais de Alto Desempenho (SSCAD 2024)*, Brasil: SBC, Oct. 2024, pp. 312–323\. doi: 10.5753/sscad.2024.244526. \[6\] A. Verbitski, A. Gupta, D. Saha, M. Brahmadesam, K. Gupta, R. Mittal, S. Krishnamurthy, S. Maurice, T. Kharatishvili, and X. Bao, "Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases," in *Proceedings of the 2017 ACM International Conference on Management of Data*, Chicago, IL, USA: ACM, May 2017, pp. 1041–1052\. doi: 10.1145/3035918.3056101. \[7\] R. Goel, "Amazon Aurora: Insights and Benchmarks for Contemporary Application Scaling," *IJCA*, vol. 186, no. 70, pp. 29–31, Mar. 2025, doi: 10.5120/ijca2025924554. \[8\] G. C. F. Ferrari, V. Filho, and M. Castro, "Comparing Burstable and On-Demand AWS EC2 Instances using NAS Parallel Benchmarks," in *Anais da XXIV Escola Regional de Alto Desempenho da Região Sul (ERAD-RS 2024)*, Brasil: SBC, Apr. 2024, pp. 61–64\. doi: 10.5753/eradrs.2024.238733. \[9\] Amazon Web Services, "DB instance class types — General-purpose instances," \[Online\]. Available: . Accessed: Feb. 9, 2026. \[10\] Amazon Web Services, "DB instance class types — Burstable performance instances," \[Online\]. Available: . Accessed: Feb. 9, 2026. \[11\] M. U. Kalay, "Concurrency Challenges in Database Systems: A Focus on Postgresql," *Bilgisayar Bilimleri ve Teknolojileri Dergisi*, vol. 6, no. 1, pp. 1–16, Jul. 2025, doi: 10.54047/bibted.1574178. \[12\] Amazon Web Services, "Unlimited mode for burstable performance instances," \[Online\]. Available: . Accessed: Mar. 17, 2026. \[13\] PostgreSQL Global Development Group, "PostgreSQL: Documentation: 16: pgbench," \[Online\]. Available: . Accessed: Apr. 2, 2026. \[14\] S. Henning, A. Vogel, E. Perez-Wohlfeil, O. Ertl, and R. Rabiser, "When Should I Run My Application Benchmark? Studying Cloud Performance Variability for the Case of Stream Processing Applications," in *Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering*, Trondheim, Norway: ACM, Jun. 2025, pp. 400–410\. doi: 10.1145/3696630.3728563. \[15\] D. Loghin, "Are ARM Cloud Servers Ready for Database Workloads? An Experimental Study," *IEEE Trans. Cloud Comput.*, vol. 12, no. 3, pp. 818–829, Jul. 2024, doi: 10.1109/TCC.2024.3393895. \[16\] A. Uhlig, I. Braun, and M. Wählisch, "The SAP Cloud Infrastructure Dataset: A Reality Check of Scheduling and Placement of VMs in Cloud Computing," in *Proceedings of the 2025 ACM Internet Measurement Conference*, Madison, WI, USA: ACM, Oct. 2025, pp. 746–760\. doi: 10.1145/3730567.3764480. ### Proxmox 8 to 9 Upgrade: Fixing Meta-Package and Repo Errors URL: https://anantafatur.dev/proxmox-8-to-9-upgrade-fixing-meta-package-and-repo-errors/ Last updated: 2026-08-14T19:15:23.000Z Upgrading a hypervisor always makes me slightly nervous. It feels like replacing core parts of a system while everything is still running. I recently upgraded a standalone Proxmox VE 8 node to Proxmox VE 9 (which moves the underlying Debian base from Bookworm to Trixie). I followed the official documentation, but I hit a problem almost immediately. Here is what went wrong, how I investigated the repository conflicts, and how I fixed it. --- ## The Scary Meta-Package Warning So here is what happened. I updated my apt sources to point to Debian Trixie and started the upgrade process. I ran `apt dist-upgrade` and was immediately greeted by a massive warning block. The `pve-apt-hook` script caught something bad. APT was trying to completely uninstall the `proxmox-ve` meta-package. ```bash W: (pve-apt-hook) !! WARNING !! W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'! W: (pve-apt-hook) W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command W: (pve-apt-hook) touch '/please-remove-proxmox-ve' ``` If I had bypassed this hook, APT would have removed the entire hypervisor from the server. ## Investigating the Conflict The [official Proxmox upgrade docs](https://pve.proxmox.com/wiki/Upgrade%5Ffrom%5F8%5Fto%5F9) explicitly mention this exact scenario. Usually, this happens if you installed Proxmox on top of a vanilla Debian setup. The standard Debian kernel package (`linux-image-amd64`) conflicts with the new Proxmox 9 kernel setup. I assumed this was my problem. I tried to remove the conflicting package. ```bash root@proxmox:~# apt remove linux-image-amd64 Reading package lists... Done Building dependency tree... Done Reading state information... Done Package 'linux-image-amd64' is not installed, so not removed ``` Well, that ruled out the easy fix. The kernel package was not even installed. I had to dig deeper to see what APT was actually looking at. I ran `apt policy` and dumped my repository lists to see where my packages were coming from. ```bash cat /etc/apt/sources.list cat /etc/apt/sources.list.d/* ``` ## Fixing the Split-Brain Repositories The output from the sources list revealed a classic split-brain configuration. My base Debian repositories were correctly pointing to `trixie`. But my Proxmox repository line for Trixie was commented out with a `#`. To make matters worse, an old script (pve-nag-buster) had left a separate file in `/etc/apt/sources.list.d/` that was actively pointing to the old Bookworm packages. Because APT saw the brand new Debian 13 packages but only saw the old Proxmox 8 packages, it panicked. It realized the old hypervisor packages were incompatible with the new base OS. To resolve the dependency nightmare, APT decided the most logical choice was to just delete Proxmox entirely. The fix was simple. I opened `/etc/apt/sources.list` and uncommented the correct Trixie line. ```text deb http://download.proxmox.com/debian/pve trixie pve-no-subscription ``` Then I deleted the leftover Bookworm file from the `sources.list.d` directory. Once the repositories were aligned, I ran `apt update` and `apt dist-upgrade`. The upgrade finished smoothly without trying to remove my hypervisor packages. ## Clearing the Pre-Flight Checklist After the core upgrade, I ran `pve8to9`. This is the built-in Proxmox checklist script that flags configuration issues. My output was not completely green. I had 3 failures and 5 warnings. I walked through them one by one. First, I had to remove the `systemd-boot` meta-package. Debian Trixie split this package into separate pieces. Leaving the old meta-package installed will break future bootloader upgrades. I removed it with a simple `apt remove systemd-boot`. Next, the script warned me about my GRUB EFI removable bootloader. I just ran the exact commands the checker provided to force the EFI extra removable flag to true, and then I reinstalled `grub-efi-amd64`. I also got hit with a missing time synchronization daemon warning. Accurate time is critical for Proxmox. I installed `chrony`, which is the standard NTP daemon for this environment. Finally, I had a notice about LVM autoactivation. Proxmox 9 changes how LVM volumes activate to prevent clustering issues. I ran the provided migration script (`/usr/share/pve-manager/migrations/pve-lvm-disable-autoactivation`) to update my local storage volumes. --- ## The Final Repositories and Microcode Honestly, I thought I was done, but I hit two final snags when trying to fix my missing Intel microcode package. When I ran `apt update`, it threw a `401 Unauthorized` error. APT was trying to pull from `enterprise.proxmox.com`. I do not have a paid enterprise subscription. I had clearly left a stale enterprise list file active. I deleted `/etc/apt/sources.list.d/pve-enterprise.list` to stop APT from checking it. Then I tried to install the microcode. ```bash apt install intel-microcode Error: Package 'intel-microcode' has no installation candidate ``` Debian Trixie only enables open-source repositories (`main` and `contrib`) by default. The Intel microcode is proprietary. I had to edit `/etc/apt/sources.list` and append `non-free` and `non-free-firmware` to the Debian repository lines. After one last `apt update`, the microcode installed perfectly. I ran `pve8to9` one final time. Zero failures. Zero warnings. I rebooted the server, and it came back up flawlessly on the new Proxmox VE 9 kernel. ### Tracing a 502: from Cloudflare to an unhandled Promise rejection URL: https://anantafatur.dev/tracing-a-502-from-cloudflare-to-an-unhandled-promise-rejection/ Last updated: 2026-08-14T17:08:31.000Z A developer on our team reported that the QA environment was returning a 502 Bad Gateway from Cloudflare. The endpoint was `/v1/redirect`, a redirect endpoint that calls an external OAuth API to generate a token. Nothing changed on our side. No recent deployments, no config updates. Yet Cloudflare was showing 502. I opened the pod logs to see what was happening. And I found something confusing. ## The log says 401, but Cloudflare says 502 I ran the usual command: ``` kubectl logs -f -n --tail=200 ``` And there it was. The service was logging a 401: ``` [2026-08-14T10:56:54.377Z] INFO: API Call url: 'https:///oauth/token' method: 'POST' data: '{"clientId":"","clientSecret":""}' You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection: [object Object] [2026-08-14T10:56:54.465Z] ERROR: Generate access token error: undefined [2026-08-14T10:56:54.466Z] FATAL: message: {"status":401,"statusText":"Error generating access token"} ``` I told the developer: we got a 401 from the external API. The credentials might be wrong, or the partner API is having issues. And then the developer asked the question that made me stop and think: > "If the error is 401, why does Cloudflare show 502? Why isn't it forwarding the 401 to the client?" That was a good question. A 401 is a valid HTTP response. If our service received a 401 from the external API and passed it back to the client, Cloudflare should have shown 401, not 502\. The fact that Cloudflare returned 502 meant something else was happening between the 401 and the response. So I tried to reproduce it locally to understand the behavior. --- ## The environment The service is a Node.js Express app running on Kubernetes. It uses `pm2-runtime` as the process manager inside the container. The Dockerfile has this as the entrypoint: ``` CMD ["./node_modules/.bin/pm2-runtime", "starter.js", "--instances", "1"] ``` The service calls an external partner API to get an OAuth access token. That token is then used to generate a redirect URL for the user. The external API lives at `https:///oauth/token`. The call is made through an internal npm package, which wraps the HTTP request with circuit breaker logic. The credentials are stored in a config file. ## Reproducing the crash I built a minimal reproduction. Two services: a Flask API that returns 401 when no valid token is provided, and a Node.js frontend that calls it. The frontend has two endpoints: one that handles the 401 properly, and one that does not. When the frontend handles the 401 properly, Cloudflare forwards the 401 to the browser. The status code is preserved. Everything works as expected. But when the Promise rejects without being caught, the Node.js process crashes. And when the process crashes, the container dies. And when the container dies, the backend is unreachable. Cloudflare, seeing an unreachable backend, returns 502. That is the answer to the developer's question. The 401 never makes it back to Cloudflare. The service crashes before it can send any response. Cloudflare only sees that the backend is gone, so it returns 502. ## The chain of events I tested the external API directly from inside the pod to confirm: ``` curl -X POST "https:///oauth/token" \ -H "Content-Type: application/json" \ -d '{"clientId":"","clientSecret":""}' ``` The response: `no healthy upstream`. So the partner's backend was down. The chain looks like this: 1. External API backend is down, returns `no healthy upstream`. 2. The internal npm package makes the request, gets an error response, but the error object is `undefined`. 3. The undefined error causes an unhandled Promise rejection. 4. Node.js triggers the `unhandledRejection` event. The service logs it as FATAL. 5. The process exits. pm2-runtime detects the child process died and stops entirely. 6. The container's main process (pm2-runtime) is gone. The container exits. 7. Kubernetes restarts the container. During the restart, the backend is unreachable. 8. Cloudflare sees the backend as unreachable and returns 502. The 401 appears in the logs because the service logged it before crashing. But the HTTP response with that 401 status code was never sent to the client. The process was already dead by then. ## What I got wrong about pm2-runtime This is where I wasted some time during the investigation. I assumed pm2-runtime would restart the child process, just like regular pm2 does. I kept checking `pm2 list` expecting to see the restart count go up. But pm2-runtime is different. From the logs: ``` 2026-08-14T17:57:23: PM2 log: Stopping app:starter id:0 2026-08-14T17:57:24: PM2 log: App [starter:0] exited with code [1] via signal [SIGINT] 2026-08-14T17:57:24: PM2 log: PM2 successfully stopped ``` pm2-runtime does not restart the child. It exits the entire container. This is by design. pm2-runtime is meant to run as PID 1 in a container and let the orchestrator handle restarts. When the child dies, pm2-runtime dies too. The container stops. Kubernetes restarts it. This means the restart takes 10 to 30 seconds. Container startup, database connection, table sync, Redis connection. All of that has to happen again. That 10 to 30 second window is when the backend is unreachable and Cloudflare returns 502. And since this is the UAT environment, we only have one pod. No replicas. There is no other pod to pick up the traffic while the container restarts. Every request during that window hits a dead backend. If the service used plain `pm2` instead of `pm2-runtime`, the child process would restart in 1 to 2 seconds and the container would stay alive. But that also means Kubernetes would not know the process crashed, which is a tradeoff. ## pm2 vs pm2-runtime, from a DevOps perspective I come from an ops background, not development, so I had to look this up. Both `pm2` and `pm2-runtime` come from the same npm package. When you run `npm install`, both binaries end up in `node_modules/.bin/`. The difference is in how they behave, not what they are. `pm2` is a process manager daemon. You run `pm2 start app.js`, and it forks itself into the background, then starts your app as a child. If your app crashes, pm2 restarts it. pm2 itself stays alive. This is designed for running on bare metal or VMs, where you want the process manager to keep things running no matter what. `pm2-runtime` is the container-friendly version. It does not daemonize. It runs as the main process (PID 1) and starts your app as a child. But here is the key difference: when your app crashes, pm2-runtime also exits. It does not restart the child. It lets the whole container die so the orchestrator (Kubernetes) can handle the restart. Why would anyone use pm2-runtime then? Because in Kubernetes, you want the container to die when the app dies. That way the liveness probe fails, the restart counter goes up, and your monitoring picks it up. With plain pm2, the container stays alive, Kubernetes thinks everything is fine, and you lose visibility into the crash. But the tradeoff is the restart time. 1 to 2 seconds versus 10 to 30 seconds is a big difference when you only have one pod and no replicas. ## Why the pod name changed (and why it did not matter) During the investigation, I noticed the pod name changed from `-r8tc2` to `-h4vbl`. I initially thought the crash caused the pod to be recreated. It did not. The pod name change was my own fault. I accidentally ran `pm2-runtime list` inside the pod, thinking it would show the process list. But `pm2-runtime` is not a CLI tool. It interprets `list` as a script path, fails to find it, and exits. That killed the container, and Kubernetes recreated the pod with a new name. This was a self-inflicted restart, completely unrelated to the crash we were investigating. The actual crash from the unhandled rejection only restarts the container inside the same pod. The pod name stays the same, and the RESTARTS counter in `kubectl get pods` increments. To confirm this, I checked: ``` kubectl get pods -n | grep ``` The pod had RESTARTS: 1, not a new pod name from the crash. ## What actually happened, and what we did about it At the end of the day, the root cause was simple: the external API backend was down, returning `no healthy upstream`. Our service called it, the request failed, and the unhandled rejection crashed the container. The fix was straightforward: ask the 3rd party to bring their backend back up. Once they fixed their load balancer and the upstream became healthy again, the requests started succeeding. No more 401, no more crash, no more 502. That said, the incident exposed two weaknesses in our code that are worth noting as tech debt: 1. The internal npm package does not handle errors properly. When the HTTP request fails, the error object is `undefined`, which causes an unhandled rejection. A well-written package should catch that and reject with a proper error object. 2. The service has no global unhandled rejection handler. A single `process.on('unhandledRejection', ...)` would have logged the error without crashing the process. The service would have stayed alive, returned a proper error response to the client, and Cloudflare would have shown 401 instead of 502. 3. Using `pm2-runtime` instead of `pm2` means every crash costs 10 to 30 seconds of container restart instead of 1 to 2 seconds of process restart. With only one pod in UAT, that window means complete downtime. These are not my scope to fix, but they are worth a ticket. --- ## Closing The question that started this investigation was a good one: "if the error is 401, why does Cloudflare show 502?" The answer is that the 401 never reached Cloudflare. The service crashed before it could send any response. The 401 exists only in the logs, as evidence of what happened. Cloudflare only saw a dead backend. The external API being down was out of our control. But crashing the entire service because of it was completely avoidable. Your service should never crash because an external dependency is unavailable. Handle it, log it, return an error to the client, and move on. ### Fixing the Docker Buildx EOF Error by Pinning Buildkit to v0.17.2 URL: https://anantafatur.dev/fixing-the-docker-buildx-eof-error-by-pinning-buildkit-to-v0-17-2/ Last updated: 2026-08-11T08:48:03.000Z ## The error The build would run to completion, push layers to the registry, and then crash right at the end. ``` ERROR: failed to receive status: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: EOF", received prior goaway: code: NO_ERROR, debug data: "graceful_stop" ``` The build itself was fine. The image was pushed. But the CI job failed, which meant the pipeline stopped. Rerunning the job fixed it about half the time. The other half, it failed again. This was not a flaky network. Something was wrong with Buildx itself. --- ## The environment Our pipeline uses a shared library function that gets loaded at runtime from a centralized script file. Here is roughly what it does: 1. Creates a Docker context pointing to a DinD service (`docker:27.3.1-dind`, TLS disabled) 2. Creates a Buildx builder with the `docker-container` driver 3. Runs `docker buildx build --push` to build and push the image The builder was not pinned to any specific Buildkit version. It was pulling whatever `moby/buildkit:buildx-stable-1` resolved to at the time. That turned out to be the problem. Here is the relevant part of the function before the fix: ```bash docker buildx create --name ${builder_name} --driver docker-container --use ${context_name} ``` No `--driver-opt` flag. No image pinning. Whatever Buildkit version Docker decided to pull, that is what we got. ## The investigation I spent the first hour blaming the network. Our DinD service talks to the runner over TCP, and EOF errors usually mean a connection dropped. I checked the DinD logs. Nothing. The connection was healthy. The build was completing. The push was completing. The error happened after the push, when the client was waiting for the final status from the Buildkit daemon. Then I searched for the exact error message and found [docker/buildx#2789](https://github.com/docker/buildx/issues/2789). The issue had multiple reports from users on self-hosted GitHub runners, all describing the same pattern: builds complete, push succeeds, then EOF at the very end. The Buildx maintainer, `crazy-max`, confirmed it was a gRPC transport bug in Buildkit. The Buildkit daemon was sending a graceful shutdown signal (`graceful_stop`) to the gRPC connection while the client was still waiting for status updates. The client saw the connection close, got an EOF, and reported it as an error. The fix was merged in [moby/buildkit#5530](https://github.com/moby/buildkit/pull/5530). The version that had the bug was whatever `buildx-stable-1` resolved to at the time. The fix landed in Buildkit v0.17.2. ## The fix One line changed. I added `--driver-opt image=moby/buildkit:v0.17.2` to the `docker buildx create` command. ```bash docker buildx create --name ${builder_name} --driver docker-container \ --driver-opt image=moby/buildkit:v0.17.2 \ --use ${context_name} ``` That is it. No more EOF errors. The build runs to completion, the status is received cleanly, and the CI job passes. The full updated function, simplified, now looks like this: ```bash # ... setup docker context, trap cleanup, TLS config ... docker buildx create --name ${builder_name} --driver docker-container \ --driver-opt image=moby/buildkit:v0.17.2 \ --use ${context_name} docker buildx inspect --bootstrap # ... resolve registry and tag based on environment ... docker buildx build -f $DOCKERFILE --progress plain --platform ${PLATFORMS} $ARGS \ -t //: \ --provenance=false . --push ``` --- ## Closing The mistake was not pinning the Buildkit image. Relying on a floating tag like `buildx-stable-1` means your CI can break silently when a new version ships with a bug. The fix is one line, but the debugging took longer than it should have because the error message does not point to Buildkit at all. It looks like a network problem. If you run Buildx with a `docker-container` driver, pin the Buildkit image. Always. It is a Docker container running inside your CI. It gets updated. It can break. The fix is `--driver-opt image=moby/buildkit:v0.17.2`, or whatever version you have tested and trust. The GitHub issue thread also mentions that the bug was fixed in Buildkit v0.17.2, so if you are running a newer version, you should not hit this. But if you are not pinning, you never know what version you are actually running until something breaks. I might revisit this later if we ever need multi-architecture builds. For now, we only build `linux/amd64`, so we could even drop Buildx entirely and use plain `docker build` and `docker push`. But keeping the Buildx setup means we are ready for multi-arch when the time comes. Just with a pinned Buildkit image this time. ### Fixing Athens Go Proxy for Private GitLab Modules URL: https://anantafatur.dev/fixing-athens-go-proxy-for-private-gitlab-modules/ Last updated: 2026-08-09T07:30:01.000Z Last week I set up Athens Go Proxy in our Kubernetes cluster. Public modules cached fine, but private modules from our internal GitLab always failed. I wrote about the compromise I settled on: use GOPRIVATE to bypass Athens for private modules, and let each Docker build handle GitLab authentication directly with a build secret. I called it tech debt. I said the fix was probably simple, I just had not figured out why Athens's Go worker processes were not picking up the git configuration. I was wrong about it being simple. But I was right that it was fixable. Here is the full investigation and the actual fix. --- ## The symptom When Athens tried to download a private module from our GitLab, it returned a 404 with this error embedded in the response: ``` not found: //@v0.0.0-20260803051323-9fc1b59e4974: invalid version: git ls-remote -q https:////.git exit status 128: fatal: could not read Username for 'https://': terminal prompts disabled ``` But if I exec'd into the pod and ran go mod download manually with GOPROXY=direct and the same gitconfig, it worked. So the git configuration was there, the token was valid, the network was fine. The problem was something specific to how Athens spawns its child processes. --- ## The investigation I started by having DeepSeek-v4-pro read the Athens source code for me. The version we are running is v0.18.1\. When Athens needs to download a module, it creates a GoGetFetcher and calls downloadModule, which runs go mod download as a child process. The AI pointed me to the critical function: prepareEnv in pkg/module/prepare\_env.go. Here is what prepareEnv does. It creates a completely clean environment for the child Go process. It sets GOPATH, GOCACHE, CGO\_ENABLED=0, and GO111MODULE=on. Then it copies only a hardcoded whitelist of environment variables from the parent process: ``` HOME, PATH, GIT_SSH, GIT_SSH_COMMAND, HTTP_PROXY, HTTPS_PROXY, NO_PROXY, GOROOT ``` That is it. Everything else is stripped. GONOSUMDB, GOPRIVATE, GONOPROXY, GIT\_CONFIG\_GLOBAL, NETRC. All gone. This is why the gitconfig at /etc/gitconfig was not working. Go's go mod download sets GIT\_CONFIG\_NOSYSTEM=1, which blocks the system gitconfig. And the chart's gitconfig feature mounts the config at /etc/gitconfig, which is exactly the system gitconfig that gets blocked. The gitconfig at $HOME/.gitconfig (/root/.gitconfig) should work though, because HOME is in the whitelist and GIT\_CONFIG\_NOSYSTEM=1 only blocks /etc/gitconfig, not $HOME/.gitconfig. But the checksum verification was also failing because GONOSUMDB was stripped. The error was: ``` verifying go.mod: reading sum.golang.org/... ``` So even if git auth worked, the checksum verification against the public sum.golang.org would fail for private modules. To confirm this, I ran an isolated test inside the Athens pod. I recreated the exact clean environment that prepareEnv creates, using env -i: ```sh kubectl exec -it deploy/athens-go -n athens -- sh -c ' env -i \ HOME=/root \ GOPATH=/tmp/gopath \ GOCACHE=/tmp/gopath/cache \ CGO_ENABLED=0 \ GO111MODULE=on \ PATH="$PATH" \ git ls-remote https:////.git ' ``` It worked. The git ls-remote returned the commit hashes. So the insteadOf gitconfig at /root/.gitconfig was being read by git in the clean environment. The git auth was not the problem. The real problem was only GONOSUMDB being stripped. --- ## The fix: ATHENS\_GO\_BINARY\_ENV\_VARS Athens has a configuration option called GoBinaryEnvVars. It is a list of KEY=VALUE pairs that get appended to the child process environment. They are added after the whitelisted variables, so they can even override them. The environment variable name is ATHENS\_GO\_BINARY\_ENV\_VARS. The format uses semicolons to separate multiple pairs: ``` ATHENS_GO_BINARY_ENV_VARS="GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct" ``` The default value is GOPROXY=direct. When you set ATHENS\_GO\_BINARY\_ENV\_VARS, it completely replaces the default, so you have to include GOPROXY=direct if you want Athens to download from source. I found this out from the EnvList.Decode function in the Athens config package, which the AI pointed me to. The comment says "env vars must override config file". I added this to the Helm chart's configEnvVars: ```yaml - name: ATHENS_GO_BINARY_ENV_VARS value: "GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct" ``` This injects GONOSUMDB and GOPRIVATE into every child Go process that Athens spawns. The wildcard pattern means "skip checksum verification for all modules" and "treat all modules as private". This is fine for our use case since Athens is an internal proxy and we do not need the public checksum database for modules cached through it. The full Athens Helm values now look like this: ```yaml configEnvVars: - name: ATHENS_GO_BINARY_ENV_VARS value: "GONOSUMDB=*; GOPRIVATE=*; GOPROXY=direct" lifecycle: postStart: exec: command: - sh - -c - | cp /etc/netrc/.netrc /root/.netrc TOKEN=$(awk '/password/ {print $2}' /etc/netrc/.netrc) cat > /root/.gitconfig << EOF [url "https://oauth2:${TOKEN}@/"] insteadOf = https:/// EOF netrc: enabled: true existingSecret: netrcsecret upstreamProxy: enabled: false ``` The postStart hook copies the .netrc file to /root/.netrc and creates a /root/.gitconfig with the insteadOf URL rewriting. This is needed because the chart's netrc feature mounts .netrc at /etc/netrc/.netrc, but git looks for it at $HOME/.netrc. The insteadOf gitconfig embeds the GitLab token directly into the URL, so git does not need to prompt for credentials. The upstreamProxy is disabled because we want Athens to download from source. When enabled, Athens redirects to the upstream proxy (proxy.golang.org) for modules it does not have cached. But the filter file was empty, so private modules were being redirected to the public upstream, which returned 404, and the client fell back to direct, which failed because git auth was not configured. --- ## The Dockerfile side On the client side, the Dockerfile needed a change too. In the first article, I used GOPRIVATE to tell Go to bypass the proxy for private modules. But now that Athens can handle private modules, I want the opposite. I want Go to use the proxy for everything, including private modules. The problem is that GOPRIVATE implicitly sets both GONOPROXY and GONOSUMDB. GONOPROXY tells Go to bypass the proxy. So even if I point GOPROXY to Athens, the private modules are fetched directly from GitLab. That is exactly what I wanted in the first article, but now I want the opposite. The fix is to replace GOPRIVATE with explicit GONOSUMDB and GONOPROXY. GONOPROXY set to empty means "use the proxy for everything". GONOSUMDB set to the GitLab host means "skip checksum verification for these modules". The Go documentation says GOPRIVATE is deprecated since Go 1.21 anyway. Here is the final Dockerfile ENV block: ```dockerfile ENV GO111MODULE=on ENV GONOSUMDB= ENV GONOPROXY= ENV GOPROXY=http://,https://proxy.golang.org,direct ``` And the RUN commands no longer need the gitlab\_token secret or the git config insteadOf: ```dockerfile COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod/ \ go mod download -x && go mod verify COPY . . RUN --mount=type=cache,target=/go/pkg/mod/ \ --mount=type=cache,target=/root/.cache/go-build/ \ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ go build -ldflags="-w -s" -o ./cmd/api ``` The go mod download -x output confirms that both public and private modules are now going through Athens. Here is a snippet from the build log: ``` # get http:///github.com/sirupsen/logrus/@v/v1.9.4.info: 200 OK (0.001s) # get http://////@v/v0.0.0-20260803051323-9fc1b59e4974.info: 200 OK (0.001s) # get http:///go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho/@v/v0.69.0.info: 200 OK (0.001s) ``` The private module is being fetched from Athens, not from GitLab directly. And it is fast because Athens has it cached. --- ## What I got wrong in the first article I said the fix was probably adding HOME=/root to GoBinaryEnvVars. That was wrong. HOME was already in the prepareEnv whitelist. The insteadOf gitconfig at /root/.gitconfig was working fine. The real problem was GONOSUMDB and GOPRIVATE being stripped, which caused the checksum verification to fail and the Go module download to error out. I also said I did not fully understand how Athens spawns its Go processes. I was right about that. The prepareEnv function is the key piece I was missing. Once the AI walked me through the source code, everything clicked. The .netrc file being at /etc/netrc/.netrc instead of /root/.netrc was another detail I missed. The chart's netrc feature mounts it at /etc/netrc/.netrc and sets ATHENS\_NETRC\_PATH, but git looks for .netrc at $HOME/.netrc. The postStart hook that copies it to /root/.netrc fixes this. If you are setting up Athens for your team, here is my advice: forget the chart's gitconfig feature. It does not work for private modules because of GIT\_CONFIG\_NOSYSTEM=1\. Instead, use a postStart hook to create /root/.gitconfig with the insteadOf URL rewriting. And use ATHENS\_GO\_BINARY\_ENV\_VARS to inject GONOSUMDB and GOPRIVATE into the child processes. Two lines of configuration, and everything works. ### Caching Go Modules with Athens Proxy on Kubernetes URL: https://anantafatur.dev/caching-go-modules-with-athens-proxy-on-kubernetes/ Last updated: 2026-08-08T16:32:02.000Z Every time our CI pipeline runs a Go build, it downloads Go modules from the internet. `go mod download` pulls from `proxy.golang.org`, GitHub, and our internal GitLab. Over dozens of pipeline runs per day across multiple services, this adds up. It slows down builds, it wastes bandwidth, and it is an external dependency that does not need to be one. The obvious fix: run a Go module proxy inside the cluster. I picked Athens because it is the most mature option, supports disk storage, and can proxy both public and private modules. Here is how it went. --- ## The setup Athens runs as a single pod in its own namespace on our Kubernetes cluster. I deployed it using the official Helm chart (`gomods/athens-charts`, chart version `0.17.1`, Athens version `v0.18.1`). The chart is managed through ArgoCD, stored in our GitOps repo. It uses disk storage backed by a 20Gi PVC with `resourcePolicy: keep`, so the cache survives pod restarts and redeployments. The upstream proxy is `proxy.golang.org`. Any module Athens does not have cached, it fetches from there. The values file is simple. Athens listens on port 80, exposed through an nginx ingress at ``. A single replica, 250m CPU request, 1Gi memory limit, and 5 Go module download workers. The gitconfig is managed as a separate Kubernetes secret and mounted through the chart's `gitconfig` section. The secret contains the `insteadOf` rule that injects the GitLab token into URLs pointing to our internal GitLab. The first thing I hit was TLS. Our ingress uses a wildcard certificate from our cloud provider for ``. But the certificate subject is `ingress.local`, not the actual hostname. So when Go tried to fetch modules over HTTPS, it got: ``` tls: failed to verify certificate: x509: certificate is valid for ingress.local, not ``` The fix was simple: disable TLS on the ingress and use plain HTTP. Since Athens is only accessed from inside the cluster (by CI runners and Docker builds), this is fine. The `GOPROXY` in our Dockerfile uses `http://`. --- ## The private module problem Here is where things got interesting. Our Go services import a private module from our internal GitLab at ``. The idea was to have Athens cache everything, including private modules. That way, every service in the company could benefit from the cache. I set up a gitconfig on the Athens pod with an `insteadOf` rule that injects the GitLab token into the URL: ``` [url "https://oauth2:@/"] insteadOf = https:/// ``` I verified this works from inside the pod. Running `git ls-remote` and even `go mod download` with `GOPROXY=direct` worked perfectly. The token was being picked up, git could authenticate, and the module downloaded fine. But when Athens itself tried to download the private module, it failed with a 404\. The error message embedded in the response was: ``` not found: //@v0.0.0-20260803051323-9fc1b59e4974: invalid version: git ls-remote -q --end-of-options https:////.git exit status 128: fatal: could not read Username for 'https://': terminal prompts disabled ``` Athens was running the same `go mod download` internally, but the git process it spawned was not picking up the system gitconfig at `/etc/gitconfig`. I spent a few hours debugging this. I checked the service routing, the ingress, the pod environment, the gitconfig location. Everything was correct. The gitconfig worked for manual commands but not for Athens's internal Go worker processes. The Athens logs showed the real story. Or rather, the lack of it. During the entire time I was testing, the Athens pod logged zero module requests. Only health checks. The 404 responses were coming from somewhere inside Athens's internal download pipeline, not from the HTTP handler I could see in the logs. At this point I had a choice. I could dig deeper into Athens's Go module downloader internals. Maybe the `GoBinaryEnvVars`, maybe the process isolation, maybe the `HOME` directory the workers run under. Or I could admit that I do not fully understand how Athens spawns its Go processes and take the pragmatic route. I took the pragmatic route. ## The compromise: GOPRIVATE Instead of forcing Athens to handle private modules, I told Go to bypass the proxy for our internal packages. The `GOPRIVATE` environment variable does exactly this: ``` GOPRIVATE= ``` Modules matching `` are fetched directly from the source, using the git token configured in the Docker build. Everything else goes through Athens. The final `GOPROXY` configuration in our Dockerfile: ``` ENV GOPRIVATE= ENV GOPROXY=http://,https://proxy.golang.org,direct ``` The fallback to `proxy.golang.org` and `direct` means if Athens is ever down, the build still works. It just falls back to the public internet, which is slower but not broken. The git authentication for private modules uses a Docker build secret. The pipeline passes the GitLab token as a secret, and the Dockerfile mounts it during the `RUN` step: ``` RUN --mount=type=secret,id=gitlab_token \ --mount=type=cache,target=/go/pkg/mod/ \ sh -c 'TOKEN=$(sed "s/^Private-Token: *//" /run/secrets/gitlab_token) && \ git config --global url."https://oauth2:${TOKEN}@/".insteadOf "https:///" && \ go mod download -x && go mod verify' ``` I added the `-x` flag to `go mod download` to see what is actually happening. The output confirms that public modules are hitting Athens: ``` # get http:///go.opentelemetry.io/otel/sdk/@v/v1.44.0.zip: 200 OK (0.006s) # get http:///go.opentelemetry.io/otel/trace/@v/v1.44.0.zip: 200 OK (0.005s) # get http:///gopkg.in/go-playground/validator.v9/@v/v9.31.0.zip: 200 OK (0.004s) ``` The private module is fetched directly from our GitLab, authenticated by the gitconfig `insteadOf` rule. The `GOPRIVATE` flag tells Go to skip the proxy, the sum database, and the checksum database for those packages. Which is what you want for internal code that is not published. --- ## Closing I do not have benchmark data. No before-and-after pipeline timings, no egress bandwidth measurements. Our pipeline speeds were inconsistent before this change (standard deviation was high), so any comparison would be noisy. I could call this CV-driven development, and honestly, that is not entirely wrong. But the reasoning is sound: fewer external downloads means less dependence on internet speed, less bandwidth cost, and a faster feedback loop for developers. The real value here is not just for my team. This Athens instance is a shared resource. Any Go service in our Kubernetes cluster can point its `GOPROXY` to `` and get the same caching. The Dockerfile pattern (`GOPRIVATE` for internal packages, `GOPROXY` pointing to Athens with fallbacks) is something other teams can copy directly. The private module situation is tech debt. Athens should be able to cache private modules too. The fix is probably simple: figure out why Athens's Go workers do not read the system gitconfig, and either fix the environment or switch to a different auth mechanism. I suspect adding `HOME=/root` to the `GoBinaryEnvVars` in the Athens config would do it, but I have not tested that yet. If you are setting up Athens for your team, my advice is: start with `GOPRIVATE` for your internal packages. Get the caching working for public modules first. Then tackle private module caching as a separate effort. The public module cache alone is worth it, and the private module problem can be a follow-up. ### GitLab Read-Only Mode on a Free License URL: https://anantafatur.dev/gitlab-read-only-mode-on-a-free-license/ Last updated: 2026-08-05T04:09:32.000Z We recently attempted to migrate our self-hosted GitLab instance from AWS to Tencent Cloud. The migration failed, and we had to roll back. But one thing that made the failure worse was that people kept pushing code to the old GitLab during the maintenance window. The VM snapshot was already taken at 21:00, but users were still committing code until 04:00\. When we tried to bring up the new instance, those commits were just... gone from the disk. So for the next attempt, we needed a way to block writes during the cutover. No pushes, no branch creation, no tag changes. Just keep everything readable but frozen. The problem is our GitLab runs on a free license. GitLab has a built-in [Maintenance Mode](https://docs.gitlab.com/administration/maintenance%5Fmode/) feature, but it requires a Premium subscription. We don't have that. But here is the thing I found while digging through the docs. GitLab has a [read-only state](https://docs.gitlab.com/administration/read%5Fonly%5Fgitlab/) feature that is available on all tiers. It is not the same as Maintenance Mode, but it gets you most of the way there. You can set individual repositories to read-only, and it blocks all write operations while keeping everything readable. --- ## How the migration failed (and why this matters) A quick recap of what happened. We used `go2tencentcloud` to migrate the VM from AWS to Tencent, and DTS to sync the PostgreSQL database. The VM sync was stopped at 21:00, but the database sync kept running for another 6 hours. The VM snapshot was frozen at 21:00, but the database had records for commits that happened after that. When the new instance came up, we saw errors like this: ``` ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "authentication_events_pkey" ``` And then we found the real problem: commits pushed after 21:00 were missing from the new instance. The repositories on disk did not have the latest data, but the database expected them to exist. We ran a script on the old AWS instance to confirm: ```bash find /var/opt/gitlab/git-data/repositories \ -path "*/+gitaly/state/*" -prune -o \ -name "*.git" -print | while read repo; do git --git-dir="$repo" rev-parse HEAD >/dev/null 2>&1 || continue repo_name=${repo#/var/opt/gitlab/git-data/repositories/} repo_name=${repo_name%.git} git --git-dir="$repo" log \ --since="2026-07-31 21:00:00" \ --until="2026-08-01 05:00:00" \ --pretty=format:"$repo_name | %ad | %an | %h | %s" \ --date=iso done ``` The output was a long list of commits from multiple users across different projects, all pushed during the window when we thought everything was frozen. Some were automated deployments from CI pipelines. Others were manual commits from developers working late. All of them were missing from the new instance. So yeah. We needed a way to actually prevent writes next time. --- ## Finding the read-only feature I went through the GitLab docs and found the [read-only state](https://docs.gitlab.com/administration/read%5Fonly%5Fgitlab/) page. It describes how to set repositories as read-only using the Rails console. The feature is available on all license tiers, including free. The key is the `repository_read_only` attribute on the `Project` model. Set it to `true`, and the repository blocks all write operations. Set it back to `false`, and everything goes back to normal. The docs actually show how to set all projects to read-only with `Project.all.find_each`. But in my case, I could not just lock everything. We have two configuration repositories that are consumed by production services. I just did not want to take any risk by touching them. So I needed to lock all 6,150 projects except those two. ## Getting all project IDs First, I needed a list of every project ID. The GitLab API has a `/projects` endpoint that returns paginated results. I wrote a small Python script to fetch all of them and export to CSV. The script is pretty simple. It paginates through the API, collects all project info, and writes two output files: a full CSV with all project metadata, and a convenience file with just the IDs in a comma-separated list. ```python # Core loop of the script while True: url = f"{base_url}/projects?page={page}&per_page=100" params = {"membership": False, "simple": True} response = api_get(url, params=params) data = response.json() if not data: break projects.extend(data) page += 1 ``` I ran into a couple of issues with this. First, the API timed out on page 4 with a 30-second timeout. I bumped it to 120 seconds and added retry logic with exponential backoff. Second, my admin user's personal access token was not returning all projects. The token needed "Allow admin mode" enabled in GitLab's token settings. Without it, the API only returned projects where the user was a direct member. That was a fun 30 minutes of confusion. Once that was sorted, the script ran and gave me a CSV with all 6,150 projects. I also had a `_ids_only.txt` file with a comma-separated list of IDs, ready to paste into the Rails console. ## Excluding the two configuration repos I already mentioned this, but the two repos we needed to exclude are configuration repositories consumed by production services. Their project IDs are `7052` and `5607`. I just did not want to take any risk by touching them. I filtered them out of the ID list. The final list went into the Rails console command. ## Enabling read-only mode Here is the actual command I ran. Open the Rails console: ```bash sudo gitlab-rails console ``` Then paste the project IDs: ```ruby # List of project IDs to set to read-only. projects = [1, 2, 3, ...] # 6,148 IDs here (6,150 total minus 2 excluded) projects.each do |p| project = Project.find(p) project.update!(repository_read_only: true) rescue ActiveRecord::RecordNotFound puts "Project ID #{p} not found" end ``` The `rescue` block is there because some project IDs might have been deleted between the time the script ran and the time you run this command. It is a safety net. We still had the CVM from the failed migration sitting around unused, so I used it for this POC. It took about 5 minutes to process all 6,148 projects. Not bad at all. ## Restoring write access When the maintenance window is over, you just flip the flag back: ```ruby Project.all.find_each { |project| project.update!(repository_read_only: false) } ``` This also took about 5 minutes. The `find_each` method processes records in batches of 1,000, which is a lot more efficient than loading all projects into memory at once. ## What actually happens when a repo is read-only This is the part where I learned the difference between "read-only" and "archived." These are not the same thing. When you set `repository_read_only: true`, the repository is not archived. There is no visual indicator in the GitLab UI. No banner, no badge, no notification. The project looks completely normal. Users can browse the code, view branches, check commit history, look at pipeline results. Everything looks fine. They only discover the restriction when they try to write something. Here is what is blocked: - **Editing files via the web UI.** You get an error: `13:update reference: running pre-receive hooks: GitLab: The repository is temporarily read-only. Please try again later.` - **Pushing from the terminal.** Same error: `The repository is temporarily read-only. Please try again later.` - **Creating a branch.** `Failed to create branch: 13:running pre-receive hooks: GitLab: The repository is temporarily read-only. Please try again later.` - **Approving a merge request.** `Something went wrong during merge pre-receive hook. The repository is temporarily read-only. Please try again later.` But you can still close the merge request, which is interesting. - **Creating or removing tags.** Both fail with variations of the read-only error. - **Creating an issue.** I got an HTTP 500 with `Response not successful: Received status code 500`. Honestly, I am not sure if this is caused by the read-only mode or by the data corruption from the failed migration. The CVM I used for testing was the same one from the rolled-back migration, so it had some database issues. I did not have a clean instance to isolate this. - **Editing wiki pages.** Also blocked. - **Updating labels.** Blocked. - **Transferring project path.** Gives `Project could not be updated!`. Changing the project name and description still works though. And here is what still works: - Browsing the repo in the UI (code, branches, commits, pipeline history) - Pulling, fetching, and cloning via HTTPS or SSH - Closing merge requests - Sending test webhooks and resending from webhook history - Changing project name and description The error message for Git operations is consistent: "The repository is temporarily read-only. Please try again later." It is clear enough that users will understand what is happening, even if they were not notified beforehand. But the non-Git errors are inconsistent. Some give HTTP 500, some give generic error messages. Users might not know what is going on. --- ## The limitations The read-only flag blocks more than I expected. It is not just Git push operations. It also blocks issues, wikis, labels, and some project settings. That is actually a good thing for our use case. We want the instance to be as frozen as possible during the migration window. The bigger concern is the inconsistent error messages. When a user tries to create an issue and gets a raw HTTP 500, they are not going to think "oh, the repo is read-only." They are going to think something is broken. Combined with the fact that there is no visual indicator of read-only mode in the UI, this is confusing. Pipeline behavior is also uncertain. When I tried to create a new pipeline from the console during read-only mode, I got `Pipeline cannot be run.` The error message is too generic to know if this is caused by the read-only setting or something else. I did not have time to dig deeper into this. For our use case, it does not matter much since the maintenance window is short and we do not expect many pipeline runs during that time. ### Upgrading SonarQube: from 10.6.0 to 26.7.0 URL: https://anantafatur.dev/upgrading-sonarqube-from-10-6-0-to-26-7-0/ Last updated: 2026-08-03T13:52:25.000Z ## The plan I needed to upgrade SonarQube. The old version was `sonarqube:10.6.0-community`, running on a Kubernetes cluster with PostgreSQL on Tencent Cloud. The database was about 11 GB. How I ended up doing this upgrade is not a grand story. My backlog was clear that day and I had nothing to do. So I proposed upgrading the SonarQube version to my manager. He approved it. Just like that. We were not really concerned about security issues or feature compatibility. Honestly, I just thought a newer version would have fewer vulnerabilities. I confirmed this later with a Docker Scout image scan. The old image had a lot of CVEs. The new one had fewer. The old version, 10.6, was released on June 25, 2024\. That means this SonarQube instance had not been maintained for almost two years. Everyone was probably too busy with their own tasks. I was confused because I had nothing to do that day, so I took the risk and decided to pay this tech debt myself. The plan was simple. Clone the database, update the image tag in the Helm chart, push to GitOps, and let the migration run. Then upgrade again to the latest version. It did not go as planned. I hit four different issues across four different upgrades and downgrades. Here is what happened. --- ## First upgrade: sonarqube:10.6.0-community to sonarqube:24.12.0.100206-community ### Database clone I cloned the existing database to a new one. The clone command was simple: ```sql CREATE DATABASE sonarqubedevops_24_12 WITH TEMPLATE sonarqubedevops OWNER user_sonarqube; ``` It completed without errors. I updated the SonarQube JDBC config to point to the new database name. Nothing else changed. ```yaml jdbcUrl: "jdbc:postgresql:///sonarqubedevops_24_12?sslmode=require&socketTimeout=1500" jdbcUsername: "user_sonarqube" jdbcPassword: "" ``` I deployed the new pod with the `sonarqube:24.12.0.100206-community` image. I expected it to just work. But here's the catch. It did not. ### Error 1: "Database connection cannot be established" I opened the setup page and clicked "Setup Migration." The page showed: ``` Database connection cannot be established. Please check database status and JDBC settings. ``` I checked everything. The JDBC URL was correct. The username and password were correct. I tested the connection from the command line and it worked. I checked `pg_stat_activity` and saw the SonarQube pod was connecting successfully. But the setup page said it was not. I was stuck. ### Error 2: Migration fails during ALTER TABLE I took a different approach. I deleted the pod and PVC, deployed a fresh one, and checked the logs directly. The migration started: ``` Database needs to be migrated Executing 43 DB migrations... ``` Then it failed at step #107004: ``` org.postgresql.util.PSQLException: ERROR: must be owner of table github_perms_mapping ``` The SQL was equivalent to: ```sql ALTER TABLE github_perms_mapping RENAME TO devops_perms_mapping; ``` ### Finding the root cause I checked table ownership in the cloned database: ```sql SELECT schemaname, tablename, tableowner FROM pg_tables WHERE schemaname='public'; ``` Almost every table was owned by `root`, not by `user_sonarqube`. I checked the original database. Same thing. The original SonarQube had been running for years with `user_sonarqube` connecting to tables owned by `root`. The difference is that normal database operations (SELECT, INSERT, UPDATE, DELETE) only need proper privileges. But database migrations run DDL statements like ALTER TABLE and RENAME, which require table ownership. The original SonarQube never needed to run DDL, so the ownership mismatch was never exposed. ### The fix I changed ownership of all SonarQube objects from `root` to `user_sonarqube`: ```sql DO $$DECLARE r record; BEGIN FOR r IN SELECT schemaname, tablename FROM pg_tables WHERE schemaname='public' AND tableowner='root' LOOP EXECUTE format('ALTER TABLE %I.%I OWNER TO user_sonarqube;', r.schemaname, r.tablename); END LOOP; FOR r IN SELECT sequence_schema, sequence_name FROM information_schema.sequences WHERE sequence_schema='public' LOOP EXECUTE format('ALTER SEQUENCE %I.%I OWNER TO user_sonarqube;', r.sequence_schema, r.sequence_name); END LOOP; FOR r IN SELECT table_schema, table_name FROM information_schema.views WHERE table_schema='public' LOOP EXECUTE format('ALTER VIEW %I.%I OWNER TO user_sonarqube;', r.table_schema, r.table_name); END LOOP; FOR r IN SELECT n.nspname, p.proname, p.oid FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = 'public' LOOP EXECUTE format('ALTER FUNCTION %I.%I(%s) OWNER TO user_sonarqube;', r.nspname, r.proname, (SELECT string_agg(format_type(t.oid, NULL), ',' ORDER BY ORDINALITY) FROM unnest((SELECT proargtypes FROM pg_proc WHERE oid = r.oid)) WITH ORDINALITY AS t(oid, ORDINALITY))); END LOOP; END$$; ``` After running the script, I recreated the pod. The migration ran to completion. All 43 migrations passed. ### Two observations The database got smaller. The original was 11 GB. The new database after migration was 5.7 GB. Almost half the size. I am not sure exactly what caused it. Maybe the migration cleaned up old data, or the new version stores things more efficiently. Fun fact: this was a nice surprise I did not expect. Elasticsearch also reindexed automatically after the migration finished. I did not need to trigger anything. It ran in the background. The UI was usable during the process, but search results might not be complete until it finishes. --- ## Second upgrade: sonarqube:24.12.0.100206-community to sonarqube:26.1.0.118079-community The first upgrade was successful. I decided to push further and upgrade to `sonarqube:26.1.0.118079-community`. The database migration went fine. Elasticsearch reindexed itself. Everything looked good. Then I tried to log in with SAML. ### The SAML error After the SAML redirect from Azure AD, I got this page: ``` You're not authorized to access this page. Please contact the administrator. ``` The SonarQube logs showed the real error: ``` WARN org.sonar.server.authentication.AuthenticationError - Fail to callback authentication with 'saml' org.springframework.security.saml2.provider.service.authentication.Saml2AuthenticationException: Did not decrypt response [...] since it is not signed ``` The stack trace pointed to OpenSamlAuthenticationProvider. This is the Spring Security SAML2 library. I checked the SAML configuration in the database: ``` sonar.auth.saml.sp.certificate.secured -> (SP encryption certificate) sonar.auth.saml.sp.privateKey.secured -> (SP private key) sonar.auth.saml.signature.enabled -> true ``` The Azure AD Enterprise Application had this setting: ``` Signing Option: Sign SAML assertion Signing Algorithm: SHA-256 ``` The Azure AD SAML response was encrypted using the SP certificate, but only the assertion inside the response was signed. The response envelope itself was not signed. ### Root cause In `sonarqube:24.12.0.100206-community`, the OpenSAML library (version 3.x) allowed decryption of encrypted responses even if the response was not signed. In `sonarqube:26.1.0.118079-community`, the Spring Security SAML2 library uses OpenSAML 4.x. OpenSAML 4.x enforces a stricter security policy: if the response is encrypted, it must also be signed. Otherwise, it refuses to decrypt. This is actually documented in the SonarQube 2025.1 release upgrade notes under **SAML configuration update required**: > When configuring SAML on your SonarQube Server instance with assertion encryption, the response signature must be enforced. If you use SAML with Microsoft Entra, make sure you sign the response by selecting **Sign SAML response** or **Sign SAML response and assertion** as the sign-in response. I should have read the release notes before upgrading. But honestly, I did not. I just assumed the upgrade would be smooth since the first one was fine. ### What I tried I tried removing the SP certificate and private key from the SAML configuration. My thinking was that without them, SonarQube would stop advertising encryption support, and Azure AD would send the response unencrypted. I cleared both fields in the SonarQube UI under Administration > Authentication > SAML > Service provider section. I saved the configuration. I deleted the properties from the database directly. I restarted the pod. The error persisted. The same "Did not decrypt response since it is not signed" error. This probably means Azure AD caches the SP metadata and still encrypts the response. Or the encryption is configured on the Azure AD side independent of the SP metadata. I could not verify this because I am not the Azure AD admin. ### The real fix The real fix was exactly what the release notes said. I asked the Azure AD admin to change the SAML signing option. In the Azure AD Enterprise Application, the admin went to **SAML-based Sign-on** \> **SAML Signing Certificate** \> **Edit**, and changed: ``` Signing Option: Sign SAML assertion ``` to: ``` Signing Option: Sign SAML response and assertion ``` This makes Azure AD sign the response envelope, which satisfies OpenSAML 4.x security policy. But I could not do this myself. I needed to ask the Azure AD admin, and it was a Friday evening. The admin would not be available until Monday. So I decided to downgrade back to `sonarqube:24.12.0.100206-community` so the team could work over the weekend. --- ## Downgrade: sonarqube:26.1.0.118079-community back to sonarqube:24.12.0.100206-community I redeployed the `sonarqube:24.12.0.100206-community` image using the same PVC. The pod went into `CrashLoopBackOff`. The log showed: ``` Could not load codec 'Lucene912'. Did you forget to add lucene-backward-codecs.jar? ``` The Elasticsearch data directory (`/opt/sonarqube/data/es8`) still contained index data created by the newer version. The older Lucene library bundled with `sonarqube:24.12.0.100206-community` could not read the `Lucene912` codec format used by the newer version. The fix was simple. Delete the Elasticsearch index directory and let SonarQube rebuild it: ```bash rm -rf /opt/sonarqube/data/es8 ``` After deleting the directory and restarting the pod, SonarQube started successfully. Elasticsearch recreated the index from the database automatically. ## SAML fix: back to sonarqube:26.1.0.118079-community On Monday, the Azure AD admin changed the signing option to "Sign SAML response and assertion." After the change, I recreated the SonarQube pod with the `sonarqube:26.1.0.118079-community` image. The SAML login worked immediately. The second upgrade was finally successful. The fix was on the Azure AD side, not the SonarQube side. The SP certificate and private key needed to stay in the SonarQube SAML configuration. The only change was the Azure AD signing option. At this point, the upgrade path was: `sonarqube:10.6.0-community` \-> `sonarqube:24.12.0.100206-community` \-> `sonarqube:26.1.0.118079-community`. Three issues, one downgrade, and a weekend of running an older version. But it got done. --- ## Third upgrade: sonarqube:26.1.0.118079-community to sonarqube:26.7.0.124771-community After the SAML encryption issue was resolved and `sonarqube:26.1.0.118079-community` was running smoothly, I decided to upgrade again to `sonarqube:26.7.0.124771-community`. I deployed the new image. The pod went into `Running` state. But the ingress at the SonarQube URL returned `503 Service Temporarily Unavailable`. I checked the pod events: ``` Readiness probe failed: sh: 5: wget: not found Liveness probe failed: sh: 2: wget: not found ``` The Helm chart's liveness and readiness probes were using `wget` for health checks. The new `sonarqube:26.7.0.124771-community` image no longer includes `wget`. The startup probe used `httpGet` (kubelet-native) and was not affected. That is why the pod appeared `Running` even though the probes were failing. But because the readiness probe never passed, the Service had no ready endpoints, and the ingress could not route traffic to the pod. ### The fix I replaced `wget` with `curl` in the probe commands across the Helm chart templates. In `templates/sonarqube-sts.yaml` and `templates/deployment.yaml`: ```diff # Liveness probe - wget --no-proxy --quiet -O /dev/null --timeout=... --header=... "...api/system/liveness" + curl -s -f --max-time ... --header ... "...api/system/liveness" > /dev/null # Readiness probe - wget --no-proxy -qO- ...api/system/status | grep -q -e '"status":"UP"' ... + curl -s --max-time ... "...api/system/status" | grep -q -e '"status":"UP"' ... ``` After applying the changes and syncing via GitOps, the probes started passing, the Service had ready endpoints, and the ingress returned `200`. This was the simplest issue so far. No database changes, no Azure AD changes, no Elasticsearch crashes. Just a missing binary in the container image. So yeah, that was the fourth issue. --- ## What I learned The real lesson is that this SonarQube instance sat untouched for almost two years. Nobody noticed. Nobody cared. Until one day I had a clear backlog and nothing to do. I hope someone in the future does not let it drift that long again. Maintaining versions is boring. But paying two years of tech debt over two weeks is worse. ### Migrating SonarQube from AAD Plugin to Built-in SAML URL: https://anantafatur.dev/migrating-sonarqube-from-aad-plugin-to-built-in-saml/ Last updated: 2026-07-25T14:53:22.000Z We had a SonarQube instance running on Kubernetes. It used the community AAD plugin (`sonar-auth-aad`) for authentication with Microsoft Entra ID. The plugin worked, but it was a third-party plugin with no guaranteed future. SonarQube 10.6 has built-in SAML support, so I wanted to migrate to that. What I thought would be a simple configuration change turned into a two-week journey through SAML encryption, callback URL mismatches, and database-level user migration. Here is what happened. ## The Setup Our SonarQube lives on a Tencent Cloud TKE cluster, behind quite a few layers: ``` User | HTTPS v Cloudflare | HTTPS v Public CLB (Tencent Cloud) <- TLS terminates here | HTTP v Private CLB (Tencent Cloud) | HTTP v NGINX Ingress Controller | HTTP v SonarQube (10.6 Community Build) ``` The key detail: TLS terminates at the public CLB. Everything downstream from that point is plain HTTP. This becomes important later. --- ## Phase 1: SAML Configuration Setting up SAML itself was simple. In the SonarQube admin UI, under **Administration > Configuration > SAML**, I filled in the values from the Entra ID Enterprise Application. The docs map everything clearly: Application ID, Provider ID, Login URL, the IdP certificate, and the user attribute claims. I hit **Test Configuration** and got a NullPointerException. Not exactly encouraging. The error was misleading. The real problem was in the logs: ``` XMLCipher::decryptElement unable to resolve a decryption key Error executing decryption: encryption.nokey ``` Our Entra ID app had **Token Encryption** enabled. It was sending encrypted SAML assertions, and SonarQube had no key to decrypt them. The fix was to generate a PKCS8 private key and certificate, upload the certificate to Entra ID, and configure both in SonarQube under `sonar.auth.saml.sp.privateKey.secured` and `sonar.auth.saml.sp.certificate.secured`. After that, the test passed. The test configuration showed all the expected attributes coming through. Things looked good. I was optimistic. --- ## Phase 2: The Callback URL Nightmare I clicked the SAML login button on the SonarQube login page. Entra ID authenticated me. Then I got this: ``` The response was received at http://sonarqube.example.com/oauth2/callback/saml instead of https://sonarqube.example.com/oauth2/callback/saml ``` SonarQube's SAML library (OneLogin) constructs the callback URL based on the perceived request scheme. The request arrives at SonarQube as HTTP, so it constructs `http://...` callbacks. But the SAML response expects `https://`. The library rejects the mismatch. This is where I spent most of the time. Here is everything I tried that did not work. ### Attempt 1: Tencent Cloud CLB Custom Config The public CLB has a custom configuration feature. I tried adding `proxy_set_header X-Forwarded-Proto https;` there. It did not work because Tencent Cloud CLB only supports a limited set of variables for `proxy_set_header`: `$vport`, `$remote_port`, `$lbid`, `$stgw_request_id`, `$request_method`, `$uri`. You cannot set a literal string value like `https`. The documentation confirmed this. ### Attempt 2: Cloudflare Request Header Transform Rules Cloudflare has rules to modify request headers. I tried setting `X-Forwarded-Proto: https` at the Cloudflare level. Cloudflare blocks this. They explicitly document that you cannot modify `x-forwarded-proto` in request header transform rules. It is a security measure to prevent header spoofing. ### Attempt 3: NGINX Ingress Controller Snippets and ConfigMaps I tried several approaches at the NGINX Ingress level: **`server-snippet` annotation:** Adding `proxy_set_header X-Forwarded-Proto https;` in a server snippet. The NGINX controller overrides this with its own `$pass_access_scheme` value, which is `http` because the incoming traffic from the CLB is HTTP. **`proxy-set-headers` ConfigMap:** Created a ConfigMap with `X-Forwarded-Proto: https` and referenced it via the `nginx.ingress.kubernetes.io/proxy-set-headers` annotation. Same problem. The controller's own proxy header logic wins. **`configuration-snippet` annotation:** Same result. Whatever I set, the controller's calculated `$pass_access_scheme` overrides it. **Changing backend protocol:** I tried setting `nginx.ingress.kubernetes.io/backend-protocol: HTTPS`. This caused the Ingress to send HTTPS to the backend. But SonarQube listens on HTTP (port 9000). The result was a 404 and the application failing to start. ### Attempt 4: Spring Boot Forward Headers Strategy I tried adding `server.forward-headers-strategy=NATIVE` to SonarQube's JVM options. This is a Spring Boot property that tells the application to trust the `X-Forwarded-*` headers. But the problem was upstream. The header was already wrong (`http`) by the time it reached this point. Trusting a wrong header does not help. ### What Actually Worked: A Sidecar NGINX The problem was that no component in the chain could set `X-Forwarded-Proto: https` as a literal value. The CLB could not do it. Cloudflare would not allow it. The NGINX Ingress controller only uses dynamic values from the connection. The only place left was inside the pod itself. I added a sidecar NGINX container that sits in front of SonarQube and hardcodes the `X-Forwarded-Proto` header. ``` SonarQube Pod ├── SonarQube Container (port 9000) └── NGINX Sidecar Container (port 8080) │ │ Sets X-Forwarded-Proto: https │ Sets X-Forwarded-Port: 443 ▼ SonarQube (localhost:9000) ``` The sidecar config is simple: ```nginx server { listen 8080; client_max_body_size 100m; location / { proxy_pass http://localhost:9000; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Port 443; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $http_host; } } ``` The Ingress now points to the sidecar (port 8080) instead of directly to SonarQube (port 9000). The sidecar receives the HTTP request from the Ingress, adds `X-Forwarded-Proto: https`, and proxies to SonarQube. SonarQube sees the correct header. The SAML callback URL is now `https://`. This is admittedly a hack. A sidecar container just to set one header. But it was the only thing that worked across all the layers we could not control. One side effect I hit later: the sidecar NGINX has a default `client_max_body_size` of 1MB. The SonarQube CE submit endpoint sends payloads larger than that. I got a 413 error. Adding `client_max_body_size 100m;` to the sidecar config fixed it. The original Ingress had `proxy-body-size: 100m`, but the sidecar was the bottleneck. --- ## Phase 3: Migrating Users from AAD to SAML The SAML login was working. But there was a problem. Our existing users were all created with `external_identity_provider = 'aad'` in the database. When they clicked SAML login, SonarQube did not match them to their existing account. Instead, it created a brand new user with `saml` provider and default permissions. No groups, no project access. The original AAD user with all the correct permissions was still there, untouched. I needed to change every user's provider from `aad` to `saml`. ### The API Approach That Failed My first instinct was to use SonarQube's Web API. There is an endpoint: `api/users/update_identity_provider`. I tried it: ```bash curl -X POST -u admin: \ "https://sonarqube.example.com/api/users/update_identity_provider?login=&newExternalProvider=saml" ``` It returned a 500 error. The endpoint was deprecated in SonarQube 10.4\. After deprecation, it just refuses to work. I tried other API approaches. Deactivating and reactivating the user. Calling `api/users/update` with different parameters. None of them worked. The `api/users/reactivate` endpoint does not even exist in 10.6. The API was not going to work. I had to go to the database. ### The Database Migration Approach The SonarQube database is PostgreSQL. The `users` table has columns `external_identity_provider`, `external_login`, and `external_id`. The migration needed to change `external_identity_provider` from `aad` to `saml`, and update the `external_login` and `external_id` to match what SAML returns. Here is the catch. SAML returns the `NameID` claim with the **domain part lowercased** but the **prefix preserved as-is**. So if the original AAD login was `USER.LOGIN@COMPANY.CO.ID`, the SAML identity would be `USER.LOGIN@company.co.id`. Not all lowercase. Just the domain. I learned this by trial and error. The first few users I migrated manually, I lowercased everything. They could not log in. I checked the SAML test configuration output and noticed the pattern: the prefix was preserved, only the domain changed. The migration script I wrote processes each user one by one: 1. Fetch all users with `external_identity_provider = 'aad'` 2. For each user, compute the lowercased domain version of `external_login` 3. Delete any existing SAML user with that external identity (these are duplicates created by SonarQube when users tried SAML login before the migration) 4. Update the user's `external_identity_provider`, `external_login`, and `external_id` The script supports dry-run mode and single-user mode for testing. The first version used `kubectl run` to create a PostgreSQL pod for each query, which was painfully slow. About 30 minutes for 100 users. I rewrote it to use `psql` directly from the jumphost, which finishes all \~300 users in about five seconds. The good news is that group memberships and permissions are preserved. The `groups_users` table references `user_uuid`, which we never change. We only update `external_identity_provider`, `external_login`, and `external_id`. So after migration, the user logs in via SAML and sees all their projects and permissions exactly as before. ## The Final State After the migration is complete, the steps are: 1. Disable the AAD plugin in the SonarQube UI 2. Delete the `sonar-auth-aad-*.jar` from the extensions directory 3. Restart SonarQube Users see only the SAML login button. They log in with their Entra ID credentials. Their permissions are the same as before. ### A Sedative Side Effect: Two Weeks of Sleep Data URL: https://anantafatur.dev/a-sedative-side-effect-two-weeks-of-sleep-data/ Last updated: 2026-07-15T09:58:18.000Z I got a medicine from someone. It was not for sleep. I googled the name and it is not classified as a sleep aid. But it has sedative effects listed in the side effects. And my sleep data showed a clear before and after. This is what I saw. ### The Setup I have been tracking my sleep with a Huawei Fit 3 smartwatch. But I do not wear it every night. The "before" data is just random nights from March and April where I happened to sleep with the smartwatch. Not every night, not consistent. The "after" data is about two weeks in July after I started the medicine. The data includes: - Sleep duration (hours and minutes) - Bed time and wake time - Sleep stages (deep, light, REM) - A sleep quality score (0-100) I plotted everything side by side to see if the effect was real. --- ### Sleep Duration Went Up Before the medicine, my average sleep duration was about 7.3 hours. Some nights I got as little as 5.8 hours. After, it jumped to about 9.3 hours. That is a 2-hour increase. Keep in mind the before data is messy. I did not track every night. But the pattern is still visible. ![Sleep duration, bed time, and score comparison](https://anantafatur.dev/content/images/2026/07/sleep_chart.png) Sleep Metrics Before vs. After Medication The individual data points show the difference clearly. Before, the points cluster around 6-8 hours. After, they sit around 9-10 hours. No overlap. **But here is the thing.** The medicine did not make me fall asleep instantly. Before, it took me 30 to 90 minutes. After, it takes me 15 to 60 minutes. So the improvement is real, but it is not magic. I still need some time to wind down. ### Bed Time Shifted Three Hours Earlier This was the most obvious change. Before the medicine, I went to bed between 00:11 and 02:51\. After, I went to bed between 21:16 and 00:25. My average bed time moved from around 01:30 AM to around 10:30 PM. That is a three-hour shift earlier. The reason is simple. Before the medicine, it took me 30 to 90 minutes to fall asleep. I knew this. So I delayed going to bed. Why lie in bed awake for an hour if I could just stay up and do something else? The medicine changed that. I felt confident I could fall asleep quickly. So I went to bed earlier without dreading the wait. My wake time stayed roughly the same (around 08:40-08:55 AM) because I have to be at work at 9:30\. So I was not going to bed earlier and waking up earlier. I was going to bed earlier and sleeping longer. ### Sleep Scores Did Not Improve Here is the strange part. My sleep quality score did not get better. It actually went down slightly. Before the medicine, my average score was around 83\. After, it was around 81\. This is not a big drop, but it is not the improvement I expected. The score comes from Huawei TruSleep. It tracks heart rate, movement, and breathing to detect sleep stages \[1\]. Huawei says a normal person usually experiences 4 to 6 sleep cycles per night, with each cycle lasting 90 to 100 minutes \[2\]. That works out to 6 to 10 hours of sleep. So the algorithm considers this the healthy range. Some of my nights after the medicine went over 10 hours (10h 43min, 10h 17min, 10h 10min). That is outside the healthy range according to Huawei. So the algorithm might be lowering the score because of that, not because the sleep quality is actually worse. It makes sense if you think about it. The score is built around a 6-10 hour window. If you sleep beyond that, the algorithm does not know how to score it positively. It treats it as outside the norm. I do not think my sleep got worse. I think the scoring system was not designed for someone sleeping 10+ hours. ## Sleep Stage Composition Changed I looked at deep, light, and REM sleep to see if the mix shifted. ![Deep, light, and REM sleep comparison](https://anantafatur.dev/content/images/2026/07/sleep_stages_chart.png) Sleep Stage Composition: Before vs. After Medication The data shows a shift in REM sleep. Before, REM made up about 15.6% of my total sleep. After, it went up to about 18.4%, with one night hitting 27.4%. The absolute minutes went up too, but that is expected since I slept longer. The percentage increase is a fairer comparison. Deep sleep stayed relatively consistent in both periods. Adequate and quality REM sleep is critical for optimizing cognitive function, emotional regulation, and memory consolidation \[3\]. I am still figuring out why my REM sleep increased. It could be because I sleep longer overall, so I naturally spend more time in REM. Or it could be a direct effect of the medicine. But I cannot say for sure if this is the medicine or just natural variation. ### Consistency vs. Timing Before the medicine, my bed time was reliably around 1-2 AM. That kind of consistency was not a good thing. After the medicine, my bed time shifted to between 9 PM and midnight. That is a much healthier range. ![Sleep schedule timeline and consistency analysis](https://anantafatur.dev/content/images/2026/07/sleep_analysis_chart.png) Sleep Analysis: Schedule, Deep Sleep, and Consistency Before the medicine, my bed time varied by about 50 minutes. After, it varied by about 65 minutes. Some nights I went to bed at 21:16\. Other nights at 00:25\. The same happened with wake time. Part of that is because I work as a DevOps engineer. Some days I still have to work until midnight. So the late bed times are not the medicine failing. They are just work. Also, I take the pill when I am ready to sleep. Not at a fixed time. So the bed time range after medicine is really just when I chose to go to bed. The medicine does not force sleep at a specific hour. It just makes it easier to fall asleep when I decide to. To be fair, the before data was already inconsistent. I was not tracking every night. So the before baseline is not a perfect measure of consistency either. It might be because I was more relaxed about my schedule after the medicine. I was not stressed about falling asleep anymore, so I let myself drift. --- ### What I Learned The medicine changed my sleep. That is clear from the data. I sleep longer, I fall asleep earlier, and my sleep stage composition shifted. But the data also shows things I did not expect. My sleep score went down. My bed time became more varied, but within a much healthier range. I still take time to fall asleep, just less than before. If I only looked at the first few nights, I would have said this was a miracle. Looking at all the data together, the picture is more mixed. This is not a recommendation. I am not saying anyone should take this medicine for sleep. I am just sharing what I observed in my own data. Your mileage will vary. Probably a lot. ### References \[1\] "HUAWEI TruSleep," Huawei Community. (accessed Jul. 15, 2026). \[2\] "Introduction to sleep monitoring principles of HUAWEI smart wearables," Huawei Support. (accessed Jul. 15, 2026). \[3\] P. Vorng, "The Role of Sleep in Cognitive Function and Memory," SSRN Journal, 2024, doi: 10.2139/ssrn.4993119. \[4\] Y. Abdullah, "Deep Sleep Therapy: A Comprehensive Guide to Restorative Sleep Solutions," Journal of Sleep Disorders & Therapy, 2024, doi: 10.35248/2167-0277.24.13.564. ### Reducing PostgreSQL Storage on TencentDB by Cleaning pg_log URL: https://anantafatur.dev/reducing-postgresql-storage-on-tencentdb-by-cleaning-pg_log/ Last updated: 2026-07-13T04:11:05.000Z Last week I got an alert from my friend's monitoring system. The PostgreSQL database storage utilization was at 139.68%. That number looked wrong. TencentDB allows storage to scale above the committed size, but 139% is still unusual. The database was not even in use yet. It was still running an incremental DTS task. How could the data be that high? I checked the TencentDB system monitoring. The used storage was valid according to the dashboard. It showed 55GB used, with only 40GB committed. ![](https://anantafatur.dev/content/images/2026/07/image-1.png) TencentDB - System Monitoring Then I checked the actual database size with a query. ```sql SELECT datname AS database, pg_size_pretty(pg_database_size(datname)) AS size FROM pg_database ORDER BY pg_database_size(datname) DESC; ``` ![](https://anantafatur.dev/content/images/2026/07/image-2.png) TencentDB - Database Management Center The result showed only 26GB across all databases. There was a gap of almost 30GB between what PostgreSQL reported and what TencentDB showed. I had no idea where the extra space went. Maybe WAL files? Maybe logs? I was not sure. So I raised a ticket to Tencent support. --- ### What Tencent Support Found Their answer came back quickly. Here is what they said: > We have confirmed that the WAL logs occupy 8GB, and the remaining 48GB of storage is taken up by pg\_log and data files. > The WAL parameters cannot be reduced, as an overly small setting will cause synchronization failures between primary and standby instances. You may consider adjusting the pg\_log retention settings to free up space. So the breakdown was: - Database: 26GB - WAL logs: 8GB (cannot be reduced) - pg\_log and data files: 48GB (could be reduced) - Total: 55GB (committed: 40GB) The pg\_log was the culprit. TencentDB for PostgreSQL retains pg\_log by default for 30 days. That adds up quickly. ### Reducing pg\_log Retention The fix was to reduce the log retention from 30 days to 7 days. TencentDB has a parameter called `log_filename` that controls this. The parameter accepts two values: - `postgresql_%d_%H.log` — keeps logs for 30 days (default) - `postgresql_%a_%H.log` — keeps logs for 7 days There is also `tencentdb_max_logfile_num` which sets the maximum number of log files. The system generates one log file per hour. So for 7 days retention, you need at least 7 x 24 = 168 files. ### Step 1: Change log\_filename In the TencentDB console, go to the instance details page, find Parameter Settings, and search for `log_filename`. Change it to `postgresql_%a_%H.log`. ### Step 2: Change tencentdb\_max\_logfile\_num In the same Parameter Settings page, search for `tencentdb_max_logfile_num` and set it to `168`. The minimum value for this parameter is 100\. The value `2147483647` means retain everything. ### Step 3: Rotate logs immediately After changing the parameters, you can force a log rotation to clean up immediately. ```sql SELECT pg_rotate_logfile(); ``` This kicks the kernel to rotate and clean old log files based on the new retention settings. ### Step 4: Verify the changes Run this query to confirm both parameters are applied. ```sql SELECT name, setting, unit, context FROM pg_settings WHERE name IN ('log_filename', 'tencentdb_max_logfile_num'); ``` The result should look like this. ``` "log_filename","postgresql_%a_%H.log",,"sighup" "tencentdb_max_logfile_num","168",,"sighup" ``` --- ### The Result After applying the changes, the used storage dropped to 36GB. That is a reduction of 19GB just from cleaning logs. ![](https://anantafatur.dev/content/images/2026/07/image-3.png) TencentDB - System Monitoring The database itself was still 26GB. The remaining 10GB overhead came from WAL (8GB) plus some extra data files that Tencent support mentioned. The WAL cannot be reduced, so this is as low as it gets. Reference: [TencentDB for PostgreSQL - Execution Log Management](https://www.tencentcloud.com/document/product/409/59363) ### Caching Terraform Modules in a Kubernetes CI Pipeline URL: https://anantafatur.dev/caching-terraform-modules-in-a-kubernetes-ci-pipeline/ Last updated: 2026-07-12T10:56:48.000Z I wrote about caching Terraform providers a while back. That solved one part of the `terraform init` slowness. But there was a bigger fish. The modules. Our Terraform code uses git-sourced modules. A lot of them. Each environment pulls in 30-50 modules from our internal git repos. In an ephemeral Kubernetes Jenkins agent, every pipeline run starts with an empty `.terraform` directory. So `terraform init` has to git-clone every single module. One by one. Over SSH. Here is what it looked like before any caching: ``` 09:57:46 - in .terraform/modules/ 09:57:46 Downloading git::ssh://git@//.git?ref= for ... 09:57:46 - in .terraform/modules/ 09:57:46 Downloading git::ssh://git@//.git?ref= for ... ... 09:58:24 Downloading git::ssh://git@//.git for ... ``` Start at 09:57:46\. The last module lands at 09:58:24\. That is 38 seconds of nothing but git clones. Not terrible in isolation. But this is just the init phase. In the MR pipeline, we run init separately for plan and apply stages. And in the full pipeline, init runs for every environment. Those seconds add up. --- ### The Problem Terraform has a built-in cache for providers. Set `TF_PLUGIN_CACHE_DIR`, point it at a shared directory, and providers get cached automatically. There is no equivalent for modules. Providers are just binaries. Terraform downloads them, checks the hash, stores them in the plugin cache. Next time, it reads the binary directly. Modules are trickier. Each module is a git checkout. A directory with `.tf` files. Terraform handles module source by cloning the git repo into `.terraform/modules//`. There is no config option to redirect that to a shared cache. So I had to build my own. ### How Terraform Decides to Skip a Module Before getting into the solution, it helps to understand how Terraform decides whether a module is "already installed." This is the key insight that makes the cache work. During `terraform init`, Terraform reads `.terraform/modules/modules.json`. This manifest maps each `module "..." {}` block in your code to a directory on disk. For each module, if three things match (Key, the module name in your code like `vpc` or `tke_cluster`, Source, the git URL and ref like `git::...?ref=`, and Dir, the local directory exists on disk), Terraform skips the git clone. It prints "already installed" for that module and moves on. If any of these fields don't match, Terraform re-downloads only that module. This is called manifest-gated skipping. The `modules.json` file is the gatekeeper. And it means the cache approach is straightforward. Save the entire `.terraform/modules/` directory (including `modules.json`) to a PVC after init. Restore it before the next init. If the source refs haven't changed, everything matches and nothing needs to download. --- ## The Approach The same PVC from the provider caching post is mounted at `/cache`. The module cache lives in `/cache/terraform/modules/`. ### Per-Project Isolation The cache key is built from the Jenkins job name and environment name. This keeps modules from different projects or environments from mixing. ```groovy MODULE_CACHE_KEY=$(echo "${JOB_NAME}_${environment}" | tr '/' '_') MODULE_CACHE_DIR="/cache/terraform/modules/${MODULE_CACHE_KEY}" ``` `JOB_NAME` is a Jenkins env variable like `///`. After replacing `/` with `_`, the cache path becomes `/cache/terraform/modules/___/`. ### Restore Before Init ```sh mkdir -p .terraform/modules if [ -d "${MODULE_CACHE_DIR}" ] && [ -n "$(ls -A ${MODULE_CACHE_DIR} 2>/dev/null)" ]; then cp -rn "${MODULE_CACHE_DIR}"/* .terraform/modules/ fi ``` This copies the cached `modules.json` and all module directories into the workspace before init runs. Terraform reads the manifest and decides what to skip. ### Save After Init ```sh mkdir -p "${MODULE_CACHE_DIR}" if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/" fi ``` After init resolves and downloads everything, the save step writes the full `.terraform/modules/` back to the PVC. This overwrites the previous cache entirely. That is intentional. When a module ref changes, `modules.json` is rewritten by init. The save step must replace the old manifest with the new one. --- ### How Changes Propagate The cache is self-updating. Here is what happens for each scenario: **Cache hit on second run.** All module sources match the restored `modules.json`. Terraform sees everything as "already installed." Zero git clones. The init phase goes from 38 seconds to about 4 seconds (the overhead of copying the cached files from the PVC into the pod). **Ref bump.** A developer changes `?ref=v1.0` to `?ref=v2.0` in one module source. The restored `modules.json` still references v1.0 for that module. Terraform detects the source mismatch. It re-downloads only the changed module from git. Every other module is still "already installed." Save step overwrites the cache with the updated manifest. **New module added.** A developer adds a new `module "foo" {}` block. The restored `modules.json` has no entry for `foo`. Terraform downloads only the new module. Save updates the cache. **Module removed.** The stale directory stays on the PVC but is never referenced by `modules.json`. Terraform ignores it. Harmless and tiny (< 200 KB). --- ### The Full Pipeline Stage Here is the relevant block from the Jenkins shared library. It runs in every stage that calls `terraform init`. ```groovy stage('Terraform Init') { steps { container('terraform') { sh ''' cd "${environment}" MODULE_CACHE_KEY=$(echo "${JOB_NAME}_${environment}" | tr '/' '_') MODULE_CACHE_DIR="/cache/terraform/modules/${MODULE_CACHE_KEY}" LOCK_CACHE_DIR="/cache/terraform/lock-files/${MODULE_CACHE_KEY}" # Restore modules from cache mkdir -p .terraform/modules if [ -d "${MODULE_CACHE_DIR}" ] && [ -n "$(ls -A ${MODULE_CACHE_DIR} 2>/dev/null)" ]; then cp -rn "${MODULE_CACHE_DIR}"/* .terraform/modules/ echo "[CACHE] Modules restored" fi # Restore lock file from cache mkdir -p "${LOCK_CACHE_DIR}" if [ -f "${LOCK_CACHE_DIR}/.terraform.lock.hcl" ]; then cp "${LOCK_CACHE_DIR}/.terraform.lock.hcl" .terraform.lock.hcl echo "[CACHE] Lock file restored" fi terraform init -input=false # Save modules to cache mkdir -p "${MODULE_CACHE_DIR}" if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/" echo "[CACHE] Modules saved" fi # Save lock file to cache if [ -f .terraform.lock.hcl ]; then cp .terraform.lock.hcl "${LOCK_CACHE_DIR}/" echo "[CACHE] Lock file saved" fi ''' } } } ``` In the pipeline there are 5 call sites wrapped with this pattern. Every place that runs `terraform init` gets the same restore/save logic. Manual plan, manual import, MR triggered plan. All of them. --- ### The Results Here is the save operation after a cached init. Terraform found all modules already present, so it did zero git downloads. The only work was copying the modules back to the PVC. ``` 16:59:37 + mkdir -p /cache/terraform/modules/___ 16:59:37 + '[' -d .terraform/modules ] 16:59:37 + ls -A .terraform/modules 16:59:37 + '[' -n ' ... ' ] 16:59:37 + cp -r .terraform/modules/ ... /cache/terraform/modules/___/ ``` Performance on a project with 30+ git-sourced modules: ``` +------------------------+-----------------------+-----------------------------+ |Scenario |Before |After | +------------------------+-----------------------+-----------------------------+ |First run (cold cache) |~38s (download all) |Same (populates cache) | |Second run (cache hit) |~38s (download all |~4s (PVC copy overhead) | | |again) | | |Module version bump |~38s |~5s (only changed module) | +------------------------+-----------------------+-----------------------------+ ``` --- ## Gotchas ### Alpine `cp` Syntax Our Jenkins agents run Alpine-based images. Busybox `cp` does not support the `dir/.` syntax for copying directory contents. It silently copies nothing. ```sh # Silent failure on Alpine cp -rn /cache/modules/. .terraform/modules/ # Works everywhere cp -rn /cache/modules/* .terraform/modules/ ``` I found this the hard way. The cache directory was populated on the PVC. The restore step ran without errors. But `terraform init` still downloaded every module because nothing was actually copied. The `/*` glob is the portable approach and works on Alpine, Ubuntu, and every other base image. ### Self-Healing on Cache Corruption The cache is best-effort only. If the PVC is unreachable or a cache directory is corrupted, the `cp` fails silently. `.terraform/modules/` stays empty. `terraform init` runs normally and downloads everything from git. The next run's save step writes a fresh copy to the PVC. No pipeline ever breaks because of the cache. ### The Empty Cache Bug There is one edge case that slipped through during development and was reported by a user after release. It was blocking their pipeline entirely. The original save check was too simple: ```sh # Before -- only checks directory exists if [ -d .terraform/modules ]; then cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/" fi ``` I assumed that if `.terraform/modules/` exists, it has content. That is not always true. Here is the scenario. A user pushed an incomplete Terraform project that had no modules defined yet. The pipeline creates `.terraform/modules/` with `mkdir -p` in the restore step. The cache is cold. Since the code has no `module "..." {}` blocks, `.terraform/modules/` stays empty through the init step. The save step checks `[ -d .terraform/modules ]` – true, the directory exists. So the `cp -r` runs. But the directory is empty. In bash, with default settings, the `*` glob expands to the literal character `*`. The command becomes `cp -r "..."/* "${MODULE_CACHE_DIR}/"`, trying to copy a file literally named `*`. That file does not exist. The cp command fails. The pipeline fails with it. The fix adds a content check alongside the existence check: ```sh # After -- also checks directory has content if [ -d .terraform/modules ] && [ -n "$(ls -A .terraform/modules 2>/dev/null)" ]; then cp -r .terraform/modules/* "${MODULE_CACHE_DIR}/" fi ``` The `[ -n "$(ls -A ...)" ]` part returns false when the directory is empty. The `cp` never runs. The pipeline continues normally. Terraform init then downloads modules from git as if there was no cache at all. I wanted to document this because the symptom was confusing. The cache directory existed on the PVC after a successful `mkdir -p`, so it looked like a successful cache. But there was nothing inside. The failure happened during the save step, not the restore step, which made it harder to trace. --- ### Concurrency The PVC is backed by CFS with `ReadWriteMany`. Multiple pods can mount and write to it at the same time. The module cache key is unique per job and environment, so concurrent runs of different pipelines do not collide. Two runs of the same pipeline at the same time would normally race to write to the same cache directory. But Jenkins is configured to only allow one concurrent run per pipeline. If a job is triggered while another is already running, it queues instead of running in parallel. So this edge case never actually happens. ### Combined With Provider Cache The full cache directory structure on the PVC: ``` /cache/ .terraformrc terraform/ plugins/ # Provider cache (built-in, TF_PLUGIN_CACHE_DIR) modules/ # Module cache (manual, per-project) lock-files/ # Lock file cache (manual, per-project) tflint/ plugins/ # TFLint plugin cache ``` The provider cache and the module cache sit side by side. The lock file cache completes the picture. Together they turn `terraform init` from a network-heavy multi-second operation into a mostly-local file copy. ### What I Would Do Differently The first run after a cache miss still downloads everything from git. This is unavoidable. But if the cache was prepopulated with known module versions from a build image or a periodic seeding job, even the first run would be fast. I haven't done this because the cache hit rate is already high. One thing I did not explore: using `rsync` instead of `cp` for the save step. It would reduce PVC I/O by only writing changed files. For now `cp -r` is good enough. The entire `.terraform/modules/` directory is maybe 50-200 MB across all projects. ### Caching Terraform Providers in a Kubernetes CI Pipeline URL: https://anantafatur.dev/caching-terraform-providers-in-a-kubernetes-ci-pipeline/ Last updated: 2026-07-24T17:50:38.000Z At work, everything is already IaC. Terraform runs in Jenkins pipelines. The flow is simple. Someone pushes IaC code to our repo, a webhook triggers the pipeline, Jenkins clones the code, runs `terraform init`, then `terraform plan` or `terraform apply`. Done. The Jenkins agents run on Kubernetes. Every agent pod is ephemeral. Created for the job, destroyed after. This means every single pipeline run starts from scratch. And that includes `terraform init` downloading providers from the internet every time. Honestly, this was never really a problem. A provider is a couple MB. The download takes a few seconds. Not something worth losing sleep over. But last week the backlog was clear. So it was time to poke around. --- ### The Setup The pod template has two main containers: `terraform` and `tflint`. Both run inside a Kubernetes pod defined in a YAML file. A PVC named `terraform-provider-cache` is mounted at `/cache` in both containers. ```yaml volumes: - name: tf-cache persistentVolumeClaim: claimName: terraform-provider-cache containers: - name: terraform image: our-registry/terraform:$TF_VERSION env: - name: TF_CLI_CONFIG_FILE value: /cache/.terraformrc - name: TF_PLUGIN_CACHE_DIR value: /cache/terraform/plugins volumeMounts: - name: tf-cache mountPath: /cache - name: tflint image: our-registry/terraform:tflint-$TFLINT_VERSION env: - name: TFLINT_PLUGIN_DIR value: /cache/tflint/plugins volumeMounts: - name: tf-cache mountPath: /cache ``` ### The Built-in Cache Terraform has a built-in provider caching mechanism. Set `plugin_cache_dir` in `.terraformrc`, or export `TF_PLUGIN_CACHE_DIR`. When terraform downloads a provider, it stores a copy in that directory. Next time `terraform init` runs and the same version is requested, it reads from the cache instead of downloading again. ```hcl # /cache/.terraformrc plugin_cache_dir = "/cache/terraform/plugins" disable_checkpoint = true ``` The pod template sets `TF_CLI_CONFIG_FILE` to point to it. No extra pipeline logic needed. ### The First Attempt The PVC was added to the pod template and mounted at `/cache`. The env vars point terraform to the right subdirectory. The `.terraformrc` file and cache directories are created in the first pipeline stage. First run: terraform downloads everything. The provider binaries land on the PVC. Then we checked the second run. Most providers loaded from cache instantly. But the tencentcloud provider kept re-downloading from the internet every time. The binary was sitting on the PVC, but terraform ignored it. --- ### The Problem Without the lock file, terraform queries the registry every run to find the latest compatible version. It downloads fresh copies instead of using the cached ones. The tencentcloud provider was the most noticeable because it is the largest (\~200 MB). The `.terraform.lock.hcl` file is generated by `terraform init` on the first run. It pins the exact provider versions. Our repo only contains the Terraform code – the lock file was never committed. And since the workspace is ephemeral, the lock file was gone after every run. ### The Fix We added a per-project cache for the lock file on the same PVC. The pattern: restore before init, save after init. ```bash LOCK_CACHE_DIR="/cache/terraform/lock-files/${JOB_NAME}" mkdir -p "${LOCK_CACHE_DIR}" # Restore before init if [ -f "${LOCK_CACHE_DIR}/.terraform.lock.hcl" ]; then cp "${LOCK_CACHE_DIR}/.terraform.lock.hcl" .terraform.lock.hcl echo "[CACHE] Lock file restored" fi export TF_PLUGIN_CACHE_DIR=/cache/terraform/plugins export TF_CLI_CONFIG_FILE=/cache/.terraformrc terraform init -input=false # Save after init if [ -f .terraform.lock.hcl ]; then cp .terraform.lock.hcl "${LOCK_CACHE_DIR}/" echo "[CACHE] Lock file saved" fi ``` First run generates the lock file. It gets saved to PVC. On the next run, it is restored before init. With the lock file present, terraform knows exactly which provider versions to use and reads them straight from the cache. Every provider, including tencentcloud. --- ## Gotchas ### Init Container to Pipeline Stage The `.terraformrc` file needs to exist before terraform runs. We used to have an init container in the pod template to create it. Later the setup was moved into the first pipeline stage. The init container was commented out entirely. ```bash if [ ! -f /cache/.terraformrc ]; then echo 'plugin_cache_dir = "/cache/terraform/plugins"' > /cache/.terraformrc echo 'disable_checkpoint = true' >> /cache/.terraformrc fi ``` Keeping it in the pipeline avoids maintaining a separate init container image and keeps everything visible in one stage. ### The tencentcloud Debugging The tencentcloud provider took the longest to debug. The binary was on the PVC. The cache path was correct. But terraform kept downloading it. The lock file was the missing link the whole time, but it was not obvious at first because hashicorp providers (random, time, helm, kubernetes) seemed to cache fine without it. They did not – the lock file fixed those too, we just did not notice because they are smaller downloads. --- ### Some Notes on the PVC The PVC is backed by Tencent Cloud File Storage (CFS) with `ReadWriteMany`. All pods can mount and write to it concurrently. But Terraform's docs say `TF_PLUGIN_CACHE_DIR` is not concurrency-safe for concurrent writers. In practice, this is rarely an issue. Running the same pipeline multiple times at the same time is unusual. And even if it happens, the failure mode is graceful. A partially written cache just means terraform re-downloads from the internet. The next successful run writes a clean copy. ### The Results Here is the init output before caching. Pay attention to the timestamps: ``` 14:48:10 Initializing provider plugins... 14:48:10 - Finding hashicorp/helm versions matching "~> 2.12"... 14:48:10 - Finding gavinbunney/kubectl versions matching ">= 1.14.0"... 14:48:10 - Finding tencentcloudstack/tencentcloud versions matching "> 1.18.1"... 14:48:10 - Finding hashicorp/kubernetes versions matching "~> 2.25"... 14:48:10 - Finding hashicorp/random versions matching ">= 3.0.0"... 14:48:10 - Finding hashicorp/time versions matching ">= 0.9.0"... 14:48:12 - Installing tencentcloudstack/tencentcloud v1.83.3... 14:48:16 - Installed tencentcloudstack/tencentcloud v1.83.3 (signed by a HashiCorp partner, key ID 84F69E1C1BECF459) 14:48:16 - Installing hashicorp/kubernetes v2.38.0... 14:48:17 - Installed hashicorp/kubernetes v2.38.0 (signed by HashiCorp) 14:48:17 - Installing hashicorp/random v3.9.0... 14:48:17 - Installed hashicorp/random v3.9.0 (signed by HashiCorp) 14:48:17 - Installing hashicorp/time v0.14.0... 14:48:18 - Installed hashicorp/time v0.14.0 (signed by HashiCorp) 14:48:18 - Installing hashicorp/helm v2.17.0... 14:48:18 - Installed hashicorp/helm v2.17.0 (signed by HashiCorp) 14:48:19 - Installing gavinbunney/kubectl v1.19.0... 14:48:21 - Installed gavinbunney/kubectl v1.19.0 (self-signed, key ID 1E1CE42504F5FBB2) ``` Start: 14:48:10\. End: 14:48:21\. That is **11 seconds** to download 6 providers. And after caching: ``` 16:59:29 Initializing provider plugins... 16:59:29 - Reusing previous version of hashicorp/random from the dependency lock file 16:59:29 - Reusing previous version of hashicorp/time from the dependency lock file 16:59:29 - Reusing previous version of hashicorp/helm from the dependency lock file 16:59:30 - Reusing previous version of gavinbunney/kubectl from the dependency lock file 16:59:30 - Reusing previous version of tencentcloudstack/tencentcloud from the dependency lock file 16:59:30 - Reusing previous version of hashicorp/kubernetes from the dependency lock file 16:59:31 - Using hashicorp/time v0.14.0 from the shared cache directory 16:59:32 - Using hashicorp/helm v2.17.0 from the shared cache directory 16:59:32 - Using gavinbunney/kubectl v1.19.0 from the shared cache directory 16:59:36 - Using tencentcloudstack/tencentcloud v1.83.8 from the shared cache directory 16:59:36 - Using hashicorp/kubernetes v2.38.0 from the shared cache directory 16:59:37 - Using hashicorp/random v3.9.0 from the shared cache directory ``` Start: 16:59:29\. End: 16:59:37\. That is **8 seconds**. From 11 seconds down to 8 seconds. A **27% reduction** for the provider init phase. ### Migrating SonarQube from AWS to Tencent Cloud URL: https://anantafatur.dev/migrating-sonarqube-from-aws-to-tencent-cloud/ Last updated: 2026-07-12T04:40:27.000Z We were doing a cloud migration from AWS to Tencent Cloud. One of the tasks I got was migrating SonarQube. We use SonarQube for code scanning in our CI pipeline. It runs on Kubernetes with ArgoCD, so it is already GitOps. The same SonarQube image tag on both sides. No version upgrade, just a lift-and-shift to a different cloud. I asked my manager: "Is downtime okay?" He said yes. I needed to know because I had to stop the SonarQube service while migrating the PV data. The migration had five parts: database, PV data, custom plugin, ArgoCD apps, and DNS repointing. Let me walk through each one. --- ### Database migration I started with the database. We used Tencent DTS (Data Transfer Service). The network team already configured the intercloud connection between AWS and Tencent Cloud, so we had private connectivity. I will not talk about that here. That is a whole different story. DTS was surprisingly easy. We used the migration with full sync mode. This means DTS does an initial full migration, then keeps replicating changes continuously. So the source and destination stay in sync. But here is the part that took some thinking. We did a two-phase cutover. First, before the actual cutover, I repointed the old SonarQube (still running on AWS) to use the new Tencent Cloud DB. I updated the `jdbcOverwrite` in the Helm values to point to the new database URL. This made sure all new scan results were written directly to the Tencent Cloud DB while the old SonarQube was still running on AWS. Once that was confirmed working, we could safely stop the DTS incremental sync. No data loss, because all new data was already going to the new DB. The DB schemas were the same on both sides (both PostgreSQL), so no migration issues there. Honestly, this was the smoothest part of the whole thing. ### PV data migration Next was the PV data. This one was straightforward. I created a migration pod in both clusters and mounted the PVCs. Then I used `tar` to pipe the data directly from the old cluster to the new cluster. ```bash kubectl \ --context=arn:aws:eks:ap-southeast-1::cluster/ \ exec -n sonarqube sonarqube-sonarqube-migration -- \ tar -C /mnt/sonarqube -cpf - . \ | \ kubectl \ --context= \ exec -i -n sonarqube sonarqube-sonarqube-migration -- \ tar -C /mnt/sonarqube -xpf - ``` The data was small. Less than 1GB. The pipe across clusters worked without issues. One thing I noticed: the old PV was 50Gi. Way too big for the actual data. This was legacy. Someone probably set it once and never revisited. On Tencent Cloud, the minimum CBS volume is 10Gi, so I set it to 10Gi. Still more than enough. I used tar for the PV migration here. It works, but it requires downtime. If I had to do this again, I would use rsync instead. Rsync can sync incrementally while the service is still running, then do a final quick sync during a brief pause. No downtime needed. --- ### Custom plugin from S3 to COS We had a custom SonarQube plugin stored in S3\. On Tencent Cloud, the object storage is COS. COS is S3-compatible, so we could still use the AWS CLI tool to download it. I just pointed it to the COS endpoint instead. On AWS, the init container looked like this: ```yaml extraInitContainers: - name: download-custom-plugin image: amazon/aws-cli:2.15.0 command: - sh - -c - | set -e echo "Downloading plugin from S3..." aws s3 cp \ s3:///.jar \ /opt/sonarqube/extensions/plugins/.jar env: - name: AWS_REGION value: ap-southeast-1 ``` On Tencent Cloud, it changed to: ```yaml extraInitContainers: - name: download-custom-plugin image: amazon/aws-cli:2.35.9 command: - sh - -c - | aws configure set default.s3.addressing_style virtual aws s3 cp s3:///.jar \ /opt/sonarqube/extensions/plugins/ \ --endpoint-url https://cos.ap-jakarta.myqcloud.com env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: cos-credentials key: secret-id - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: cos-credentials key: secret-key - name: AWS_DEFAULT_REGION value: ap-jakarta ``` Two things I had to figure out. First, COS needs `virtual` addressing style. Without `aws configure set default.s3.addressing_style virtual`, the CLI fails. I got an error on my first attempt and had to look it up. Fun fact: this is a common gotcha with S3-compatible object stores that nobody tells you about upfront. Second, on AWS we used IRSA (IAM Roles for Service Accounts). No explicit credentials in the values. On Tencent Cloud, COS does not have the same IRSA-like integration. Or maybe it does and I was too lazy to set it up. So I used explicit credentials from a Kubernetes secret instead. It works. I did the plugin copy before deploying the ArgoCD app. The init container downloads the plugin on pod startup, so the file needs to already be in COS before the pod tries to pull it. --- ### ArgoCD apps in the new environment With the DB running, PV data transferred, and plugin in COS, I deployed the Helm chart on the new cluster. This was mostly copying it and adjusting things for Tencent Cloud. Storage class changed from `ebs-sc` (EKS) to `cbs` (TKE). Different cluster, different defaults. Nothing complicated. But here is where I made a mistake. During the initial setup on Tencent Cloud, I minimized the resource requests and limits. The cluster was new and I did not want to waste resources since no other workloads were running yet. So I set them too low. After cutover, I realized the SonarQube pods were struggling. The resources were too small. I had to quickly bump them up. Not a disaster, just an annoying thing I forgot to adjust. Looking back, I should have set them correctly from the start. ### DNS repointing All the infrastructure was ready. Database migrated, PV data transferred, plugin in COS, ArgoCD app running. But nobody was hitting the new SonarQube yet. The DNS for `` runs on Cloudflare. The record pointed to the AWS ALB and needed to change to the Tencent CLB. There was one problem. I did not have permission to change the DNS record myself. So my manager changed it for me. --- ### How long did this take? The whole thing took about two weeks from planning to done. Not because it was hard. There were just many small things to remember. The actual hands-on work was maybe a few hours. The rest was waiting. Waiting for DTS sync, waiting for approvals, waiting for testing, and fixing things I forgot. That is it. SonarQube is running on Tencent Cloud now. The pipeline is happy. Hopefully the next migration will be smoother. ### Upgrading Ghost and hitting MariaDB's charset trap URL: https://anantafatur.dev/upgrading-ghost-and-hitting-mariadbs-charset-trap/ Last updated: 2026-07-10T10:07:34.000Z I wanted to upgrade my Ghost blog from 6.26.0 to 6.49.0\. Should be straightforward, right? Change the image tag, restart, watch migrations roll. Here's what I got instead: ``` [2026-07-10 09:36:13] ERROR alter table `gifts` add constraint `gifts_buyer_member_id_foreign` foreign key (`buyer_member_id`) references `members` (`id`) on delete SET NULL - Can't create table `ghost`.`gifts` (errno: 150 "Foreign key constraint is incorrectly formed") ``` Ghost created a backup, tried to run migrations, and then immediately crashed. Then it restarted, created another backup, crashed again. Classic restart loop. ### Environment - Ghost: 6.26.0 to 6.49.0 (Docker) - MariaDB: 10.6.24 (also Docker) - Behind a reverse proxy. - Database connection charset forced to `utf8mb4` via env var (more on that later) ### First thought: what's actually wrong? Errno 150 with "Foreign key constraint is incorrectly formed" usually means one of two things: the column types don't match, or the charsets and collations don't match. The migration was trying to create a new `gifts` table with a foreign key pointing to `members.id`. Both columns should be `varchar(24)`. Seemed simple enough. I checked the `members` table: ```sql CREATE TABLE `members` ( `id` varchar(24) NOT NULL, ... ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ``` `varchar(24)`, `utf8mb4_unicode_ci`. Fine. I checked the MariaDB defaults: ``` DEFAULT_CHARACTER_SET_NAME: utf8mb4 DEFAULT_COLLATION_NAME: utf8mb4_unicode_ci ``` Also fine. So I did the obvious thing and tried creating the table manually: ```sql CREATE TABLE gifts ( id varchar(24) NOT NULL, buyer_member_id varchar(24) NULL, FOREIGN KEY (buyer_member_id) REFERENCES members(id) ON DELETE SET NULL ) ENGINE=InnoDB; ``` It worked. No error. So what was Ghost doing differently? ### Following the breadcrumbs Ghost's migration uses Knex 2.4.2 to create tables. The `addTable` utility calls `commands.createTable`, which generates SQL through Knex's MySQL table compiler. I pulled the source from Ghost's image to see what's happening. The key code in Knex's MySQL table compiler: ```javascript const charset = this.single.charset || conn.charset || ''; const collation = this.single.collate || conn.collate || ''; const engine = this.single.engine || ''; if (charset && !like) sql += ` default character set ${charset}`; if (collation) sql += ` collate ${collation}`; ``` So Knex reads `conn.charset` from the connection settings and adds `DEFAULT CHARACTER SET utf8mb4` to the CREATE TABLE. I checked where that comes from in Ghost's database initialization: ```javascript if (client === 'mysql2') { dbConfig.connection.timezone = 'Z'; dbConfig.connection.charset = 'utf8mb4'; dbConfig.connection.decimalNumbers = true; } ``` The `database__connection__charset` line I had in my compose file was redundant. Ghost hardcodes it anyway when it detects the `mysql2` client. ### The actual problem Here's the catch. MariaDB 10.6's default collation for the `utf8mb4` character set is `utf8mb4_general_ci`, not `utf8mb4_unicode_ci`. These are different. When Knex generated: ```sql CREATE TABLE `gifts` (...) DEFAULT CHARACTER SET utf8mb4 ENGINE = InnoDB ``` MariaDB interpreted that as "use `utf8mb4` charset with its default collation," which is `utf8mb4_general_ci`. But the `members` table was explicitly created with `COLLATE=utf8mb4_unicode_ci` (from the database default that existed before Knex started overriding things). I confirmed this by creating a test table with just the charset and no collation: ```sql CREATE TABLE test_knex_sim (id varchar(24) NOT NULL) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4; ``` Result: ```sql CREATE TABLE `test_knex_sim` (...) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ``` There it is. `utf8mb4_general_ci`, not `utf8mb4_unicode_ci`. MariaDB rejects the foreign key because you can't reference a `utf8mb4_unicode_ci` column from a `utf8mb4_general_ci` column. ### The fix I needed Knex to also emit an explicit `COLLATE` clause in the CREATE TABLE, so the new table would match the existing tables. Knex's table compiler looks at `conn.collate` from the connection settings. Ghost's nconf-based config loader maps env vars to nested config keys, so I added this to the compose file: ```yaml database__connection__collate: utf8mb4_unicode_ci ``` This makes Knex generate: ```sql CREATE TABLE `gifts` (...) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB ``` Which finally matches what the `members` table has. But there was one more problem. The `email_design_settings` table (created during a previous upgrade) also had the wrong collation. Only one table in the whole database: ``` TABLE_NAME TABLE_COLLATION email_design_settings utf8mb4_general_ci ``` So I converted it too: ```sql ALTER TABLE email_design_settings CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ``` After restarting, Ghost ran all the migrations from 6.27 through 6.49 successfully. The latest entry in the migrations table clocks in at version 6.48. ### The warning that bugs me There's a benign warning that shows up on every database query now: ``` Ignoring invalid configuration option passed to Connection: collate. This is currently a warning, but in future versions of MySQL2, an error will be thrown if you pass an invalid configuration option to a Connection ``` The `collate` option is used by Knex's table compiler but mysql2's connection doesn't recognize it. It's harmless for now and Knex still reads it correctly from `connectionSettings`, but a future mysql2 version might reject the connection entirely. I'll probably need to revisit this if I upgrade the npm packages. Honestly, the cleaner fix would be to patch this at the Ghost config level, but I'm not about to fork Ghost's container image for a collation issue. So this tech debt lives on my compose file until either Knex or Ghost adds official collation support in their database config. Hopefully not me who fixes this later. ### What I learned The root cause was a mismatch between how MariaDB resolves default collations and how Knex generates CREATE TABLE statements. When you specify `DEFAULT CHARACTER SET utf8mb4` without an explicit collation, MariaDB uses the charset's hardcoded default collation (`utf8mb4_general_ci`), not whatever your server or database defaults are. This is a subtle trap because the database default collation is `utf8mb4_unicode_ci`, so most operations behave as if that's the system-wide setting. But Knex's explicit charset override short-circuits that inheritance. If I were designing this from scratch, I'd either never set a connection-level charset in the first place (let the database defaults flow through), or always pair it with an explicit collation. Ghost does the first half (forces charset) but misses the second half. ### Colophon This whole thing, from root cause analysis to fix to this write-up, was done by Zed's AI agent using the deepseek-v4-flash model. I just described the problem and reviewed the results. ![](https://anantafatur.dev/content/images/2026/07/zed-1.png) The agent dug through Knex source code inside the Docker image, queried the running MariaDB, ran test queries, traced Ghost's config loader, and wrote this post. Pretty wild. ### Cleaning Up My Instagram Following List with Python URL: https://anantafatur.dev/cleaning-up-my-instagram-following-list-with-python/ Last updated: 2026-07-12T10:02:10.000Z Over the weekend I noticed my Instagram following count was close to 1000\. I opened the app and scrolled through my following list. Honestly, I had no idea who half of those people were. Old classmates, random accounts I followed years ago, bots. It was a mess. I figured it was time to clean up. But manually checking each account is painful when you are close to the 1000 mark. So I looked for a faster way. Here is how I did it. --- ### Step 1: Export your Instagram data Meta lets you download your Instagram data. It is an official feature, so no third-party tools or shady websites involved. You can access it from your account settings, or use this direct link: Click "Download or transfer information," then "Download your information." Select Instagram, choose "Some of your information," and pick "Followers and following" under the Connections section. Uncheck everything else to keep the export small. For the date range, I selected "All time." For the format, I chose JSON. JSON is easy to parse with Python. ![](https://anantafatur.dev/content/images/2026/07/image.png) Confirm your export Start the export. It will take a few minutes depending on how many followers and following you have. In my case, it took about 5 minutes. I got two email notifications: one saying the download request was in progress, and another saying the download was ready. ### Step 2: Download and extract the zip Once the export is ready, download the zip file from Meta and extract it. You will find the data files you need inside. If you have fewer than 10,000 followers and following, you will see `followers.json` and `following.json`. If you are over that limit, Meta splits the data into multiple files. In my case I got `followers_1.json` and `followers_2.json`. I just merged them before running the script. ### Step 3: The Python script I wrote a quick Python script to compare the two sets and find accounts I follow that do not follow me back. Here is the code: ```python import json with open("followers.json", encoding="utf-8") as f: followers = json.load(f) with open("following.json", encoding="utf-8") as f: following = json.load(f) followers_set = { item["string_list_data"][0]["href"].rstrip("/").split("/")[-1] for item in followers if item.get("string_list_data") } following_set = { item["string_list_data"][0]["href"].rstrip("/").split("/")[-1] for item in following["relationships_following"] if item.get("string_list_data") } not_following_back = sorted(following_set - followers_set) with open("not_following_back.txt", "w", encoding="utf-8") as f: for username in not_following_back: f.write(f"https://www.instagram.com/{username}\n") print(f"Saved {len(not_following_back)} accounts to not_following_back.txt") ``` The logic is simple. I load the followers and following data, extract the username from each entry, put them into sets, and compute the difference. Accounts in my following set but not in my followers set are the ones not following me back. One thing to note: the structure for following and followers JSON is slightly different. Followers is a flat list, while following is wrapped inside a `relationships_following` key. The script handles that. ### Step 4: Review the output The script writes a list of Instagram profile URLs into `not_following_back.txt`. ```text https://www.instagram.com/username_1 https://www.instagram.com/username_2 ... ``` In my case, I had about 350 accounts that were not following me back. Some of them were media and brand accounts that I still want to follow. That is fine. I was mostly looking for personal accounts that do not follow me back so I could clean them out. I opened the file, visited the profiles I was unsure about, and manually unfollowed the ones I did not need anymore. --- ### Closing thoughts This approach saved me a lot of time compared to manually scrolling through my following list. The Instagram data export is a solid feature. It gives you clean JSON files that you can process however you want. One thing I might explore next is automating the unfollow step using Instagram's API or something like Playwright. But for now, manual review was good enough. I only had 350 accounts to sort through, and most of them I recognized immediately. ### Expanding EC2 Root EBS Volume Without Downtime URL: https://anantafatur.dev/expanding-ec2-root-ebs-volume-without-downtime/ Last updated: 2026-05-24T05:38:48.000Z Ubuntu 24.04.4 LTS (ARM), root volume on EBS, managed via Terraform. Disk usage was already near full: ```bash root@anantafatur:~/anantafatur-tfworkload# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 23G 22G 1.2G 95% / ``` Current disk layout: ```bash root@anantafatur:~/anantafatur-tfworkload# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.2M 1 loop /snap/amazon-ssm-agent/13008 loop1 7:1 0 69M 1 loop /snap/core22/2412 loop2 7:2 0 42.6M 1 loop /snap/snapd/26869 nvme0n1 259:0 0 24G 0 disk ├─nvme0n1p1 259:1 0 23G 0 part / ├─nvme0n1p15 259:2 0 99M 0 part /boot/efi └─nvme0n1p16 259:3 0 923M 0 part /boot ``` Updated only the root volume size in Terraform: ```hcl root_block_device { volume_size = 26 } ``` Terraform showed in-place modification: ```bash root@anantafatur:~/anantafatur-tfworkload# terraform plan Terraform will perform the following actions: # aws_instance.main will be updated in-place ~ resource "aws_instance" "main" { ~ root_block_device { ~ volume_size = 24 -> 26 } } Plan: 0 to add, 1 to change, 0 to destroy. ``` Applied: ```bash root@anantafatur:~/anantafatur-tfworkload# terraform apply ``` After apply, EBS volume size increased immediately, but partition size was still unchanged: ```bash root@anantafatur:~/anantafatur-tfworkload# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.2M 1 loop /snap/amazon-ssm-agent/13008 loop1 7:1 0 69M 1 loop /snap/core22/2412 loop2 7:2 0 42.6M 1 loop /snap/snapd/26869 nvme0n1 259:0 0 26G 0 disk ├─nvme0n1p1 259:1 0 23G 0 part / ├─nvme0n1p15 259:2 0 99M 0 part /boot/efi └─nvme0n1p16 259:3 0 923M 0 part /boot ``` Expanded partition online: ```bash root@anantafatur:~/anantafatur-tfworkload# sudo growpart /dev/nvme0n1 1 CHANGED: partition=1 start=2099200 old: size=48232415 end=50331614 new: size=52426719 end=54525918 ``` Then resized ext4 filesystem: ```bash root@anantafatur:~/anantafatur-tfworkload# sudo resize2fs /dev/nvme0n1p1 resize2fs 1.47.0 (5-Feb-2023) Filesystem at /dev/nvme0n1p1 is mounted on /; on-line resizing required old_desc_blocks = 3, new_desc_blocks = 4 The filesystem on /dev/nvme0n1p1 is now 6553339 (4k) blocks long. ``` Final result: ```bash root@anantafatur:~/anantafatur-tfworkload# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 25G 22G 3.1G 88% / ``` ```bash root@anantafatur:~/anantafatur-tfworkload# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.2M 1 loop /snap/amazon-ssm-agent/13008 loop1 7:1 0 69M 1 loop /snap/core22/2412 loop2 7:2 0 42.6M 1 loop /snap/snapd/26869 nvme0n1 259:0 0 26G 0 disk ├─nvme0n1p1 259:1 0 25G 0 part / ├─nvme0n1p15 259:2 0 99M 0 part /boot/efi └─nvme0n1p16 259:3 0 923M 0 part /boot ``` Notes: - AWS volume state was still `optimizing` while resizing filesystem. No issue observed. - Nitro instances expose EBS as NVMe (`/dev/nvme0n1`). - `growpart` came from `cloud-guest-utils`. - No reboot needed during the process. ### Cleaning Up a 600k+ Object Versioned S3 Bucket Using Lifecycle Policies URL: https://anantafatur.dev/cleaning-up-a-600k-object-versioned-s3-bucket-using-lifecycle-policies/ Last updated: 2026-05-17T11:43:48.000Z I ran into this recently while cleaning up a temporary bucket that had accumulated a large number of objects over time. The bucket also had versioning enabled, which changes the deletion behavior quite a bit. At first I used the usual command: `aws s3 rm s3://my-bucket --recursive.` Technically, it worked. Practically, it was deleting objects one by one for a very long time. You can literally watch the CLI spam delete operations continuously. For small buckets this is fine. For buckets with many object versions and delete markers, it becomes annoying very quickly. The important detail here is versioning. With versioning enabled, deleting an object does not necessarily remove the underlying data immediately. S3 creates delete markers, old versions remain, and the bucket is still considered non-empty. So even after the recursive delete finishes, you may still fail when attempting to remove the bucket itself. I remembered reading an older approach a few years ago: instead of trying to aggressively delete everything from the client side, let S3 clean itself up using lifecycle rules. So instead of fighting the bucket manually, I attached a lifecycle configuration that aggressively expires: - current objects - non-current versions - delete markers - incomplete multipart uploads This was the lifecycle configuration I used: ```json { "Rules": [ { "ID": "DeleteAllObjects", "Status": "Enabled", "Filter": {}, "Expiration": { "Days": 1, "ExpiredObjectDeleteMarker": true }, "NoncurrentVersionExpiration": { "NoncurrentDays": 1, "NewerNoncurrentVersions": 0 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] } ``` Then applied it using: ```bash aws s3api put-bucket-lifecycle-configuration \ --bucket my-bucket \ --lifecycle-configuration file://lifecycle.json ``` And verified it: ```bash aws s3api get-bucket-lifecycle-configuration \ --bucket my-bucket ``` After that, there was nothing else to do except wait. One thing worth noting: lifecycle execution is not immediate. Even if the rule says `Days: 1`, S3 does not guarantee deletion exactly after 24 hours. In my case, I just left it for a few days. Around day 3, the bucket finally became empty enough to delete cleanly. ![](https://anantafatur.dev/content/images/2026/05/image-2.png) ![](https://anantafatur.dev/content/images/2026/05/image-3.png) The nice part of this approach is that S3 handles deletion internally instead of your local machine issuing massive delete requests continuously. No long-running terminal session, no retry loops, no watching object counters decrease one by one. If the requirement is immediate deletion, this is probably not the right approach. But for old temporary buckets where time is less important than operational simplicity, lifecycle policies are significantly less painful than hammering the bucket with recursive delete commands for hours. Simple enough. Sometimes the best cleanup job is the one you let AWS do for you. ### Rundeck on Kubernetes with RDS and S3 Backend URL: https://anantafatur.dev/rundeck-on-kubernetes-with-rds-and-s3-backend/ Last updated: 2026-04-29T01:07:34.000Z ### Context and Goal In one staging environment we needed a simple way to execute operational tasks across several machines without SSH-ing into each node manually. Typical tasks included restarting services, running maintenance scripts, or triggering operational workflows. The goal was not to build a full CI/CD platform, just to have a central place where engineers could execute tasks across infrastructure in a controlled way. For this, we deployed [Rundeck](https://www.rundeck.com/). Rundeck fits this use case because the model is simple: define nodes, define jobs, and execute from a UI. There’s no pipeline abstraction or heavy integration layer. It works well when the problem is “run this across these machines” rather than “build a delivery system.” --- ### Environment and Constraints The environment runs on Kubernetes (EKS), and an important constraint is that the cluster uses spot instances. That means pods can disappear at any time when underlying nodes are reclaimed. Because of that, we avoided storing anything important inside the cluster. Rundeck is deployed as a stateless container (`rundeck/rundeck:5.16.0`), while all persistent data, projects, jobs, execution history, and configuration lives in PostgreSQL on RDS. Node inventory is handled separately. --- ### Database Setup PostgreSQL is used as backend storage. ```sql create database rundeck; create user rundeckuser with password 'rundeckpassword'; grant all privileges on database rundeck to rundeckuser; \c rundeck grant all privileges on schema public to rundeckuser; ``` For more details, please refer to [Rundeck Documentation](https://docs.rundeck.com/docs/administration/configuration/database/postgres.html#setup-rundeck-database) --- #### Architecture Rundeck runs inside Kubernetes and handles UI and execution logic. PostgreSQL stores all state including jobs, configuration, and execution history. Node definitions are external, and jobs typically execute against a reachable machine (often a bastion-style EC2 instance) that can access internal infrastructure. No persistent volume is attached to the Rundeck pod, which keeps it stateless but also exposed an issue early on. ![](https://anantafatur.dev/content/images/2026/04/image-1.png) Rundeck - Simple Architecture EKS in this architecture means the bastion host has access to the internal cluster and is used to run operational tasks. In our case, it’s used to scale down applications every night. ![](https://anantafatur.dev/content/images/2026/04/image-2.png) Rundeck - Job History --- #### Node Source Issue (and Fix) Initially, node definitions were stored locally inside the pod as `resources.json`. This worked until pods restarted. Because the cluster runs on spot instances, node termination caused pod rescheduling, and the local filesystem disappeared. As a result, Rundeck suddenly showed no nodes in the UI and jobs couldn’t run. ![](https://anantafatur.dev/content/images/2026/04/image-3.png) Rundeck - Nodes This wasn’t a Rundeck issue; it was a bad assumption about persistence. Instead of attaching a persistent volume just to keep a single JSON file, node definitions were moved to S3\. The structure didn’t change: ```json { "ec2-rundeck-client": { "nodename": "ec2-rundeck-client", "hostname": "xxx", "osVersion": "ubuntu-noble-24.04-arm64-server-20251001", "osFamily": "unix", "osArch": "aarch64", "ssh-key-storage-path": "keys/project/staging/ssh-privatekey", "description": "ec2-rundeck-client", "osName": "Linux", "tags": "ec2", "username": "ubuntu" } } ``` The file is uploaded to: ``` s3://s3-poc-rundeck/resources.json ``` Then configured in Rundeck: - Project Settings → Edit Nodes - Add Source → AWS S3 Model Source - Access Key / Secret Key → IAM credentials - Region → ap-southeast-1 - Bucket → s3-poc-rundeck - File → resources.json - Writeable → Yes Setting it as writeable allows Rundeck to update the node file directly from the UI. After moving to S3, the node disappearance issue stopped completely. --- #### S3 Configuration Gotcha One small issue: the `ap-southeast-3` AWS region wasn’t available in the Rundeck S3 dropdown. The workaround was to create the bucket in a nearby region (`ap-southeast-1`). Since the file is very small, cross-region latency is negligible. Not ideal, but acceptable. IAM permissions used: ```json { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::s3-poc-rundeck", "arn:aws:s3:::s3-poc-rundeck/*" ] } ``` --- #### Notifications Two mechanisms are currently used: SMTP for email alerts and a webhook integration handled by a small Node.js service. Rundeck sends execution events to the webhook, which can then process or forward them as needed. The webhook source code is available here: . ![](https://anantafatur.dev/content/images/2026/04/image-4.png) Rundeck - Job Notifications This implementation is simple but flexible enough to extend for additional notification channels or custom logic. ### WebSocket-Based Jenkins Agents in a Reverse Proxy Environment URL: https://anantafatur.dev/websocket-based-jenkins-agents-in-a-reverse-proxy-environment/ Last updated: 2026-04-27T15:27:45.000Z We run Jenkins in a fairly standard but constrained setup: Jenkins controller sits on a CVM (cloud virtual machine) inside a Docker container, exposed through a CLB (HTTP/HTTPS layer 7). Kubernetes is used for dynamic build agents via the Jenkins Kubernetes plugin. At a high level, the flow should be simple: Jenkins schedules a job, spins up a pod in Kubernetes, and the pod connects back to the controller as a JNLP agent. In practice, the networking layer in front of Jenkins made this more fragile than expected. --- ### Environment and constraints The controller is not directly exposed. Traffic goes through a CLB and then to Nginx inside the VM, which proxies into the Jenkins container. We already configured Jenkins with the public domain (`jenkins.company.com`) so agents are supposed to connect using that endpoint instead of internal IPs. Kubernetes agents are configured using the standard JNLP launcher. Initially, everything looked fine during provisioning. Pods were created successfully, and logs showed discovery and handshake steps. --- ### The actual issue When the agent pod started, the remoting logs showed something like this: ``` INFO: Agent discovery successful Agent address: jenkins.company.com Agent port: 50000 INFO: Handshaking INFO: Server reports protocol JNLP4-connect-proxy not supported, skipping INFO: Trying protocol: JNLP4-connect INFO: Connecting to jenkins.company.com:50000 ``` The key problem here is subtle but important: even though we had a domain setup and everything behind L7, the agent still attempted to connect via port `50000` using the classic JNLP TCP channel. That worked in older setups where we exposed port 50000 directly. In this environment, that port is not properly reachable through CLB/L7 routing. So agents were effectively trying to open a raw TCP connection through a path that does not exist anymore. --- ### Why this breaks Jenkins remoting has multiple transport modes: - Classic JNLP TCP (port 50000) - WebSocket over HTTP(S) Our infrastructure only reliably supports HTTP(S) traffic through CLB + Nginx. So anything relying on direct TCP (like 50000) becomes unreliable or completely blocked. Also, the log line: `JNLP4-connect-proxy not supported, skipping` indicates that the agent is trying to negotiate a protocol that depends on proxy-aware TCP forwarding, which our L7 setup does not support. --- ### Decision: switch to WebSocket agent transport Instead of fighting TCP routing, we moved the agent communication to WebSocket mode. ![](https://anantafatur.dev/content/images/2026/04/image.png) Jenkins - Clouds Configuration This changes the model completely: instead of Jenkins opening a raw TCP channel, the agent connects back over HTTP(S) and upgrades the connection to WebSocket. This fits cleanly into our existing CLB + Nginx reverse proxy setup. Reference behavior comes from Jenkins WebSocket remoting support: --- ### Nginx configuration change To support WebSocket upgrade, we adjusted the reverse proxy layer: ```nginx location / { proxy_pass http://10.51.3.208:8080; #Jenkins Service proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } ``` The important part is not the basic proxying, it’s the upgrade headers. Without those, WebSocket silently falls back or fails during handshake. --- ### Jenkins agent configuration change On the Kubernetes agent side, we explicitly switched the launcher to WebSocket mode instead of JNLP TCP. Once that was applied, the behavior changed immediately. Logs after fix: ``` INFO: Using Remoting version: 3355.v388858a_47b_33 INFO: Using /home/jenkins/agent/remoting as a remoting work directory INFO: WebSocket connection open INFO: Connected ``` No reference to port 50000 anymore. No TCP handshake attempts. Just HTTP upgrade and connection success. --- ### Result and cleanup Once WebSocket mode was stable, we were able to safely remove dependency on port 50000 entirely: - Closed port 50000 in Jenkins container configuration - Removed exposure in internal networking rules - Reduced attack surface (no raw JNLP port exposed anymore) This also simplified the infra a bit because everything is now strictly HTTP(S), based through CLB. ### Misreading Terraform Plan Led to Instance Destruction URL: https://anantafatur.dev/misreading-terraform-plan-led-to-instance-destruction/ Last updated: 2026-04-21T17:12:44.000Z This incident happened on Tencent Cloud and involved a CVM instance that I originally created manually for a PoC. Over time, that instance was reused for production without being rebuilt properly under Terraform. I later brought it into Terraform management, but I did not enable deletion protection, which became an important gap. The problem started from a small configuration change. I updated `associated_ip_public` from `true` to `false`, assuming it would be applied in place. In reality, this attribute forces resource replacement. Terraform already showed this clearly in the plan output with `destroy and then create replacement` and a note indicating that the change forces replacement. I saw the plan, but I did not fully process what it meant. I focused too much on the summary and not enough on the actual behavior of the resource. When I ran `terraform apply`, the existing CVM instance was destroyed and replaced with a new one. Since the instance was directly exposed using an Elastic IP and not behind any load balancer, the service immediately became unavailable. The downtime lasted around 15 minutes. There was no data loss because the system was still in early production use, but the situation could have been much worse. To recover, I recreated the system disk from the latest snapshot, converted it into a custom image, and launched a new CVM instance from that image. I then reallocated an Elastic IP using the same public IP address, which allowed me to avoid any DNS changes because the A record was already pointing to that IP. Looking back, the root cause was not Terraform itself but how I handled the plan. I assumed the change was safe, while Terraform clearly indicated a full replacement. I also did not have deletion protection enabled, so there was nothing preventing the destroy action from going through. After this incident, I enabled deletion protection on production resources and increased snapshot frequency to reduce recovery gaps. I also became much stricter in reviewing `terraform plan`, especially when I see indicators that a resource will be replaced. For production changes, I strongly recommend having a second person review the plan together, since it is easy to miss critical details when reviewing alone. ### Remove “et al.” in IEEE Zotero Style (Show All Authors) URL: https://anantafatur.dev/remove-et-al-in-ieee-zotero-style-show-all-authors/ Last updated: 2026-04-19T09:30:18.000Z I got a revision from my committee about reference style. I’m using Zotero with the default IEEE citation style configuration. By default, if authors are more than 7, it prints the first author followed by “et al.”. The committee requires avoiding “et al.” and showing all authors. Here’s a simple trick. No programming needed. It’s just a small config change. ### Example (Default Output) \[1\] A. Verbitski et al., “Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases,” in Proceedings of the 2017 ACM International Conference on Management of Data, Chicago Illinois USA: ACM, May 2017, pp. 1041–1052\. doi: 10.1145/3035918.3056101. ### Steps On Zotero, click Edit → Settings → Cite → IEEE reference guide → Style Editor. In the Style Editor, change `et-al-min="7"` to `et-al-min="100"`. There are 2 occurrences of `et-al-min`, usually around line 125 and 143\. Use Ctrl + F to find them. ### Style Configuration Update the metadata to avoid overwriting the default style. ```xml IEEE Reference Guide version 11.29.2023 Institute of Electrical and Electronics Engineers http://www.zotero.org/styles/ieee ``` Change it to: ```xml IEEE Full Author IEEE Full Author IEEE Full Author ``` ### Apply the Style Save as a new file, for example IEEE Full Author.csl. Then go back to Zotero, click the plus button, import the file, and select it as your citation style. ### Example (After Fix) \[1\] A. Verbitski, A. Gupta, D. Saha, M. Brahmadesam, K. Gupta, R. Mittal, S. Krishnamurthy, S. Maurice, T. Kharatishvili, and X. Bao, “Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases,” in Proceedings of the 2017 ACM International Conference on Management of Data, Chicago Illinois USA: ACM, May 2017, pp. 1041–1052\. doi: 10.1145/3035918.3056101. ### Understanding Non-Suicidal Self-Injury and Informal Support Approaches URL: https://anantafatur.dev/understanding-non-suicidal-self-injury-and-informal-support-approaches/ Last updated: 2026-04-17T11:55:44.000Z ## Background I wrote this because someone close to me shows these behaviors, specifically skin cutting and skin pinning. I realized I didn’t fully understand it, so I started exploring the topic more seriously. The goal is simple. When they talk about it, I don’t want to misinterpret or react the wrong way. I want to understand what they mean so the conversation feels more aligned and not disconnected. --- ## Definition At a high level, self-harm behaviors are commonly split into two categories, non-suicidal self-injury (NSSI) and suicidal behavior (SB) \[7\]. In this write-up, the focus is on NSSI. Self-harm, or non-suicidal self-injury, refers to deliberate and intentional injury to body tissue in the absence of suicidal intent. It is defined as the act of injuring oneself deliberately without the aim of ending one’s life, usually causing mild to moderate physical damage that is considered socially unacceptable. Common examples include burning, hitting oneself, or cutting the skin with a sharp object \[6\]. In many cases, self-harm is done as a way to shift from emotional pain to physical pain, which can feel more manageable or provide temporary relief \[1\]. Self-injury often begins during adolescence, usually between the ages of 12 and 14\. Several studies show that the prevalence peaks around age 15 and then decreases in early adulthood, although some findings suggest a later onset between ages 17 and 20, which means the pattern can vary \[2\]. Adolescence itself refers to the transitional stage between ages 10 and 19 \[2\]. It is a sensitive and vulnerable period where a person learns how to internalize and express emotions. When unhealthy coping mechanisms are learned or used \[3\], a wide range of problematic behaviors can appear. In short, self-harm is often a coping mechanism for teenagers who experience psychosocial pressure, such as family conflict \[4\] and academic pressure \[5\]. --- ## The Psychological Meaning of Skin The skin is a fairly simple organ from a physiological perspective, but socially and psychologically it is highly complex. People are often judged based on their skin, and even concepts of beauty are closely linked to it. The skin also acts like a visible surface where emotional states are displayed. Emotions such as rage, shame, or fear can show through physical changes like flushing, blushing, or becoming pale. At the same time, the skin functions as a boundary between the outer world and the inner self. It separates the environment from the individual. Even though people can sometimes feel emotionally merged with others in certain situations, in everyday life we still experience ourselves as being contained within our own skin. What is inside the skin is considered the self, and what is outside is not. Because of this symbolic importance, it is not surprising that the skin becomes a common site for self-injury. It is most often cut or burned, but it can also be scratched, slapped, pinned with sharp objects, or deliberately infected. In this way, self-injury can be understood as making internal emotional distress visible through the body \[8\]. --- ## Helping a Teenager Who Self-Injures (NSSI) When helping a teenager who self-injures as a non-medical supporter, the first step is to create a safe and non-judgmental space where open conversation is possible. This means being able to talk about the behavior without reacting with shock, while still gently checking whether there are any thoughts related to suicide. It is important to understand that self-injury often serves a purpose, such as reducing overwhelming emotions or managing social stress. Because of that, instead of reacting immediately, it is more helpful to focus on what the behavior might be communicating. Involving the family is also important, since family support, warmth, and good communication can act as strong protective factors and help create a more stable environment. A practical step is to work together on a simple crisis plan. This can include identifying early warning signs, listing coping strategies or distractions, identifying people they can reach out to, and limiting access to objects that may be used for self-harm. Encouraging professional help is also necessary, especially through evidence-based approaches such as Dialectical Behavior Therapy for Adolescents, which focuses on building emotion regulation skills. It is also important to understand that no medication has been proven to directly treat self-injury, so the focus remains on emotional and relational support. In situations where there is acute distress or very high stress, a brief hospital stay may be needed to ensure safety while the individual learns to manage their distress in a more controlled environment \[9\]. --- ## References \[1\] Y. Y. Zeng, S. Saeed, and S. H. Hu, “Non-Suicidal self-injury: Pain addiction mechanisms, neurophysiological signatures, and therapeutic advances,” J. Clin. Med. Res., vol. 17, no. 10, pp. 537–549, Oct. 2025, doi: [10.14740/jocmr6332](https://doi.org/10.14740/jocmr6332). \[2\] L. Tilton-Weaver, D. Latina, and S. K. Marshall, “Trajectories of nonsuicidal self-injury during adolescence,” J. Adolesc., vol. 95, no. 3, pp. 437–453, Apr. 2023, doi: [10.1002/jad.12126](https://doi.org/10.1002/jad.12126). \[3\] Y. Wu, Y. Zhang, C. Wang, and B. Huang, “A meta-analysis on the lifetime and period prevalence of self-injury among adolescents with depression,” Front. Public Health, vol. 12, p. 1434958, Jul. 2024, doi: [10.3389/fpubh.2024.1434958](https://doi.org/10.3389/fpubh.2024.1434958). \[4\] S. E. Victor, A. E. Hipwell, S. D. Stepp, and L. N. Scott, “Parent and peer relationships as longitudinal predictors of adolescent non-suicidal self-injury onset,” Child Adolesc. Psychiatry Ment. Health, vol. 13, no. 1, p. 1, Dec. 2019, doi: [10.1186/s13034-018-0261-0](https://doi.org/10.1186/s13034-018-0261-0). \[5\] A. Steinhoff, L. Bechtiger, D. Ribeaud, M. Eisner, and L. Shanahan, “Stressful life events in different social contexts are associated with self-injury from early adolescence to early adulthood,” Front. Psychiatry, vol. 11, p. 487200, Oct. 2020, doi: [10.3389/fpsyt.2020.487200](https://doi.org/10.3389/fpsyt.2020.487200). \[6\] S. Kim, S. Woo, and J.-S. Lee, “Investigation of the subtypes of nonsuicidal self-injury based on the forms of self-harm behavior: Examining validity and utility via latent class analysis and ecological momentary assessment,” J. Korean Med. Sci., vol. 38, no. 17, p. e132, 2023, doi: [10.3346/jkms.2023.38.e132](https://doi.org/10.3346/jkms.2023.38.e132). \[7\] A. Poudel, A. Lamichhane, K. R. Magar, and G. P. Khanal, “Non-suicidal self-injury and suicidal behavior among adolescents: Co-occurrence and associated risk factors,” BMC Psychiatry, vol. 22, no. 1, p. 96, Dec. 2022, doi: [10.1186/s12888-022-03763-z](https://doi.org/10.1186/s12888-022-03763-z). \[8\] A. R. Favazza and R. J. Rosenthal, “Varieties of pathological self-mutilation,” Behav. Neurol., vol. 3, no. 2, pp. 77–85, 1990, doi: [10.1155/1990/572716](https://doi.org/10.1155/1990/572716). \[9\] M. Apicella et al., “Non-suicidal self-injury in adolescents: A clinician’s guide to understanding the phenomenon, diagnostic challenges, and evidence-based treatments,” Front. Psychiatry, vol. 16, p. 1605508, Jul. 2025, doi: [10.3389/fpsyt.2025.1605508](https://doi.org/10.3389/fpsyt.2025.1605508). ### Jenkins on Docker (VM) with Kubernetes Agents URL: https://anantafatur.dev/jenkins-on-docker-vm-with-kubernetes-agents/ Last updated: 2026-04-13T10:10:59.000Z The main constraint is that Jenkins must run on a VM. Moving it to a fully Kubernetes-managed setup was not an option, so the compromise was to keep the controller on VM and offload workloads to Kubernetes agents. The Kubernetes cluster in this setup is managed (TKE), which means there is no direct access to nodes. Everything is done through the Kubernetes API, so authentication and RBAC become the critical parts of the integration. --- # Jenkins Server Setup (VM-Based) I started from a clean setup using the official `jenkins/jenkins:2.546` image. I didn’t build a custom image because I wanted to keep the setup minimal and avoid introducing unnecessary variables early on. Instead of running containers manually, I used Docker Compose to make the setup easier to manage. ```yaml version: "3.8" services: jenkins: image: jenkins/jenkins:2.546 container_name: jenkins restart: always ports: - "8080:8080" - "50000:50000" environment: - JAVA_OPTS=-Xms1024m -Xmx1024m - TZ=Asia/Jakarta volumes: - /root/data/jenkins/home:/var/jenkins_home - /root/data/jenkins/logs:/var/log/jenkins - /var/run/docker.sock:/var/run/docker.sock - /usr/bin/docker:/usr/bin/docker - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro ``` The important detail here is persistence. Jenkins stores everything in `/var/jenkins_home`, so that directory is mapped to the host. Without this, all configuration would be lost on restart. Mounting the Docker socket allows Jenkins jobs to run Docker commands directly. This is convenient but gives the container high privileges on the host, so it’s a trade-off I accepted for this setup. Timezone mismatch was another small but annoying issue. By default, Jenkins runs in UTC, which makes logs confusing. Setting `TZ` and mounting system time files fixes that. After the container is up, ports 8080 and 50000 need to be accessible. If port 50000 is blocked, Kubernetes agents will fail to connect later, and the error is not immediately obvious. --- ## Backup Strategy Since Jenkins state is stored on disk, I added snapshot-based backups using Tencent Cloud CBS. ```hcl resource "tencentcloud_cbs_snapshot_policy" "jenkins_server" { snapshot_policy_name = "snapshot-policy-jenkins-server" repeat_weekdays = [0,1,2,3,4,5,6] repeat_hours = [2] retention_days = 3 } resource "tencentcloud_cbs_snapshot_policy_attachment" "jenkins_server" { storage_id = "disk-xxxxxxxx" snapshot_policy_id = tencentcloud_cbs_snapshot_policy.jenkins_server.id } ``` This gives a simple daily backup with short retention, enough for recovery from most operational mistakes. --- # Kubernetes as Jenkins Agent Before configuring Jenkins, I prepared the Kubernetes side first. I verified cluster connectivity with `kubectl get nodes`. All nodes must be in `Ready` state. If not, agents will fail to schedule later. --- ## Namespace, Service Accounts, and RBAC I created a dedicated namespace to isolate Jenkins workloads, along with two service accounts: one for Jenkins and one for Terraform jobs. ```yaml apiVersion: v1 kind: Namespace metadata: name: jenkins-agent --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: jenkins-agent --- apiVersion: v1 kind: ServiceAccount metadata: name: terraform namespace: jenkins-agent ``` Then I defined RBAC permissions so Jenkins can create and manage pods: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jenkins-agent-role namespace: jenkins-agent rules: - apiGroups: [""] resources: ["pods", "pods/log", "pods/exec", "secrets", "services"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: ["batch"] resources: ["jobs"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "create", "delete"] ``` And bound it to the Jenkins service account: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins-agent-rolebinding namespace: jenkins-agent subjects: - kind: ServiceAccount name: jenkins roleRef: kind: Role name: jenkins-agent-role apiGroup: rbac.authorization.k8s.io ``` Terraform gets broader permissions because it modifies resources. After that, I applied everything: `kubectl apply -f manifest.yaml` --- ## Private Registry Secret To allow pulling images from a private registry: ```bash kubectl create secret docker-registry dockerhub \ --docker-server=YOUR_REGISTRY \ --docker-username=USERNAME \ --docker-password='PASSWORD' \ -n jenkins-agent ``` Without this, pods will fail with `ImagePullBackOff`. --- # Adding Kubernetes Credentials into Jenkins This is the step that ties everything together. Jenkins needs credentials to authenticate to the Kubernetes API. Instead of using kubeconfig, I used a service account token. First, I retrieved the token: ```bash kubectl get secret jenkins-sa-token \ -n jenkins-agent \ -o jsonpath='{.data.token}' | base64 -d ``` Then I retrieved the CA certificate: ```bash kubectl get secret terraform-sa-token \ -n jenkins-agent \ -o jsonpath='{.data.ca\.crt}' | base64 -d ``` With these two values, I went into Jenkins UI. From the dashboard, I navigated to: ``` Manage Jenkins → Manage Credentials ``` I added a new credential with the following configuration: - Kind: **Secret Text** - Secret: (paste the service account token) - ID: `jenkins-sa-token` - Description: Kubernetes service account token One thing to note is that Jenkins doesn’t validate this at creation time. If the token is wrong, you only find out later when the Kubernetes cloud fails to connect. --- # Configuring Kubernetes Cloud in Jenkins After adding the credentials, I configured the Kubernetes cloud. Navigate to: `Manage Jenkins → Manage Nodes and Clouds → Configure Clouds` Then add a new Kubernetes cloud configuration. The key fields are: - Kubernetes URL → cluster endpoint (usually from load balancer) - Credentials → the `jenkins-sa-token` created earlier - Kubernetes Namespace → `jenkins-agent` - Kubernetes server certificate key → paste the CA certificate After saving, I used the “Test Connection” button. If everything is correct, Jenkins reports a successful connection to the cluster. In my case, it showed something like: ``` Connected to Kubernetes v1.34.x ``` If it fails, the most common issues I hit were: - Token mismatch or expired token - Missing RBAC permissions - Wrong API endpoint - Security group blocking access --- # Result At this point, Jenkins is fully connected to Kubernetes. Each job that uses the Kubernetes plugin will dynamically provision a pod in the `jenkins-agent` namespace. Once the job finishes, the pod is deleted automatically. This keeps the system clean and avoids resource buildup. ### Connecting Azure and AWS with a Site-to-Site VPN URL: https://anantafatur.dev/connecting-azure-and-aws-with-a-site-to-site-vpn/ Last updated: 2026-04-13T04:38:15.000Z This setup establishes a secure IPsec Site-to-Site VPN between Azure and AWS, enabling private connectivity from AWS workloads to an Azure-hosted PostgreSQL database running in a restricted subnet. A few months ago, the requirement was to connect the AWS environment with the Azure environment as part of a planned database migration from Azure PostgreSQL to AWS RDS. Since both environments were fully private with no public IP exposure, direct connectivity was not possible. The only viable approach was to establish a Site-to-Site VPN between the two clouds to allow controlled network-level access. After the VPN connection was successfully established, the network integration itself worked as expected. However, during later stages of validation, the migration plan using AWS DMS was eventually cancelled due to changes in requirements. Even so, the hybrid connectivity setup remained useful and became a solid reference implementation for IPsec Site-to-Site VPN between AWS and Azure. --- ## 1\. Azure Network Environment The Azure resources are deployed under the resource group `service-kubernetes-production`. The main virtual network used is `service-production-aks-vnet`, which operates on the CIDR range `10.3.0.0/16`. Within this virtual network, a dedicated subnet named `pg-subnet-service-production` is configured with the CIDR `10.3.10.0/24`. This subnet is used exclusively for PostgreSQL workloads and is intentionally isolated as a private network segment without direct public access. ``` +--------------------------------------------------------+ | Azure: service-kubernetes-production | | | | +------------------------------------------------+ | | | Virtual Network: service-production-aks-vnet | | | | CIDR: 10.3.0.0/16 | | | | | | | | +--------------------------------------+ | | | | | Subnet: pg-subnet-service-production | | | | | | CIDR: 10.3.10.0/24 | | | | | | | | | | | | PostgreSQL (Private, no public IP) | | | | | +--------------------------------------+ | | | | | | | +------------------------------------------------+ | | | +--------------------------------------------------------+ ``` Azure Architecture --- ## 2\. Azure VPN Gateway Setup To allow external connectivity from AWS, an Azure Virtual Network Gateway is provisioned. The gateway is named `vpn-azure-aws` and deployed in the Indonesia Central region using the VPN Gateway type with SKU `VpnGw2AZ` and Generation 2\. It is associated with the virtual network `service-production-aks-vnet` and exposed through a public IP resource named `pip-vpn-azure-aws`. The public IP is dynamically assigned, and both active-active mode and BGP are disabled to keep the setup simple and static-route based. This gateway serves as the Azure-side endpoint for the IPsec tunnel. --- ## 3\. AWS Network Environment On the AWS side, the infrastructure is built around a VPC with CIDR `10.1.0.0/16`. To support VPN connectivity, a Virtual Private Gateway named `vpg-aws-azure` is attached to this VPC. In addition, a Customer Gateway is created in AWS using the public IP address of the Azure VPN Gateway. This establishes AWS’s reference point for the Azure-side endpoint. ``` +------------------------------------------------------+ | AWS Account / VPC | | CIDR: 10.1.0.0/16 | | | | +----------------------------------------------+ | | | VPC: aws-production-vpc | | | | | | | | +--------------------+ +---------------+ | | | | | Virtual Private | | Customer | | | | | | Gateway | | Gateway | | | | | | vpg-aws-azure | | cg-azure-vpn | | | | | +--------------------+ | (Azure VPN | | | | | | | Public IP) | | | | | | +---------------+ | | | | | | | | | +------ VPN Connection -----------+ | | | | | | +----------------------------------------------+ | | | +------------------------------------------------------+ ``` AWS Architecture --- ## 4\. AWS Site-to-Site VPN Configuration A Site-to-Site VPN connection is then created in AWS with the name `vpn-aws-azure`. The connection uses the Virtual Private Gateway as the target and the previously defined Customer Gateway as the remote endpoint. Static routing is selected as the routing option, and the only advertised remote network is the Azure PostgreSQL subnet `10.3.10.0/24`, since that is the primary workload AWS needs to access. After creation, the VPN configuration file is downloaded using generic vendor settings with IKEv2\. This file contains the tunnel outside IP addresses, pre-shared key, and IPsec configuration details required for Azure-side setup. --- ## 5\. Azure Local Network Gateway Configuration On Azure, a Local Network Gateway named `lng-azure-aws` is created to represent the AWS environment. It is deployed in the same resource group `service-kubernetes-production` and configured with the AWS VPC CIDR `10.1.0.0/16` as the address space. The gateway IP is set to the AWS VPN tunnel endpoint obtained from the downloaded AWS VPN configuration file. This step effectively tells Azure how to reach the AWS network through the VPN tunnel. --- ## 6\. Azure VPN Connection Setup Once both sides are defined, the VPN connection is established in Azure. A connection named `connection-azure-aws` is created between the Azure Virtual Network Gateway `vpn-azure-aws` and the Local Network Gateway `lng-azure-aws`. The connection uses Site-to-Site IPsec with IKEv2, and the pre-shared key is taken directly from the AWS VPN configuration file. This completes the Azure-side tunnel configuration and links both cloud environments. --- ## 7\. AWS Route Table Configuration To ensure traffic flows correctly between environments, the AWS route tables are updated. A static route is added for destination `10.3.10.0/24`, which represents the Azure PostgreSQL subnet, and the target is set to the Virtual Private Gateway `vpg-aws-azure`. This ensures that any traffic originating from AWS destined for Azure is correctly routed through the VPN tunnel. --- ## 8\. Final Connectivity Outcome After all configurations are completed and the tunnel becomes active, a secure hybrid connection is established between AWS and Azure. The AWS VPC `10.1.0.0/16` can now reach the Azure VNet `10.3.0.0/16`, specifically the PostgreSQL subnet `10.3.10.0/24`, over a private IPsec tunnel. This enables secure cross-cloud communication for use cases such as AWS DMS replication, hybrid application architectures, and data access patterns where AWS services need to interact with Azure-hosted databases. ### Sentiment Analysis on Journaling Data with Python and OpenAI URL: https://anantafatur.dev/sentiment-analysis-on-journaling-data-with-python-and-openai/ Last updated: 2026-04-17T11:55:14.000Z ### Background Last month I was trying to get to know a girl. She mentioned that she does daily journaling, so I decided to start journaling as well so we would have something in common to talk about. It’s been around 15 days since I started, and I want to write about this experience as fairly as possible from a non-medical perspective. --- ### Introduction Both of my friends who consulted a psychologist were encouraged to start journaling or note-taking. I think this aligns with some research showing that journaling has been associated with improvements in mental and physical health by reducing anxiety, lowering stress, and promoting adaptive coping mechanisms \[1\]. Journaling is also considered a common non-pharmacological tool in managing mental health issues \[2\]. Based on these points, it seems reasonable that psychologists often suggest journaling. However, at this point I realized something important: I cannot directly assume that what the psychologist suggested was strictly “journaling” or “writing therapy”, because these terms have different meanings and purposes. Writing therapy refers to writing as a method to express and reflect on oneself, often used as a tool for healing and personal growth \[3\]. It usually involves structured reflection such as “why it matters” and “what it means”. On the other hand, journaling is more general and does not require specific instructions or methodologies \[3\]; in practice, it often focuses on recording what happened. There is also expressive writing, which focuses on putting feelings and thoughts into written words to cope with stressful or traumatic situations \[3\]. --- ### Tools Used Since my background is in DevOps engineering, I used tools I was already familiar with. I wrote my journal entries in [Docmost](https://docmost.com/), which I had previously used for work documentation. I separated entries by page, with one page per day/date. At the beginning, I didn’t take this project too seriously, so I also included my daily to-do lists in the same page. This likely introduced unnecessary noise into the data when performing analysis later. For analysis, I used Python along with the OpenAI API to process and extract sentiment from the journal entries. --- ### Analyzing the Emotions As mentioned earlier, I used Python to call the OpenAI API and analyze the journal data stored in JSON format. The goal was to extract structured information such as sentiment, emotions, and a short reason for each entry. Because the data was stored in Docmost, I first exported it by dumping the PostgreSQL database into JSON format so it could be processed by the script. ### Exporting the Docmost Data ```sql SELECT text_content, title FROM public.pages WHERE parent_page_id IN ( '019d530a-8086-7daa-bf70-5944cc4c6f6d', -- April '019d2141-0c8b-79d3-8c11-e750ef994ecc' -- March ) ORDER BY title ASC; ``` ### Sentiment Analysis Script ```python import json import os import re from openai import OpenAI from tqdm import tqdm client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) INPUT_FILE = "journal.json" OUTPUT_FILE = "journal_sentiment.json" def extract_json(text): try: return json.loads(text) except: match = re.search(r'\{.*\}', text, re.DOTALL) if match: return json.loads(match.group()) else: raise ValueError("No JSON found in response") def analyze_sentiment(text): text = text[:2000] prompt = f""" Analyze this journal entry. Return ONLY JSON: {{ "sentiment": "positive|neutral|negative", "score": number (-1 to 1), "emotions": ["happy","sad","lonely","stressed","angry","anxious","excited","burnout","neutral"], "reason": "max 12 words" }} Text: \"\"\"{text}\"\"\" """ try: response = client.chat.completions.create( model="gpt-4.1-mini", messages=[ {"role": "system", "content": "Return strictly JSON only."}, {"role": "user", "content": prompt} ], temperature=0 ) content = response.choices[0].message.content.strip() return extract_json(content) except Exception as e: return { "sentiment": "error", "score": 0, "emotions": [], "reason": str(e) } def main(): with open(INPUT_FILE, "r") as f: data = json.load(f) entries = list(data.values())[0] results = [] for entry in tqdm(entries): text = entry.get("text_content", "") title = entry.get("title", "") sentiment = analyze_sentiment(text) results.append({ "date": title, "text": text, "sentiment": sentiment.get("sentiment"), "score": sentiment.get("score"), "emotions": sentiment.get("emotions"), "reason": sentiment.get("reason") }) with open(OUTPUT_FILE, "w") as f: json.dump(results, f, indent=2) print(f"Done. Saved to {OUTPUT_FILE}") if __name__ == "__main__": main() ``` ### Example Output ```json { "date": "March 31, 2026", "text": "...", "sentiment": "neutral", "score": 0, "emotions": ["sad", "stressed"], "reason": "Missed chatting, but completed tasks and paper submission" } ``` ### Aggregating Emotion Data To make the results easier to interpret, I aggregated all emotions and counted their occurrences: ```python import json from collections import Counter with open("journal_sentiment.json", "r") as f: data = json.load(f) counter = Counter() for entry in data: emotions = entry.get("emotions", []) counter.update(emotions) for emotion, count in counter.most_common(): print(f"{emotion}: {count}") ``` ### Example Output ``` stressed: 14 anxious: 12 sad: 4 neutral: 4 burnout: 4 lonely: 2 angry: 2 excited: 2 focused: 1 hopeful: 1 frustrated: 1 happy: 1 ``` --- ### Results and Discussion Based on around 15 days of journaling data, the top three emotions were negative (stressed, anxious, sad). Since the data source is my own writing, I can explain this: this suggests my journaling behavior is reactive rather than reflective, as I tend to write when processing negative emotions instead of experiencing positive ones. This behavior aligns more with expressive writing than general journaling. Another possibility is that I prefer to share positive experiences directly with friends instead of writing them down. There is also research suggesting that listening to happy stories can improve people’s mood \[4\]. For a more balanced analysis, it would be better to also include positive experiences in the journal. --- ### Conclusion Besides the initial motivation of getting closer to someone, daily journaling helped me express my emotions more clearly. It also served as a place to track my daily tasks. For future improvements, if the goal is self-understanding, journaling needs to capture both positive and negative experiences. It should intentionally include at least one positive and one negative reflection per entry. From a technical perspective, using LLMs for emotion analysis has limitations. The outputs are not fully consistent due to the generative nature of the models. Even with the same input and model, results may vary. Future work could explore more deterministic approaches using pretrained NLP models such as IndoBERT. Additionally, my writing style is inconsistent, including typos and slang, which may affect analysis accuracy. Standardizing writing style could improve results. --- ### References \[1\] G. Raj, J. Jayson, P. Deshpande, and N. Sood, “Theoretical and practical aspects of modern psychology,” ch. 3, 2025. \[2\] M. Sohal, P. Singh, B. S. Dhillon, and H. S. Gill, “Efficacy of journaling in the management of mental illness: A systematic review and meta-analysis,” Fam. Med. Community Health, vol. 10, no. 1, p. e001154, Mar. 2022, doi: [10.1136/fmch-2021-001154](https://doi.org/10.1136/fmch-2021-001154). \[3\] C. Ruini and C. C. Mortara, “Writing technique across psychotherapies—From traditional expressive writing to new positive psychology interventions: A narrative review,” J. Contemp. Psychother., vol. 52, no. 1, pp. 23–34, Mar. 2022, doi: [10.1007/s10879-021-09520-9](https://doi.org/10.1007/s10879-021-09520-9). \[4\] K. I. Batcho, “When nostalgia tilts to sad: Anticipatory and personal nostalgia,” Front. Psychol., vol. 11, p. 1186, May 2020, doi: [10.3389/fpsyg.2020.01186](https://doi.org/10.3389/fpsyg.2020.01186). ### Bitwarden vs Vaultwarden Resource Comparison URL: https://anantafatur.dev/bitwarden-vs-vaultwarden-resource-comparison/ Last updated: 2026-04-09T05:33:32.000Z Last year, I implemented Vaultwarden for my previous office because it’s much lighter than Bitwarden and already covered our needs, like storing credentials and API keys. For personal use, I’ve been running Bitwarden on my homelab, so this comparison comes from actual usage, not just reading docs. Recently, I moved my password manager from my homelab to a VPS for better uptime. My homelab is not running 24/7, and for something as critical as a password manager, that’s not ideal. The problem is the VPS itself. It has limited CPU and RAM, so I needed to optimize resource usage. That’s where I decided to migrate from Bitwarden to Vaultwarden. This post focuses only on resource usage. No feature comparison, no deep security discussion. Just what I observed from running both. At the time of testing, I used: Bitwarden (lite) version 2026.3.2 Vaultwarden version 1.35.4-alpine --- ### Bitwarden Setup Here is the Docker Compose I used for Bitwarden: ```yaml services: bitwarden-app: depends_on: - bitwarden-mariadb env_file: - .env image: ghcr.io/bitwarden/lite:2026.3.2 restart: always ports: - "80:8080" volumes: - bitwarden-app:/etc/bitwarden bitwarden-mariadb: image: mariadb:10 restart: always env_file: - .env environment: MARIADB_USER: ${MARIADB_USER} MARIADB_PASSWORD: ${MARIADB_PASSWORD} MARIADB_DATABASE: ${MARIADB_DATABASE} MARIADB_RANDOM_ROOT_PASSWORD: "true" volumes: - bitwarden-mariadb:/var/lib/mysql volumes: bitwarden-app: bitwarden-mariadb: ``` Bitwarden runs with two containers. One for the application and one for MariaDB. Even using the lite image, the total size is still quite large. The Bitwarden image is around 1.75 GB and MariaDB adds another 331 MB, so in total it’s about 2.08 GB. When I checked `docker stats` in idle condition, the Bitwarden app container was using around 771 MB of RAM with a small but steady CPU usage. The MariaDB container added about 95 MB. So overall, it’s close to 850 MB RAM just sitting idle. ### Vaultwarden Setup Now let’s compare that with Vaultwarden. ```yaml services: vaultwarden-app: image: vaultwarden/server:1.35.4-alpine container_name: vault-app restart: always environment: DOMAIN: ${DOMAIN} SIGNUPS_ALLOWED: ${SIGNUPS_ALLOWED} INVITATIONS_ALLOWED: ${INVITATIONS_ALLOWED} ADMIN_TOKEN: ${ADMIN_TOKEN} volumes: - /root/production/vault/vault-data/:/data/ ports: - 127.0.0.1:8003:80 ``` Vaultwarden only uses a single container. It relies on SQLite, so there’s no separate database service. I mounted the data directory to the host to make backups easier, but that doesn’t affect runtime usage. The image size is only around 168 MB, which is drastically smaller than Bitwarden. Looking at `docker stats`, the container was using around 42 MB of RAM and almost zero CPU while idle. That’s the total usage, since there’s no extra container behind it. --- ### Final Thoughts After migrating, the difference was obvious. Bitwarden used around 850 MB RAM in idle state, while Vaultwarden dropped to about 42 MB, more than 90 percent reduction. Disk usage also went down from over 2 GB to under 200 MB, which makes a big difference on a small VPS. For my setup, Vaultwarden is the more practical choice. It’s simpler, much lighter, and easier to manage since there’s no separate database. It still covers everything I need, so there’s no real downside for my use case. Both setups were tested with the same data and similar uptime, all measured in idle conditions. This doesn’t mean Vaultwarden is always better, but if your priority is low resource usage, it’s clearly the better fit here. ### Ghost Dark Mode on Headline Theme URL: https://anantafatur.dev/ghost-dark-mode-on-headline-theme/ Last updated: 2026-04-07T15:14:41.000Z Moved to a new room, and the lighting is darker than usual. Working on my laptop made me realize… my blog was too bright! I love dark mode on my laptop, so why not on my blog too? Some sites support it automatically, but my personal blog didn’t. So I asked ChatGPT to help me add dark mode. Now, users can read articles comfortably in darker environments, and I can switch themes anytime! > **Note:** Only affects code injection; safe, but backup Ghost files first at `/var/lib/ghost`. ### Try It Out! Click the 🌙 button at the bottom-right corner to toggle dark/light mode. ✨ --- ### Site Header (CSS Injection) ```html ``` --- ### Site Footer (Button & Script Injection) ```html ``` --- ### Closing That’s it! 🎉 Dark mode is now live on my blog. Whether you’re reading in a bright room or a dark one, you can switch themes anytime using the 🌙 / ☀️ button. ### Experimental Setup: Cost-Optimized Site-to-Site VPN Using StrongSwan URL: https://anantafatur.dev/experimental-setup-cost-optimized-site-to-site-vpn-using-strongswan/ Last updated: 2026-04-06T07:29:54.000Z ### **Why This Exists** Managed Site-to-Site VPN provides convenience, but comes with a fixed hourly cost that may not be efficient for all workloads. This setup explores a self-managed VPN using StrongSwan on EC2, focusing on reducing cost while maintaining strong performance. The objective is to evaluate whether a small ARM-based instance can handle real traffic without becoming a bottleneck. This exploration is currently implemented **within AWS only**, connecting **two separate VPCs**. While the configuration is structured to support multi-cloud scenarios such as AWS to GCP, that part has **not yet been implemented or validated**. ### **Scope & Limitations** This setup connects two AWS VPCs using EC2 instances as VPN gateways. Latency testing was not performed. Some configurations such as restrictive security group rules and advanced routing were not deeply validated, since initial testing used permissive ingress. This should be treated as an exploratory baseline rather than a production-ready reference. --- ### **1\. Architecture Overview** This setup uses one EC2 instance in each VPC acting as a VPN gateway. These instances establish an IPsec tunnel using StrongSwan, allowing private CIDR communication across VPC boundaries. ```text +---------------------+ IPsec VPN +---------------------+ | AWS VPC A | <---------------------> | AWS VPC B | | 10.0.0.0/16 | | 20.0.0.0/16 | | | | | | EC2 (VPN Gateway) | | EC2 (VPN Gateway) | | 10.0.0.242 | | 20.0.4.187 | | 3.109.85.41 (pub) | | 13.233.242.92 (pub)| +---------------------+ +---------------------+ ``` Traffic between subnets is routed through these VPN instances via an encrypted tunnel. ### **2\. Installation (Both Instances)** StrongSwan must be installed on both instances. Additional plugins are included to support extended cryptographic features. ```bash sudo apt update sudo apt install strongswan strongswan-pki libcharon-extra-plugins -y ``` ### **3\. AWS VPC A Configuration** This configuration defines the IPsec tunnel from VPC A to VPC B and is stored in `/etc/ipsec.conf`. It specifies local and remote endpoints, subnets, authentication method, and encryption algorithms. IKEv2 is used for key exchange, with multiple crypto proposals provided to improve compatibility across different environments. ```bash config setup charondebug="none" uniqueids=yes strictcrlpolicy=no conn %default ikelifetime=28800s lifetime=3600s rekeymargin=540s rekeyfuzz=100% keyingtries=3 keyexchange=ikev2 authby=psk mobike=no conn aws-a-to-b left=10.0.0.242 leftid=3.109.85.41 leftsubnet=10.0.0.0/16 leftauth=psk leftfirewall=yes right=13.233.242.92 rightid=13.233.242.92 rightsubnet=20.0.0.0/16 rightauth=psk auto=start type=tunnel dpdaction=restart dpddelay=30s dpdtimeout=120s ike=aes256-sha256-modp2048,aes256-sha256-modp3072,aes256-sha256-modp4096! esp=aes256-sha256,aes256-sha512,aes256gcm16! aggressive=no compress=no forceencaps=yes closeaction=restart replay_window=1024 ``` ### **4\. AWS VPC B Configuration** This configuration mirrors the setup from the opposite side and is also defined in `/etc/ipsec.conf`. The `left` and `right` roles are reversed, but all cryptographic parameters, lifetimes, and tunnel settings must remain consistent to ensure successful negotiation. ```bash config setup charondebug="none" uniqueids=yes strictcrlpolicy=no conn %default ikelifetime=28800s lifetime=3600s rekeymargin=540s rekeyfuzz=100% keyingtries=3 keyexchange=ikev2 authby=psk mobike=no conn aws-b-to-a left=20.0.4.187 leftid=13.233.242.92 leftsubnet=20.0.0.0/16 leftauth=psk leftfirewall=yes right=3.109.85.41 rightid=3.109.85.41 rightsubnet=10.0.0.0/16 rightauth=psk auto=start type=tunnel dpdaction=restart dpddelay=30s dpdtimeout=120s ike=aes256-sha256-modp2048,aes256-sha256-modp3072,aes256-sha256-modp4096! esp=aes256-sha256,aes256-sha512,aes256gcm16! aggressive=no compress=no forceencaps=yes closeaction=restart replay_window=1024 ``` ### **5\. Pre-Shared Key Configuration** StrongSwan separates authentication secrets from tunnel configuration. The Pre-Shared Key (PSK) is stored in `/etc/ipsec.secrets`, and must match on both sides. The identifiers used here must align with `leftid` and `rightid` in the main configuration. Generate a strong key with `openssl rand -hex 32`, then apply on both instances: ```bash # /etc/ipsec.secrets 3.109.85.41 13.233.242.92 : PSK "your-generated-key" ``` ### **6\. System Configuration** IP forwarding must be enabled so the EC2 instance can route traffic between interfaces. IPv6 is disabled here to simplify the setup and avoid unintended routing behavior. ```bash sudo tee -a /etc/sysctl.conf << EOF net.ipv4.ip_forward=1 net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 EOF sudo sysctl -p ``` ### **7\. Routing** Each VPC must know that traffic destined for the remote CIDR should be forwarded to the VPN instance. This is done through route tables. **VPC A Route Table** ```text Destination: 20.0.0.0/16 Target: ENI of VPN EC2 ``` **VPC B Route Table** ```text Destination: 10.0.0.0/16 Target: ENI of VPN EC2 ``` ### **8\. Firewall Rules** The VPN requires specific protocols and ports to function. These rules allow IKE negotiation, NAT traversal, and encrypted traffic flow between both endpoints. ``` +----------+------+---------------------------+ | Protocol | Port | Purpose | +----------+------+---------------------------+ | UDP | 500 | IKE (ISAKMP) | | UDP | 4500 | NAT Traversal | | ESP | 50 | Encrypted IPsec traffic | | AH | 51 | IPsec AH (optional) | +----------+------+---------------------------+ ``` These rules were not strictly tested, as the setup was validated using open inbound access (0.0.0.0/0) during the exploration phase. In a production environment, source IPs should be restricted to the peer VPN endpoint to reduce exposure. Protocol 50 (ESP) is required for carrying encrypted payloads, while Protocol 51 (AH) is optional and typically not used when ESP is enabled. It is also important to note that some cloud platforms require ESP and AH to be allowed as protocols rather than traditional port-based rules. ### **9\. Start Service** Once configuration is complete, StrongSwan must be restarted and enabled to start automatically. ```bash sudo systemctl restart strongswan-starter sudo systemctl enable strongswan-starter ``` ### **10\. Performance Benchmark** To evaluate throughput, `iperf3` is used to generate traffic across the tunnel. Multiple parallel streams help simulate realistic load. ```bash #!/bin/bash TARGET_IP="10.0.0.242" DURATION=60 LOGFILE="vpn_iperf_stress_$(date +%F_%H-%M-%S).log" while true; do iperf3 -c "$TARGET_IP" -t "$DURATION" -P 4 | tee -a "$LOGFILE" done ``` Across 10 runs, the total transferred data reached 146.0 GB, with an average throughput of approximately 1.91 Gbit/s using a `t4g.micro` instance. This indicates that even a small instance can achieve high throughput under burst conditions. During this test, outbound traffic (egress) was billed for the full 146 GB transferred. The exact cost was not recorded, but this highlights an important consideration: while compute costs can be optimized, data transfer charges may become a significant portion of the total cost depending on traffic volume. --- ### **Cost Savings** Managed VPN costs $36.50/month (calculated as $0.05/hour × 730 hours), while the self-managed setup costs $11.39/month (t4g.micro: $0.0106 × 730 = $7.74, Elastic IP: $0.005 × 730 = $3.65). This results in an estimated saving of $25.11/month, or approximately 69%. Even without including external cloud costs, this already demonstrates a strong cost advantage for simple VPN use cases. In practice, the cost can be optimized further, as the VPN service does not strictly require a dedicated EC2 instance and can be integrated into an existing instance such as a bastion host or utility server. Another important consideration is the network performance characteristics of the instance type. The `t4g.micro` instance used in this setup provides a baseline bandwidth of approximately 87 Mbps, with the ability to burst up to around 2085 Mbps under favorable conditions. This explains how high throughput (around 1.91 Gbit/s) can be observed during testing, even on a small instance, as it leverages burst capability rather than sustained baseline performance. However, this performance is not guaranteed continuously, as it depends on available credits and workload patterns. If a more consistent baseline throughput or higher sustained bandwidth is required, it is recommended to choose a different instance type with higher guaranteed network performance. AWS provides detailed specifications for each instance family, including baseline and maximum bandwidth, which can be referenced here: --- ## **When This Approach Makes Sense** This setup is suitable when cost optimization is important and there is willingness to manage infrastructure manually. It works well for low to medium traffic scenarios where performance is still required. It is less suitable for environments requiring managed service guarantees, built-in high availability, or minimal operational overhead. ### Resolving EDAS PDF Submission Issues: Bookmarks and Links URL: https://anantafatur.dev/resolving-edas-pdf-submission-issues-bookmarks-and-links/ Last updated: 2026-04-03T10:59:53.000Z When uploading my final manuscript to EDAS, I encountered the following errors: - **pdf links (URLs) are not allowed (**[**FAQ 221**](https://edas.info/faq221)**)** - **pdf Bookmarks are not allowed (**[**FAQ 115**](https://edas.info/faq115)**)** These errors prevented submission. Although they may seem like small issues, in practice they were not trivial and took a long time to debug, especially because I was using Google Docs instead of Microsoft Word. Honestly, even though the issue is “simple,” I spent around 2 hours solving it because I don’t fully understand how Google Docs or Word handles bookmarks and links; I’m just a DevOps engineer. --- ## Issue 1: Links (URLs) from Zotero I used Zotero as my citation tool, which by default creates links between in-text citations, the references section, and the Zotero application. While these links are helpful during writing, EDAS detects them as URLs and rejects the PDF. The solution is relatively straightforward. First, create a copy of the Google Docs file and open the copied version. Then use Zotero → Unlink Citations to remove all links. It is important to always create a copy before unlinking, since the process is irreversible and citations cannot be relinked automatically afterward. For any further revisions, edits should be made in the main or master document, followed by creating a new copy and unlinking citations again before submission. --- ## Issue 2: Bookmarks (Unexpected and Hidden) This part was more confusing. In Google Docs, a bookmark is a feature that acts as a pointer to a specific location in the document. However, during writing, I did not intentionally use bookmarks at all. Despite that, EDAS reported that the PDF contained bookmarks. At this point, I assumed it was a simple issue and tried removing bookmarks using PDF tools. --- ### Attempt 1: BentoPDF (Self-hosted) Since the document was confidential, I could only use self-hosted PDF tools and could not rely on public services. I first tried BentoPDF, exploring features such as removing annotations, securing the PDF, and editing bookmarks. Despite these efforts, EDAS still rejected the file. ### Attempt 2: StirlingPDF (Self-hosted) Because BentoPDF did not work, I tried another self-hosted tool, StirlingPDF. I repeated the same steps, but the result was unchanged: EDAS continued to detect bookmarks. At this stage, I needed to confirm whether bookmarks were actually present in the PDF. ### Attempt 3: Adobe Acrobat During my research, I found a discussion on the Adobe forum about removing bookmarks and decided to try Adobe Acrobat. After opening the file, I made two findings: the old PDF file contained bookmarks, but the new PDF file processed through BentoPDF did not show any bookmarks. I also tried to remove bookmarks in Acrobat as suggested by the forum. Despite this, EDAS still rejected the file. This was confusing because Acrobat could read the bookmark list in my old PDF but showed none in the new PDF, yet EDAS continued to detect bookmarks. It made me realize there must be hidden bookmarks that Acrobat does not show. ### Attempt 4: Microsoft Word (Hidden Bookmarks) To investigate further, I downloaded the document as a `.docx` and opened it in Microsoft Word. By checking Insert → Bookmark → Hidden bookmarks, I found multiple hidden bookmarks. It was possible to delete them manually, but the layout changed and the page count increased from 6 to 7 pages. Since this was a camera-ready paper, I did not want to risk altering the layout. ### Verification with `pdftk` To answer the question: *does the PDF really contain bookmarks?* I used `pdftk your.pdf dump_data | grep Bookmark`. The result: ``` BookmarkTitle: I. INTRODUCTION BookmarkTitle: II. EXPERIMENTAL METHODOLOGY ... ``` This confirmed that bookmarks were indeed present in the PDF, even though some tools did not display them. ### Final Solution: Removing Bookmarks with `qpdf` The goal was to remove bookmarks from the PDF metadata without altering the layout. I first installed `qpdf` using `sudo apt install qpdf`. Then, I rebuilt the PDF without bookmarks by running `qpdf --empty --pages your.pdf 1-z – no_bookmarks.pdf`. To verify that all bookmarks were removed, I used `pdftk no_bookmarks.pdf dump_data | grep Bookmark`, which returned `Warning: no info dictionary found`. After this step, the PDF was finally compliant with EDAS requirements. --- ## Final Result After successfully removing all links and hidden bookmarks, I uploaded the cleaned PDF to EDAS. The paper was accepted, but two warnings appeared: `Not certified by PDF eXpress”` and `“PDF version 1.3 (recommended 1.4+)”`. These warnings did not block the submission, and the committee indicated that authors do not need to use PDF eXpress themselves, so I considered them safe to ignore for now. > Honestly i feel really tired, it took me around 2 hours to solve this because of my limited understanding and i couldn’t even share the story with `adalahpokoknya` since we’re in our silent phase, hiks 😭🥀 ### Automating GitLab Artifact Cleanup with Jenkins URL: https://anantafatur.dev/automating-gitlab-artifact-cleanup-with-jenkins/ Last updated: 2026-03-31T07:40:29.000Z In our organization, we use a self-hosted GitLab instead of the SaaS version. While this gives us full control, it also means that storage management becomes our responsibility. During my first month, I was tasked with reducing disk usage on GitLab, but with a constraint: I didn’t have access to the GitLab server directly. This meant no manual checks, only API access. To identify which repositories were consuming the most storage, I wrote a Python script to fetch project statistics from GitLab. The script collects total storage size, repository size, job artifacts size, container registry size, wiki size, LFS usage, and the last activity timestamp. Exporting the results to a CSV gave us a clear overview and helped us pinpoint the main culprit: job artifacts. These are files produced by CI/CD jobs, such as build outputs, test reports, logs, or packaged applications, which can grow uncontrollably if not cleaned up regularly. ### Python Script to Collect GitLab Storage Data ```python import requests import csv from datetime import datetime from zoneinfo import ZoneInfo GITLAB_URL = "https://git.company.com" PRIVATE_TOKEN = "adalahpokoknya" headers = {"PRIVATE-TOKEN": PRIVATE_TOKEN} all_projects = [] for page in range(1, 11): r = requests.get( f"{GITLAB_URL}/api/v4/projects", headers=headers, params={ "statistics": True, "order_by": "storage_size", "sort": "desc", "per_page": 100, "page": page } ) if r.status_code != 200: print("Error:", r.text) break data = r.json() if not data: break all_projects.extend(data) with open("gitlab_storage_report.csv", mode="w", newline="", encoding="utf-8") as file: writer = csv.writer(file) writer.writerow([ "Project", "Namespace", "Total_Storage_MB", "Repository_MB", "Artifacts_MB", "Registry_MB", "Wiki_MB", "LFS_MB", "Visibility", "Last_Activity" ]) for p in all_projects: stats = p.get("statistics", {}) last_activity_raw = p.get("last_activity_at") if last_activity_raw: dt = datetime.fromisoformat(last_activity_raw) dt_gmt7 = dt.astimezone(ZoneInfo("Asia/Jakarta")) last_activity = dt_gmt7.strftime("%Y-%m-%d %H:%M:%S") else: last_activity = "" writer.writerow([ p.get("name"), p.get("path_with_namespace"), round(stats.get("storage_size", 0) / (1024 * 1024), 2), round(stats.get("repository_size", 0) / (1024 * 1024), 2), round(stats.get("job_artifacts_size", 0) / (1024 * 1024), 2), round(stats.get("container_registry_size", 0) / (1024 * 1024), 2), round(stats.get("wiki_size", 0) / (1024 * 1024), 2), round(stats.get("lfs_objects_size", 0) / (1024 * 1024), 2), p.get("visibility"), last_activity ]) ``` After generating the CSV, it became evident that most storage was consumed by artifacts. Many projects were retaining artifacts indefinitely, and old pipelines were never cleaned. To reduce risk while reclaiming storage, we decided to keep the latest 70 percent of jobs and delete the oldest 30 percent. This approach protects recent pipelines while significantly reducing storage usage. For automation, we leveraged Jenkins, which was already integrated in our CI/CD ecosystem. The cleanup pipeline runs daily and has two stages: a dry run and execution. The dry run fetches jobs, calculates total and deletable artifact storage, and generates a log and a deletion list without actually deleting anything. The execution stage deletes artifacts using the GitLab API. ### Jenkins Pipeline for Artifact Cleanup ```groovy pipeline { agent { kubernetes { label 'cleanup-agent' defaultContainer 'cleanup' yaml """ apiVersion: v1 kind: Pod spec: containers: - name: cleanup image: alpine:3.23.3 command: - cat tty: true """ } } parameters { string(name: 'PROJECT_IDS_INPUT', defaultValue: '1938,5638,2568,2569,2566,2576,2570,5302,2007', description: 'Comma-separated Project IDs') booleanParam(name: 'RUN_DELETE', defaultValue: false, description: 'Set true to actually delete artifacts') } environment { LOG_FILE = 'artifact_cleanup.log' TOKEN = credentials('pat-gitlab-cleanup') } triggers { cron('H 3 * * *') } stages { stage('Cleanup GitLab Artifacts - Dry Run') { steps { container('cleanup') { sh ''' apk add --no-cache curl jq coreutils echo "=== Artifact Cleanup Dry Run (Keep 70%) ===" > $LOG_FILE : > delete_ids.txt PROJECT_IDS=${PROJECT_IDS_INPUT:-'1938,5638,2568,2569,2566,2576,2570,5302,2007'} ... ''' } } } stage('Cleanup GitLab Artifacts - Execute') { when { anyOf { expression { return params.RUN_DELETE == true } triggeredBy 'TimerTrigger' } } steps { container('cleanup') { sh ''' apk add --no-cache curl echo "=== Executing Artifact Cleanup ===" >> $LOG_FILE ... ''' } } } stage('Archive Logs') { steps { container('cleanup') { archiveArtifacts artifacts: 'artifact_cleanup.log', allowEmptyArchive: true } } } } } ``` ### Example Output ``` Project 1938 - Total Jobs: 500 Keep: 350 Delete: 150 Total Artifact Storage : 2048 MB To Be Deleted : 820 MB Remaining After Cleanup: 1228 MB Global Summary: Projects processed : 9 Total Artifact Storage : 10,240 MB Scheduled for Deletion : 4,200 MB Remaining After Cleanup : 6,040 MB ``` This pipeline allowed us to automatically clean up old artifacts, reclaiming storage without affecting active pipelines. By combining Python scripts for analysis, a clear retention policy, and Jenkins for automation, we were able to keep the GitLab instance manageable even without direct server access. The approach is safe, repeatable, and can be adapted for different retention policies per project or branch. ### ssh: could not resolve hostname - Kubernetes Jenkins Agent DNS Failure URL: https://anantafatur.dev/ssh-could-not-resolve-hostname-kubernetes-jenkins-agent-dns-failure/ Last updated: 2026-03-27T14:56:53.000Z We were setting up Jenkins from scratch for some reason. The setup is simple: Jenkins listens to push events from GitLab (`https://git.company.com`), then runs a pipeline to clone the repo and execute `terraform plan / apply / destroy`. The flow is straightforward: clone repo → `terraform init` → `terraform plan/apply`. The issue started after cloning. Repo cloning via SSH works fine, no problem at all. But when Terraform runs `init` to download modules (which are also hosted in the same GitLab), it fails with: ```bash ssh: could not resolve hostname: try again ``` This is confusing. Same GitLab domain, same SSH method, but behavior is different. Clone works, module download doesn’t. We tried the usual things: restarting `tke-cni-agent`, `kube-proxy`, and `coredns`. No change. Still failing. At this point we suspected DNS, but not sure how bad it was. So we spun up a debug daemonset across all 10 nodes using the same image, and ran `nslookup` to `git.company.com`. The result was random. Sometimes only 1–2 pods could resolve it, sometimes none, sometimes all worked. No consistent pattern. So this is not Terraform, not Jenkins, not SSH. This is cluster DNS being unreliable under this condition. Terraform just exposed it because it does multiple fetches during `init`. We didn’t try to fix DNS properly. That’s a different problem and would take longer. We just needed the pipeline to work. So we forced the pod to use Google DNS instead of `coredns`: ```yaml apiVersion: v1 kind: Pod metadata: name: terraform pipeline spec: ######Fix DNS Issue###### dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 searches: - default.svc.cluster.local ######Fix DNS Issue###### containers: - name: terraform image: hashicorp/terraform:1.14.3 imagePullPolicy: Always command: - cat tty: true stdin: true securityContext: runAsUser: 0 fsGroup: 0 ``` By doing this, `/etc/resolv.conf` inside the container uses `8.8.8.8` instead of cluster DNS. After that, Terraform module downloads worked consistently. This is obviously tech debt, but at least the pipeline works and we can deliver the MVP. Proper fix should be investigating `coredns` or network layer, but that’s for later (hopefully not me). ### Enabling SSH Password Auth on EC2 (Ubuntu 22.04) with Cloud-Init Override URL: https://anantafatur.dev/enabling-ssh-password-auth-on-ec2-ubuntu-22-04-with-cloud-init-override/ Last updated: 2026-03-24T08:22:52.000Z I ran into this while trying to send a large and confidential file from one EC2 to another. Both instances are running Ubuntu 22\. The source EC2 is internal (company instance), no public IP, and only accessible via AWS Session Manager. Because of that, a lot of options are not viable. S3 is not an option because I don’t want to configure AWS auth on that instance. It’s risky and might leave traces. FTP also doesn’t make sense here due to networking constraints. So the only realistic option left is `scp`. Normally I would do: ``` scp -i /path/to/private-key.pem file.txt ubuntu@target-ip:/remote/path/ ``` But I can’t bring the private key into the source EC2\. Even if temporary, it can be logged and questioned later. So I need the target EC2 to accept SSH without key, meaning password-based authentication. --- First attempt, modify `/etc/ssh/sshd_config` by setting `PasswordAuthentication yes` and `PubkeyAuthentication yes`, then restart using `sudo systemctl restart ssh`. After that, test the connection with `ssh ubuntu@target-ip`, which results in `Permission denied (publickey)`, meaning it still requires public key authentication. Second attempt, disable public key authentication by setting `PasswordAuthentication yes` and `PubkeyAuthentication no`, then restart the SSH service again. After testing, the error changes to `Permission denied, please try again.`, which indicates that password authentication is now being attempted, but still failing. --- After some digging, it became clear this wasn’t just a simple SSH config issue. On EC2 Ubuntu, cloud-init is involved in the initial VM setup and it can inject or override SSH configuration behind the scenes. Checking `ls /etc/ssh/sshd_config.d/` reveals an additional config file, `60-cloudimg-settings.conf`, which is not part of the main `sshd_config` but still gets loaded by SSH. Looking into it with `grep -Ri passwordauthentication /etc/ssh/sshd_config.d/` shows `/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no`. At this point the behavior makes sense, because even though `PasswordAuthentication` was already set to `yes` in the main config, this cloud-init managed file is applied later and overrides it back to `no`. --- The fix ends up being a single change on the cloud-init managed config. Update it directly with `sudo sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config.d/60-cloudimg-settings.conf`, then verify using `grep -Ri passwordauthentication /etc/ssh/sshd_config.d/`, which should now return `/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication yes`. After that, restart SSH with `sudo systemctl restart ssh` and test again using `ssh ubuntu@target-ip`. --- Environment for this write-up is Ubuntu 22.04.5 LTS using AMI ID `ami-07216ac99dc46a187`. This wasn’t tested on other Ubuntu versions, but should be reproducible in environments where cloud-init is involved. > This is not something to leave enabled permanently. This is just a workaround to transfer files under constraints. After done, it should be reverted. ### io.fabric8.kubernetes.client.KubernetesClientException: A Jenkins Plugin Version Mismatch URL: https://anantafatur.dev/io-fabric8-kubernetes-client-kubernetesclientexception-a-jenkins-plugin-version-mismatch/ Last updated: 2026-03-17T16:39:18.000Z ### Issue Overview We hit an issue while trying to add a new Kubernetes cluster into an existing Jenkins setup. During “Test Connection” in Jenkins cloud config, it failed with: `io.fabric8.kubernetes.client.KubernetesClientException` At first glance, this looked like a typical infra issue. Firewall, RBAC, credentials, something along those lines. But the behavior didn’t match. The same cluster and credentials worked perfectly when tested from another Jenkins environment. That was the first hint this wasn’t about access. ### Jenkins & Kubernetes Setup The setup itself matters here. We have two Jenkins environments: - Old Jenkins → `2.492.2`, Java `17`, kubernetes plugin `4306.vc91e951ea_eb_d`, kubernetes-client-api `6.10.0-251.v556f5f100500` - New Jenkins → `2.546`, Java `21`, kubernetes plugin `4416.v2ea_b_5372da_a_e`, kubernetes-client-api `7.3.1-256.v788a_0b_787114` And two kubernetes clusters: - Old cluster → `v1.32.12` - New cluster → `v1.34.1` Behavior was consistent: - Old cluster → Old Jenkins → works - New cluster → New Jenkins → works - New cluster → Old Jenkins → fails Same endpoint, same credentials. So we ruled out network, firewall, and RBAC pretty early, even though we still double-checked out of habit. From the Jenkins VM, a simple: `curl https:///version` returned a valid response. That confirmed the API was reachable and TLS was fine. The failure was happening inside Jenkins itself. ### Investigation & Clues The real clue came from the logs: `Unrecognized field "emulationMajor" (class io.fabric8.kubernetes.client.VersionInfo)` That’s not connectivity. That’s parsing failure. So we checked the `/version` endpoint manually. Old cluster response: ```json { "major": "1", "minor": "32", "gitVersion": "v1.32.12" } ``` New cluster response: ```json { "major": "1", "minor": "34", "emulationMajor": "1", "emulationMinor": "34", "minCompatibilityMajor": "1", "minCompatibilityMinor": "33", "gitVersion": "v1.34.1" } ``` The new cluster introduces additional fields like `emulationMajor` and `minCompatibilityMajor`. The older Fabric8 client used by `kubernetes-client-api 6.10.0` doesn’t recognize these fields and fails during deserialization. It doesn’t ignore unknown fields, it just throws an exception. That’s why the connection test fails even though everything else is fine. This also explains why the new Jenkins works. It uses `kubernetes-client-api 7.3.1`, which already supports the updated API response. Same cluster, same credentials, different client version, different result. ### Solution The fix was straightforward once the root cause was clear. We upgraded the Kubernetes plugin in the old Jenkins. Through incremental testing, we found that versions below `4355.v37e9e7c240e6` still fail, and starting from that version the issue is resolved. We didn’t jump straight to latest, just moved step by step until it worked. The upgrade wasn’t isolated. Jenkins pulled a chain of dependencies along with it: - **Kubernetes Client API** - Old Version: `6.10.0-251.v556f5f100500` - Upgraded Version: `7.3.1-256.v788a_0b_787114` - **Kubernetes** - Old Version: `4306.vc91e951ea_eb_d` - Upgraded Version: `4355.v37e9e7c240e6` - **Docker Commons** - Old Version: `451.vd12c371eeeb_3` - Upgraded Version: `457.v0f62a_94f11a_3` - **Kubernetes Credentials** - Old Version: `192.v4d5b_1c429d17` - Upgraded Version: `203.v85b_9836a_f44b_` - **Folders Plugin (cloudbees-folder)** - Old Version: `6.1012.v79a_86a_1ea_c1f` - Upgraded Version: `6.1026.ve06dfa_cf31c3` - **Metrics** - Old Version: `4.2.30-471.v55fa_495f2b_f5` - Upgraded Version: `4.2.32-476.v5042e1c1edd7` We didn’t selectively upgrade plugins. This followed Jenkins dependency resolution. After the upgrade, both clusters connected successfully: - **Old cluster:** Status: Connected, Version: v1.32.12 - **New cluster:** Status: Connected, Version: v1.34.1 Looking back, the main mistake was assuming this was an infrastructure issue. We spent time checking things that were already implicitly proven working. The error message itself was the strongest signal, but easy to overlook because it doesn’t look like a typical Kubernetes failure. > One important note: all of this was done in staging. We haven’t rolled the plugin upgrade into production yet. Upgrading Jenkins plugins can have side effects on pipelines and other integrations. ### Backing Up a Kubernetes to S3 with Velero URL: https://anantafatur.dev/backing-up-a-kubernetes-to-s3-with-velero/ Last updated: 2026-03-09T06:06:12.000Z I wanted to sanity-check something simple: if a namespace disappears from a cluster, how quickly can I bring it back? This wasn’t meant to be a full disaster recovery experiment or a production-grade benchmark. The goal was just to validate that a namespace with a running application could be backed up, deleted, and restored without much friction. There are several Kubernetes backup tools available. Some are quite feature-heavy and designed for enterprise environments. One example is Kasten K10, which is powerful but follows a pricing model. For this quick test I preferred something lightweight and open source, so I used Velero. Velero’s design is relatively straightforward: Kubernetes resource manifests are backed up, volume data can optionally be snapshot through the storage provider, and everything ends up stored in object storage. In this test the object storage backend was Amazon S3, and the cluster itself was a small K3s environment running `v1.33.5+k3s1`. I intentionally skipped PersistentVolumes for this run because I only wanted to confirm the mechanics of backing up and restoring namespace resources first. --- ### Installing the Velero CLI Velero distributes a client binary via GitHub releases. I downloaded the Linux tarball for version `v1.18.0`. ```bash wget https://github.com/vmware-tanzu/velero/releases/download/v1.18.0/velero-v1.18.0-linux-amd64.tar.gz ``` Download result: ``` Saving to: ‘velero-v1.18.0-linux-amd64.tar.gz’ velero-v1.18.0-linux-amd64.tar.gz 100% 52.71M 27.0MB/s in 2.0s ``` Then extract the archive. ```bash tar -xvf velero-v1.18.0-linux-amd64.tar.gz ``` Extraction produced the following files. ``` velero-v1.18.0-linux-amd64/LICENSE velero-v1.18.0-linux-amd64/examples/minio/00-minio-deployment.yaml velero-v1.18.0-linux-amd64/examples/nginx-app/README.md velero-v1.18.0-linux-amd64/examples/nginx-app/base.yaml velero-v1.18.0-linux-amd64/examples/nginx-app/with-pv.yaml velero-v1.18.0-linux-amd64/velero ``` After that I moved the binary into the system path so it can be used anywhere. ```bash mv velero-v1.18.0-linux-amd64/velero /usr/local/bin/ ``` To confirm the client installation: ```bash velero version ``` Output: ``` Client: Version: v1.18.0 Git commit: 6adcf06b5b0e6fb93998d3e101e2cbdc134fa3c3 ``` At this stage only the CLI exists locally. Nothing has been installed in Kubernetes yet. --- ### Preparing Object Storage Velero requires an object storage location to store backup metadata. In this case I used an S3 bucket. ```bash BUCKET=s3-anantafatur-velero REGION=ap-southeast-3 aws s3api create-bucket \ --bucket $BUCKET \ --region $REGION \ --create-bucket-configuration LocationConstraint=$REGION ``` The bucket itself is just a storage target for backup artifacts. Velero will later store compressed archives containing Kubernetes manifests and metadata. ### IAM Setup Because Velero interacts with S3 and potentially EC2 snapshots, it needs an IAM identity with the appropriate permissions. I created a dedicated IAM user named `velero`. ```bash aws iam create-user --user-name velero ``` Result: ``` { "User": { "UserName": "velero", "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/velero" } } ``` Next I created a policy granting permissions for EC2 snapshot operations and S3 object access within the Velero bucket. ```bash cat > velero-policy.json < SecretAccessKey: ``` Velero expects these credentials in a small configuration file. ```bash cat < credentials-velero [default] aws_access_key_id= aws_secret_access_key= EOF ``` --- ### Installing Velero in the Cluster With the CLI and credentials ready, Velero can install its controller components into Kubernetes. ```bash velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.14.0 \ --bucket "$BUCKET" \ --secret-file ./credentials-velero \ --backup-location-config region="$REGION" \ --snapshot-location-config region="$REGION" ``` The installer creates several CustomResourceDefinitions and deploys the Velero controller. ``` CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/restores.velero.io: created Namespace/velero: created ServiceAccount/velero: created Secret/cloud-credentials: created Deployment/velero: created ``` Once installation finished, I checked that the deployment was running. ```bash kubectl get all -n velero ``` Result: ``` pod/velero-57d646d4f-dbqpj 1/1 Running deployment.apps/velero replicaset.apps/velero ``` At this point the Velero controller was active in the cluster. ### Deploying a Test Application The cluster didn’t have any workloads running yet, so I deployed a small demo application to serve as backup material. First I created a namespace. ```bash kubectl create ns velero-demo ``` Then I deployed a simple JavaScript game application. ```bash kubectl apply -f https://raw.githubusercontent.com/mimqupspy/k3s-yaml/refs/heads/main/js-game-app.yml -n velero-demo ``` Checking the resources confirmed that the application was running. ```bash kubectl get all -n velero-demo ``` Result: ``` pod/js-game-app-995486fc9-zrzcb 1/1 Running service/js-game-app-service deployment.apps/js-game-app ``` For easier testing I changed the service type to NodePort so the application could be accessed externally. ```bash kubectl patch svc js-game-app-service -p '{"spec":{"type":"NodePort"}}' -n velero-demo ``` The service then looked like this: ``` service/js-game-app-service NodePort 3000:31361/TCP ``` ![](https://anantafatur.dev/content/images/2026/03/image.png) http://172.16.0.101:31361 At this point the application was running normally in the namespace. --- ### Creating the Backup With a working namespace in place, I triggered a Velero backup. ```bash velero backup create velero-demo-backup --include-namespaces velero-demo ``` Velero immediately accepted the request. ``` Backup request "velero-demo-backup" submitted successfully. ``` Checking the backup status showed that the operation finished quickly. ```bash velero backup describe velero-demo-backup ``` Important fields from the output: ``` Phase: Completed Started: 11:29:19 Completed: 11:29:26 Total items to be backed up: 17 Items backed up: 17 ``` On this cluster the entire backup finished in roughly seven seconds. ### Verifying Backup Data in S3 To confirm that the backup actually reached S3, I checked the bucket. ```bash aws s3 ls s3://s3-anantafatur-velero/backups/velero-demo-backup/ ``` Objects stored in the bucket included: ``` velero-backup.json velero-demo-backup-itemoperations.json.gz velero-demo-backup-logs.gz velero-demo-backup-resource-list.json.gz velero-demo-backup.tar.gz ``` The compressed archive contains the Kubernetes manifests for the backed-up resources. Because this test only involved metadata, the files were very small. --- ### Simulating a Failure To simulate an accidental deletion, I removed the entire namespace. ```bash kubectl delete ns velero-demo ``` After the namespace deletion completed, the application was gone. ### Restoring the Backup First I checked which backups Velero knew about. ```bash velero get backups ``` Output: ``` NAME STATUS CREATED velero-demo-backup Completed ``` Then I initiated the restore. ```bash velero restore create --from-backup velero-demo-backup ``` Velero created a restore request. ``` Restore request "velero-demo-backup-20260309113414" submitted successfully ``` Checking the restore details showed that it completed successfully. ```bash velero restore describe velero-demo-backup-20260309113414 ``` Result: ``` Phase: Completed Total items to be restored: 10 Items restored: 10 ``` ### Verifying the Restored Application After the restore finished I checked the namespace again. ```bash kubectl get all -n velero-demo ``` The application resources were recreated: ``` pod/js-game-app-995486fc9-zrzcb Running deployment.apps/js-game-app service/js-game-app-service ``` One detail changed compared to the original deployment. The NodePort assigned to the service was different. Original service: ``` 3000:31361 ``` Restored service: ``` 3000:31159 ``` ![](https://anantafatur.dev/content/images/2026/03/image-1.png) http://172.16.0.101:31159 Kubernetes simply allocated a new port when recreating the service. --- Overall the backup and restore workflow behaved exactly as expected. Velero successfully captured the namespace resources and restored them after deletion, and the artifacts stored in S3 were very small since they only contained Kubernetes manifests rather than application data. During the restore process, the Deployment, ReplicaSet, Pod, and Service were recreated without issues. This run also intentionally skipped several areas that would matter in a more realistic setup. The test did not cover backing up PersistentVolumes, restoring stateful workloads, restoring into a different cluster, configuring scheduled backups, or setting retention policies. Those scenarios are usually where backup tooling becomes more interesting and complex. Testing PVC backups would probably be the next logical step, but for this proof of concept I didn’t feel like extending the scope that far. ### Streamline Email Handling with Microsoft Power Automate and Planner URL: https://anantafatur.dev/streamline-email-handling-with-microsoft-power-automate-and-planner/ Last updated: 2026-03-04T05:56:49.000Z We have a team distribution list, for example `devops@company.com`. Everyone in the DevOps team receives those emails. That gives visibility, but it does not give ownership. Over time, the manager noticed the same pattern: some emails were answered late, and some were ignored because everyone assumed someone else would handle them. The manager asked a simple question: “Who handles this email?” There was no deterministic answer. We already use Microsoft 365 for email, collaboration, and task tracking. The constraint was to stay inside the ecosystem. The goal was not to build a ticketing system. The goal was to enforce single ownership per incoming email thread. The solution is a Power Automate flow that converts incoming emails into Planner tasks and assigns them sequentially based on a rotation stored in SharePoint. This write-up explains every step and expression used. --- ## High-Level Architecture We use: - Outlook (distribution list) - Power Automate (orchestration) - SharePoint (state storage) - Microsoft Planner (task tracking) There are three SharePoint lists involved: 1. **DevopsRotation – Engineers List** ``` +------------+------------------------+------------+--------+ | Title | Email | OrderNumber| Active | +------------+------------------------+------------+--------+ | Engineer 1 | engineer1@example.com | 1 | True | | Engineer 2 | engineer2@example.com | 2 | True | | Engineer 3 | engineer3@example.com | 3 | True | | Engineer 4 | engineer4@example.com | 4 | True | | Engineer 5 | engineer5@example.com | 5 | True | +------------+------------------------+------------+--------+ ``` SharePoint Lists - DevopsRotation The OrderNumber determines the rotation sequence, while an engineer marked as Active = True is included in the rotation. If someone is on leave, their Active status should be set to False. 1. **DevopsRotationState – Rotation Pointer** ``` +---------+-----------+ | Title | LastOrder | +---------+-----------+ | Rotation| 4 | +---------+-----------+ ``` SharePoint Lists - DevopsRotationState The last assigned engineer had an OrderNumber of 4, so the next email in the rotation will be directed to the engineer with OrderNumber 5. 1. **EmailConversations – Deduplication** ``` +----------------------------+--------------------------------------+ | Title | ConversationId | +----------------------------+--------------------------------------+ | (empty or email subject) | (ConversationId of email thread) | +----------------------------+--------------------------------------+ ``` SharePoint Lists - EmailConversations We use ConversationId instead of Subject because replies maintain the same ConversationId, while the subject line may change. --- ## Conceptual Architecture Diagram ``` ┌─────────────────────────┐ │ Distribution List │ │ devops@company.com │ └────────────┬────────────┘ │ ▼ ┌──────────────────┐ │ Outlook Inbox │ │ (Flow Trigger) │ └─────────┬────────┘ │ ▼ ┌──────────────────┐ │ Power Automate │ │ Orchestration │ └─────────┬────────┘ │ ┌────────────────────┼────────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Email │ │ DevopsRotation │ │ DevopsRotation │ │ Conversations │ │ (Engineers List) │ │ State │ │ (Dedup Check) │ │ (Active + Order) │ │ (Last Pointer) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ │ ▼ ┌──────────────────┐ │ Microsoft │ │ Planner │ │ (Task Created │ │ + Assigned) │ └──────────────────┘ ``` ## Flow Overview ``` [Trigger: New Email] | v [Compose_ConversationId] | v [Get Items: EmailConversations] | v [Compose_Debug] | v [Compose_Length] | v [Condition: Length == 0 ?] / \ No Yes | | [Stop] v [Get DevopsRotationState] | v [Compose_LastOrder] | v [Get Active Engineers] | v [Compose_MaxOrder] | v [Compose_NextOrder] | v [Filter_array by OrderNumber] | v [Compose_SelectedEmail] | v [Get_user_profile_(V2)] | v [Compose_UserId] | v [Compose_UserAssignments] | v [Create Planner Task] | v [Delay 5 seconds] | v [Update Task Details] | v [Update DevopsRotationState] | v [Create EmailConversations item] ``` --- ### 1\. Trigger – When a New Email Arrives We start with the flow watching the **DevOps distribution list**. Every incoming email sparks the automation. - **Connector:** Office 365 Outlook - **Trigger:** When a new email arrives (V3) - **Parameters:** - `To = devops@company.com` - `FolderPath = Inbox` Because this is a shared distribution list and you’re part of it, the email lands in your mailbox, our flow is listening there. Every new thread is captured automatically. ### 2\. Extract ConversationId Instead of relying on the email subject (which can change with replies), we grab the **ConversationId**,a unique identifier that persists across the email thread. - **Action:** Compose - **Name:** `Compose_ConversationId` - **Expression:** ``` @triggerOutputs()?['body/conversationId'] ``` This ensures that **all replies stay linked** to the same task, preventing duplicates. ### 3\. Check for Existing Conversation Before creating a task, we check if this conversation has already been handled. - **Action:** Get Items - **List:** EmailConversations - **Filter Query:** ``` ConversationId eq '@{outputs('Compose_ConversationId')}' ``` - **Top count:** 1 Then we debug and count: - **Compose\_Debug:** `@body('EmailConversations')?['value']` - **Compose\_Length:** `@length(body('EmailConversations')?['value'])` **Condition:** - If `Compose_Length > 0` → Stop (this thread already has a task) - If `Compose_Length = 0` → Continue (new thread, ready for assignment) This sets the stage: only **fresh email threads** move forward to the rotation assignment, keeping everything clean and deterministic. ### 4\. Read Rotation State To figure out who should get the next email, we check the rotation’s current position. We use **Get Items** on the `DevopsRotationState` list with this filter: ``` Title eq 'Rotation' ``` We only need the first item (`Top Count = 1`) because there’s just one rotation record. Next, we extract the last assigned engineer’s order number so we can determine who comes next: ``` @int(first(body('DevopsRotationState')?['value'])?['LastOrder']) ``` This value (`LastOrder`) acts as a pointer in our round-robin rotation, helping the flow assign the next task in sequence. ### 5\. Get Active Engineers Now that we know where the rotation left off, we need the list of engineers who are available. Using **Get Items** on the `DevopsRotation` list, we filter only those marked as active: ``` Active eq 1 ``` We also order them by `OrderNumber` ascending so the rotation sequence is preserved. This gives us a clean, ordered pool of engineers ready to receive the next task. ### 6\. Compute Maximum Order With the active engineers in hand, we need to know the size of our rotation. Using a **Compose** action, we calculate the total number of active engineers: ``` @length(body('DevopsRotation')?['value']) ``` This `MaxOrder` becomes our ceiling for the round-robin logic. Think of it as the last number on the spinning wheel, once we hit it, we loop back to the start. ### 7\. Compute Next Order (Circular Logic) Time to pick the next engineer. Here we implement circular logic with another **Compose** action: ``` @if( equals(outputs('Compose_LastOrder'), outputs('Compose_MaxOrder')), 1, add(outputs('Compose_LastOrder'), 1) ) ``` - If the last assigned order equals the maximum → start over at 1. - Otherwise → simply move to the next order. This ensures a smooth **round-robin rotation**, giving each engineer a fair turn without skipping anyone. ### 8\. Select Engineer by OrderNumber Now that we know the `NextOrder`, it’s time to pick our engineer. We use a **Filter Array** action on the `DevopsRotation` list: ``` From: @body('DevopsRotation')?['value'] Filter Query: @equals(string(item()?['OrderNumber']), string(outputs('Compose_NextOrder'))) ``` Then, with a **Compose** action, we extract the selected engineer’s email: ``` @first(body('Filter_array'))?['Email'] ``` This is the lucky engineer who will own the incoming email thread. ### 9\. Get Azure AD User Profile Next, we fetch the user profile from Azure AD so Planner can recognize them: - **Action:** Get\_user\_profile\_(V2) - **Parameter:** `@outputs('Compose_SelectedEmail')` From here, extract the **User ID** using another **Compose**: ``` @body('Get_user_profile_(V2)')?['id'] ``` This ID is what Planner needs to assign tasks properly. ### 10\. Build Planner Assignment Object Planner wants a structured JSON for assignments. We create it in a **Compose** action: ``` @json(concat( '{ "', outputs('Compose_UserId'), '": { "@odata.type": "#microsoft.graph.plannerAssignment", "orderHint": " !" } }' )) ``` This is basically wrapping the engineer’s ID in Planner-friendly armor. ### 11\. Create Planner Task With everything ready, we now create the task in Planner: - **Title:** `@concat('[PA TEST] ', triggerOutputs()?['body/subject'])` - **Start Date:** `@triggerOutputs()?['body/receivedDateTime']` - **Due Date:** ``` @formatDateTime( addDays( convertTimeZone(triggerOutputs()?['body/receivedDateTime'], 'UTC', 'SE Asia Standard Time'), 7 ), 'yyyy-MM-ddTHH:mm:ss' ) ``` - **Assignments:** `@outputs('Get_user_profile_(V2)')?['body/id']` This creates a task for exactly one engineer, the one next in rotation. ### 12\. Delay A small pause is needed. We wait **5 seconds** to ensure Planner fully registers the task before we add details. ### 13\. Update Task Details Now that the task exists, we enrich it with the email’s content so the engineer has full context. - **Task Id:** `@body('Create_a_task')?['id']` - **Description:** ``` @concat( 'From: ', triggerOutputs()?['body/from'], decodeUriComponent('%0A'), 'Received: ', formatDateTime( convertTimeZone(triggerOutputs()?['body/receivedDateTime'], 'UTC', 'SE Asia Standard Time'), 'HH:mm ''WIB'' - dddd, dd MMMM yyyy' ), decodeUriComponent('%0A%0A'), '---', decodeUriComponent('%0A%0A'), if( empty(triggerOutputs()?['body/bodyPreview']), 'This email is protected or has no preview available.', substring( coalesce(triggerOutputs()?['body/bodyPreview'], ''), 0, min(5000, length(coalesce(triggerOutputs()?['body/bodyPreview'], ''))) ) ) ) ``` This captures who sent the email, when it was received, and a preview of the message. The engineer now has all info in one place. ### 14\. Update DevopsRotationState To keep the rotation moving, we update the “last assigned engineer” pointer: - **Patch DevopsRotationState Id:** `@first(body('DevopsRotationState')?['value'])?['ID']` - **LastOrder:** `@outputs('Compose_NextOrder')` This ensures the next incoming email will go to the next engineer in line, round-robin style. ### 15\. Create EmailConversations Item Finally, we record that this conversation has been handled so we don’t duplicate tasks: - **Title:** `@triggerOutputs()?['body/subject']` - **ConversationId:** `@outputs('Compose_ConversationId')` With this, the flow has full memory of every thread, ensuring **single ownership** per email. --- ### 🎯 Final Notes - Engineers on leave? Set `Active = 0` in `DevopsRotation` They automatically skip the rotation. - Concurrency isn’t locked; simultaneous emails could theoretically race, but in practice it’s rare. - Every new email thread results in **exactly one Planner task**, assigned deterministically. - When someone asks, “Who handles this email?”, the answer is visible in Planner, clear as day. This flow is still evolving, but it’s already a **solid framework for predictable ownership** in a team inbox. ### Accessing a Private K3s Control Plane from a VPS over Tailscale URL: https://anantafatur.dev/accessing-a-private-k3s-control-plane-from-a-vps-over-tailscale/ Last updated: 2026-02-26T07:09:05.000Z I’m running a single-node K3s cluster inside my home lab on Proxmox VE. It’s intentionally boring: one control-plane node, no workers, no public IP. The VM sits on my LAN with `172.16.0.101`, and that’s it. No port forwarding, no exposure to the internet. The version is `v1.33.5+k3s1`, installed via the standard script. Separately, I have a VPS running code-server. That VPS has a public IP, but I don’t want to expose Kubernetes API port 6443 publicly just so I can run `kubectl`. The requirement is simple: from the VPS, I want to run `kubectl get nodes` against the home K3s control plane. I was already using Tailscale to access my homelab remotely from my laptop. So instead of inventing something new, I just extended the same mesh to include the VPS. On both machines, I installed Tailscale: ```bash curl -fsSL https://tailscale.com/install.sh | sh sudo tailscale up ``` Both nodes authenticated into the same Tailscale account. On the K3s node, I checked the assigned IP: ```bash tailscale ip -4 ``` It returned: ``` 100.76.241.16 ``` From the VPS, I tested connectivity: ```bash ping 100.76.241.16 ``` It worked immediately. I didn’t touch any firewall rules on either side. As long as both machines are inside the same Tailscale network, the mesh handles the routing. That part was uneventful. Next step was kubeconfig. On the K3s node, the config file lives at: ``` /etc/rancher/k3s/k3s.yaml ``` I copied it manually to the VPS. Inside that file, the `server:` field originally pointed to localhost: ```yaml server: https://127.0.0.1:6443 ``` I changed it to the Tailscale IP: ```yaml server: https://100.76.241.16:6443 ``` At this point I expected it to just work. Network connectivity was there, port 6443 was reachable over the mesh, and nothing was blocked. Then I ran: ```bash kubectl get nodes ``` And got: ``` x509: certificate is valid for 127.0.0.1, 172.16.0.101, not 100.76.241.16 ``` This is where it gets interesting. The failure wasn’t network-related. It was TLS. K3s had generated its API server certificate with Subject Alternative Names that included `127.0.0.1` and the LAN IP `172.16.0.101`, but not the Tailscale IP. So when I connected using `100.76.241.16`, the certificate validation failed exactly as it should. The fix is straightforward but easy to forget. On the K3s node, I edited: ``` /etc/rancher/k3s/config.yaml ``` and added: ```yaml tls-san: - 127.0.0.1 - 172.16.0.101 - 100.76.241.16 ``` Then restarted K3s: ```bash systemctl restart k3s ``` K3s regenerated the serving certificate with the updated SAN entries. No manual certificate work. No reinstallation. It handled it cleanly. Back on the VPS, I ran again: ```bash kubectl get nodes ``` This time the output was: ``` NAME STATUS ROLES AGE VERSION k3s-master Ready control-plane,etcd,master 141d v1.33.5+k3s1 ``` That’s it. No public IP. No port forwarding. No firewall rules changed. The VPS talks to the control plane entirely over the Tailscale mesh, and TLS is properly validated. ### Setting up Jenkins on Kubernetes via Helm (and Fixing Restart Issues) URL: https://anantafatur.dev/setting-up-jenkins-on-kubernetes-via-helm-and-fixing-restart-issues/ Last updated: 2026-02-25T08:06:46.000Z This was a small POC, not production. The goal was simple: regularly clean old GitLab artifacts without touching the existing GitLab pipeline. I initially considered adding a new cleanup stage inside the existing pipeline, but as the most junior in the room, modifying shared CI configuration felt risky. If I broke something, it would affect everyone. We already had Jenkins in our infra, so I proposed using a scheduled Jenkins pipeline to handle artifact cleanup instead. ### Installing Jenkins via Helm Jenkins provides a maintained Helm chart at `https://charts.jenkins.io`, I added the repo and installed it: ``` helm repo add jenkins https://charts.jenkins.io helm repo update ``` In theory, we can just run: `helm upgrade --install jenkins jenkins/jenkins`. And yes, that works initially. Jenkins comes up, you log in, everything looks fine. But after a cluster or node restart, the Jenkins pod did not behave cleanly. ### The Problem After Restart After restarting the k3s node, the Jenkins pod came back in an unhealthy state, looking at the init container logs: `kubectl logs jenkins-0 -n jenkins -c init`. ``` Setting checksum for: configuration-as-code to xA895ZNcY2HZoS9kNLE9KX+ew6VaI3tPoljFbNGTJQA= Setting checksum for: configuration-as-code to xA895ZNcY2HZoS9kNLE9KX+ew6VaI3tPoljFbNGTJQA= ... Done copy plugins to shared volume cp: overwrite '/var/jenkins_plugins/antisamy-markup-formatter.jpi'? cp: overwrite '/var/jenkins_plugins/apache-httpcomponents-client-4-api.jpi'? cp: overwrite '/var/jenkins_plugins/asm-api.jpi'? ... cp: overwrite '/var/jenkins_plugins/workflow-support.jpi'? ``` The init container was trying to copy plugins again into `/var/jenkins_plugins`. Since we were using persistence (default chart behavior with PVC), the plugin directory already existed from the previous run. On restart, the init container re-ran and attempted to overwrite existing plugin files. This looked like re-initialization behavior rather than PVC corruption. Storage was local-path on a single node, so there was no cross-node scheduling issue. The PVC was reused correctly. The friction was coming from the initialization logic itself. ### Inspecting and Modifying values.yaml Instead of guessing Helm chart behavior, I dumped the chart values: ``` helm show values jenkins/jenkins >> jenkins-values.yaml ``` After reviewing the configuration, I found: ``` initializeOnce: false ``` The default behavior allows initialization logic to run more than once. For a persistent controller, that is not what I wanted. So I changed it: ``` sed -i 's/initializeOnce: false/initializeOnce: true/' jenkins-values.yaml ``` With `initializeOnce: true`, the initialization phase runs only on the first startup. On subsequent pod restarts, it skips re-initialization. After this change, restarting the node no longer triggered repeated plugin copy attempts. Then I also set the external URL to avoid reverse proxy warnings: ``` sed -i 's|jenkinsUrl:.*|jenkinsUrl: https://jenkins.anantafatur.dev|' jenkins-values.yaml ``` Finally, I applied the configuration: ``` helm upgrade --install jenkins jenkins/jenkins \ --namespace jenkins \ --create-namespace \ -f jenkins-values.yaml ``` After this, restarts were clean. No repeated plugin copy prompts, no initialization noise. ### Retrieving the Admin Password To retrieve the initial admin password: ``` kubectl get secret jenkins -n jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode && echo ``` Example output: ``` Bfh8gMi4AERZDRiQoedsef ``` For now, this approach gave me a safe playground to experiment without touching shared CI configuration. ### Setting up ArgoCD with NGINX Ingress and Cloudflare Origin Cert URL: https://anantafatur.dev/setting-up-argocd-with-nginx-ingress-and-cloudflare-origin-cert/ Last updated: 2026-02-25T06:48:28.000Z Recently I needed to expose ArgoCD on my cluster with Cloudflare in full-strict TLS mode. Full-strict means Cloudflare will verify the backend certificate, so we must use a Cloudflare Origin Cert attached to the NGINX Ingress. This is a simple setup, but since I forget things easily, I’m writing it down for my own reference (and maybe yours). ![](https://anantafatur.dev/content/images/2026/02/image-17.png) Cloudflare SSL/TLS Environment: single-node Kubernetes (K3s), Helm 4, Cloudflare-managed domain `anantafatur.dev`. Requirement: expose ArgoCD via `argocd-aws.anantafatur.dev` with HTTPS strictly validated end-to-end. ## Helm & NGINX Ingress Install Helm: ```bash curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 && chmod 700 get_helm.sh && ./get_helm.sh ``` Create namespace for ArgoCD: ```bash kubectl create namespace argocd ``` Add NGINX Ingress repo and install: ```bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace ``` Check NGINX Ingress pods: ```bash root@ip-172-31-18-80:~# kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-controller-696dc57956-bbk7r 1/1 Running 0 81s ``` Check NGINX Ingress services: ```bash root@ip-172-31-18-80:~# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.43.112.204 172.31.18.80 80:30110/TCP,443:30248/TCP 84s ingress-nginx-controller-admission ClusterIP 10.43.140.196 443/TCP 84s ``` ## Cloudflare Origin Certificate Downloaded `origin.crt` and `origin.key` from [Cloudflare dashboard](https://dash.cloudflare.com/?to=/:account/:zone/ssl-tls/origin). Create Kubernetes TLS secret: ```bash kubectl create secret tls argocd-cloudflare-origin --cert=origin.crt --key=origin.key -n argocd ``` This secret will be referenced in the Ingress for full-strict TLS. ## ArgoCD Ingress Ingress configuration: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd-ingress namespace: argocd annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" nginx.ingress.kubernetes.io/ssl-redirect: "false" nginx.ingress.kubernetes.io/force-ssl-redirect: "false" nginx.ingress.kubernetes.io/proxy-buffer-size: "16k" nginx.ingress.kubernetes.io/proxy-read-timeout: "300" nginx.ingress.kubernetes.io/proxy-send-timeout: "300" spec: tls: - hosts: - argocd-aws.anantafatur.dev secretName: argocd-cloudflare-origin rules: - host: argocd-aws.anantafatur.dev http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 443 ``` argocd-ingress.yaml Apply it: ```bash kubectl apply -f argocd-ingress.yaml ``` Check ingress: ```bash root@ip-172-31-18-80:~# k get ingress -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE argocd argocd-ingress argocd-aws.anantafatur.dev 172.31.18.80 80, 443 3s ``` Host resolved correctly, TLS terminates with Cloudflare Origin Cert. ## ArgoCD Deployment Fetch Helm chart values: ```bash helm show values argo/argo-cd > argocd-values.yaml ``` Update `argocdUrl` in values: ```bash sed -i 's|argocdUrl: ""|argocdUrl: "argocd-aws.anantafatur.dev"|' argocd-values.yaml kubectl apply -f argocd-values.yaml ``` Install ArgoCD via Helm: ```bash helm install --values argocd-values.yaml argocd argo/argo-cd --namespace argocd ``` Check Helm releases: ```bash root@ip-172-31-18-80:~# helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION argocd argocd 1 2026-02-23 07:37:42.679479302 +0000 UTC deployed argo-cd-9.2.4 v3.2.3 ingress-nginx ingress-nginx 1 2026-02-23 07:33:18.848391993 +0000 UTC deployed ingress-nginx-4.14.1 1.14.1 ``` Retrieve admin password: ```bash root@ip-172-31-18-80:~# kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo i5SCpYon7mh2419R ``` ![](https://anantafatur.dev/content/images/2026/02/image-16.png) argocd-aws.anantafatur.dev ## Note on k3s Since I’m running k3s, remember that by default it bundles Traefik as the ingress controller. I didn’t want Traefik interfering with NGINX, so I disabled it during installation: ```bash curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --disable traefik" sh ``` Simple enough, now NGINX is the only ingress in charge, no conflicts, and ArgoCD can sit nicely behind it. ### Domain Migration for Ghost CMS Using Cloudflare Redirects URL: https://anantafatur.dev/domain-migration-for-ghost-cms-using-cloudflare-redirects/ Last updated: 2026-02-21T11:12:50.000Z This migration was done on a small but production setup: Ghost CMS running in Docker, exposed through Cloudflare Tunnel, with DNS and edge rules managed in Cloudflare. Google Search Console was already configured for the old subdomain. The blog originally lived at `docs.anantafatur.dev`, while `anantafatur.dev` served as a simple one-page personal landing page linking to LinkedIn, YouTube, and the blog. After getting my first job, that landing page no longer had a real purpose. The blog was the only meaningful content, so it made more sense for it to live on the root domain. The constraints were clear. All existing URLs must continue to work. Old links must redirect automatically. Search engines must understand that this is a permanent move. ## Freeing the Root Domain Before touching redirects, I first moved the landing page content to another archival domain. This ensured that `anantafatur.dev` was no longer serving anything. Only after the root domain was fully free did I repoint it to Ghost. This avoids any overlap period where both domains might serve different content. Nothing fancy here. Just making sure the target domain is clean before redirecting traffic into it. ## Implementing the Redirect at the Edge The redirect is the most critical part of the migration. I chose to implement it in Cloudflare rather than inside Ghost. Redirecting at the edge ensures the application never even sees requests for the old domain, and it keeps the backend configuration simpler. In Cloudflare Redirect Rules, I created a rule with the condition: `http.host eq "docs.anantafatur.dev" `and the action uses a dynamic expression: `concat("https://anantafatur.dev", http.request.uri)` with status code set to 301. ![](https://anantafatur.dev/content/images/2026/02/image-13.png) Cloudflare - Redirect Rules Initially, I tried a static redirect pointing to `https://anantafatur.dev`. That worked, but it dropped the path entirely. So any deep link redirected only to the homepage. That’s not acceptable. I also experimented with `${path}`, but Cloudflare encoded it into `%7Bpath%7D`, treating it as a literal string. That one wasted a few minutes. Using `http.request.uri` preserves both path and query string. After applying the rule, a request to: `https://docs.anantafatur.dev/devops-job-hunting-flow-a-sankey-analysis/` correctly redirects to: `https://anantafatur.dev/devops-job-hunting-flow-a-sankey-analysis/` Once this was confirmed, the redirect layer was considered stable. ## Updating Ghost Configuration Since Ghost is running in Docker, configuration is controlled via environment variables. I updated the `.env` file to: `GHOST_URL=https://anantafatur.dev` Then restarted the service: `docker compose down && docker compose up -d` Ghost uses this value for canonical URLs and sitemap generation. After redeploying, I verified that `https://anantafatur.dev/sitemap.xml` contained only root-domain URLs. I also checked the page source of a blog post to confirm that the `` tag pointed to the new domain. No database changes were required. Ghost handled the domain switch cleanly as long as `GHOST_URL` was correct. ## Updating cloudflared Tunnel The Cloudflare Tunnel already existed for the subdomain, pointing to `http://localhost:8088`. I reused the same tunnel and simply added a new public hostname entry for `anantafatur.dev`, pointing to the same backend address. ![](https://anantafatur.dev/content/images/2026/02/image-14.png) Cloudflare - Published application routes This means: - Root domain now serves Ghost through the tunnel. - The redirect is handled at the Cloudflare edge before traffic is forwarded to the tunnel, so the backend does not receive requests for the old subdomain. No changes were required to container networking or service ports. ## Verifying Redirect Behavior Before touching Google Search Console, I verified everything at HTTP level. I ran: `curl -I https://docs.anantafatur.dev/devops-job-hunting-flow-a-sankey-analysis/` ![](https://anantafatur.dev/content/images/2026/02/image-15.png) cURL Response The response returned `HTTP/2 301` with a `location` header pointing to the correct root-domain URL, and `server: cloudflare`, confirming the redirect happens at the edge. Downtime during the switch was around one minute, mostly during container restart. From observation, redirect response is effectively immediate, but no formal measurement was taken. ## Updating Google Search Console In the old property for `docs.anantafatur.dev`, I used the Change of Address tool to notify Google of the migration to `anantafatur.dev`. After that, I submitted the new sitemap under the root-domain property. I also used URL Inspection on a few old URLs to confirm Google sees the 301 response. After that, I left it alone. No forced reindexing or aggressive resubmission. ## Post-Migration Observations After the migration, old URLs consistently returned 301 responses, new URLs loaded correctly, canonical tags pointed to the root domain, and the sitemap contained only the new domain. I have not measured ranking changes or traffic differences yet. From a technical standpoint, the migration follows expected best practices. How Google adjusts ranking will depend on its crawling and indexing timeline. The entire process took under ten minutes of actual work, with about one minute of downtime. And honestly, having the blog directly on the root domain just feels cleaner. ### Ghost CMS 6.19.1 Upgrade [SQLi Patch CVE-2026-26980] URL: https://anantafatur.dev/ghost-cms-6-19-1-upgrade-sqli-patch-cve-2026-26980/ Last updated: 2026-02-22T16:36:24.000Z One morning I got an alarming email from Ghost: my site was vulnerable to a SQL injection in the Content API (CVE-2026-26980). Basically, anyone could read arbitrary database data with the public API key. The patch was out in `6.19.1`, so the first step was obvious — upgrade from `ghost:6.10.3-alpine3.23` to `ghost:6.19.1-alpine3.23`. MariaDB stayed on `10.6.24-jammy`. I knew this wouldn’t be just a “docker pull and go” situation, the database already had some custom tables, so migrations could easily break. ![](https://anantafatur.dev/content/images/2026/02/image-4.png) Ghost Security Issue For reference, the official Ghost security advisory is here: https://github.com/TryGhost/Ghost/security/advisories/GHSA-w52v-v783-gw97 ### Step 1 — Backup Everything Before touching anything, I backed up the Ghost content and the database. Always save yourself from a potential disaster. **Backup Ghost content:** ```bash docker cp ghost-cms:/var/lib/ghost/content /tmp/ghost-content-backup ``` **Backup MariaDB:** ```bash docker exec ghost-mariadb mysqldump -uroot -p ghost > /tmp/ghost-db-backup.sql ``` Good. Content safe. Database safe. Heartbeat slightly steadier. ### Step 2 — First Attempt at Upgrade Updated `docker-compose.yml` to: ```yaml #image: ghost:6.10.3-alpine3.23 image: ghost:6.19.1-alpine3.23 ``` Then did the usual: ```bash docker compose down && docker compose up ``` MariaDB started fine. Ghost… failed immediately. Logs showed this horror: ``` Can't create table `ghost`.`automated_email_recipients` (errno: 150 "Foreign key constraint is incorrectly formed") ``` Ah yes, the infamous FK error. I knew this looked familiar: Ghost migrations assume numeric IDs for some tables, but in my case, `automated_emails.id` was `varchar(24)`. Not compatible. Checked it just to confirm: ```sql SHOW CREATE TABLE automated_emails; ``` Yep. `id` was varchar. No wonder MySQL/MariaDB refused the foreign key. ### Why I Call This a "Custom Table" When I say "custom table", I don’t mean a plugin added it or Ghost feature magic. I mean the schema in the database didn’t match what Ghost expected for migrations. This happened because I previously migrated from MySQL (`mysql:8.0-bookworm`) to MariaDB(`mariadb:10.6.24-jammy`). That migration subtly changed some column definitions, like `automated_emails.id` becoming `VARCHAR(24)` instead of a numeric type. Ghost migrations tried to make `automated_email_recipients.automated_email_id` a `BIGINT UNSIGNED` and reference `automated_emails.id`, and MariaDB refused, hence the foreign key error. Essentially, the table is “custom” because its schema diverged from the default Ghost schema due to a previous migration. You can read more about that MySQL → MariaDB journey here: ### Step 3 — Manual Fix Then I created `automated_email_recipients` manually with the correct column types to satisfy the foreign key constraint. ```sql CREATE TABLE automated_email_recipients ( id VARCHAR(24) NOT NULL, automated_email_id VARCHAR(24) NOT NULL, recipient_email VARCHAR(191) NOT NULL, created_at DATETIME NOT NULL, updated_at DATETIME DEFAULT NULL, PRIMARY KEY (id), KEY automated_email_id_index (automated_email_id), CONSTRAINT automated_email_recipients_automated_email_id_foreign FOREIGN KEY (automated_email_id) REFERENCES automated_emails (id) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; ``` Restarted Ghost container, and finally migrations completed without a single error. Site came up. Sweet relief. ### Step 4 — Cloudflare WAF Double Protection ![](https://anantafatur.dev/content/images/2026/02/image-3.png) Cloudflare WAF Even though Ghost 6.19.1 patched the SQLi, I wanted extra insurance. Because the Content API key is public by design, restricting access does nothing. So I set up a Cloudflare WAF rule to block obvious malicious queries: ```text (http.request.uri.query contains "slug%3A%5B") or (http.request.uri.query contains "slug:[") ``` Action: **Block**, then saved & deployed. Some legitimate slug filters may break, but I accepted that risk until everything stabilized. > All patched up, database happy, and the WAF doing its thing. Not the smoothest ride, but it worked, and that’s what matte ### DevOps Job Hunting Flow — A Sankey Analysis URL: https://anantafatur.dev/devops-job-hunting-flow-a-sankey-analysis/ Last updated: 2026-02-15T16:37:22.000Z This is a breakdown of my job search process while still an active student, before officially graduating in January 2026. At that time I was interning full-time 9–17, had around one year of DevOps-related intern and work experience, and was not under pressure to accept any offer immediately. My initial goal was not to get hired right away but to validate CV positioning and understand market expectations. Throughout December I iterated on my CV whenever I saw weak signals from applications. I used a single ATS-friendly version because I only applied to DevOps-related roles such as DevOps, Cloud, Infra, or SRE. All data below is from LinkedIn applications. I also applied through Glints and Dealls but got zero responses there so they are excluded. > Total applications: 112. I visualized the application flow using a Sankey diagram to show all paths from Applied to final outcomes, including ghosted, withdrawn, rejected, declined, and accepted. This shows not just overall movement but the distribution of outcomes at each stage. ![](https://anantafatur.dev/content/images/2026/02/sankeymatic_20260215_231406_2000x1200.png) Made at SankeyMATIC.com ## Stage 1 — Applied (112 → 18 HR) Out of 112 applications 94 ghosted and 18 moved to HR interview, about 84% ghost rate. I did not tailor CVs per company because I only applied to DevOps-related roles. I did not care about location. Onsite, hybrid, shift, no shift, I still applied because at this stage it was more about testing signal strength than filtering based on location or work arrangement. > Most drop happened here. Observation: automated screening and experience requirements were the main bottlenecks. Many roles asked for 1–3 years experience and being labeled a fresh graduate probably hurt visibility. From this stage it was clear visibility mattered more than technical ability. ## Stage 2 — HR Interview (18 → 10 Assessment) Out of 18 HR interviews 10 progressed to assessment, 5 I withdrew, and 3 ghosted, conversion about 55%. HR questions were mostly about conflict resolution in previous teams, not tools or technical depth. When asked about experience I referenced my internship and prior work, showing roughly one year of practical DevOps exposure. Salary expectation was 10–20% above minimum wage, which I could afford because I was not in urgent need of income. Withdrawals were due to already signing another contract, roles that were mislabeled as DevOps but mostly web development, in-person requirements in Jakarta, and scheduling issues with no follow-up. > Observation: this stage filtered for alignment more than technical capability. ## Stage 3 — Assessment (10 → 8 User Interview) Assessment included psychotests and take-home technical tests. 8 moved to user interview, 2 I withdrew, giving about 80% conversion. Technical scope included CI/CD pipelines, Docker, Kubernetes, Ansible, Terraform, VPN setup, and cloud architecture scenarios. I built everything from scratch, no template reuse. Each assessment took around 3–4 days because I was still interning full-time so most work was done at night or weekends. > Observation: execution quality mattered more than resume wording. ## Stage 4 — User Interview (8 → 2 Offer) Out of 8 user interviews 2 offers, 4 withdrawals, 2 rejections, about 25% conversion. Interviews were with future supervisors and focused on tool understanding, when to use which tool, troubleshooting scenarios, hypothetical “what if” situations, and deep dives into past projects. Withdrawals were mostly strategic because of in-person requirements, red-flag signals, or already accepting another contract. > Observation: evaluation was mutual, both sides assessing fit. ## Stage 5 — Offer (2 → 1 Accepted) The first offer came from a small city in Central Java, salary roughly 180% of minimum wage. Financially attractive but growth environment limited, exposure scale small, so I rejected it after clear communication with HR. The second offer was from a large telco company with hybrid setup, salary slightly below expectation. Factors that led to acceptance included exposure to corporate-scale infrastructure, learning opportunities, and fundamental skill strengthening. > Also observation: 112 applications to 2 offers is about 1.8% offer rate, so rejecting both would increase uncertainty. > > Personal factor: wanted to avoid visible gap after graduation month, preferred immediate transition rather than staying as intern post-graduation. ## Observations The diagram shows that most drop happened at the cold application stage. Ghosting was high but expected due to automated screening and experience mismatch. Once candidates passed HR and assessments, conversion rates were strong, indicating that technical ability was sufficient. Withdrawals at HR, assessment, and user interviews were mostly strategic alignment decisions based on role, location, or work arrangement rather than outright rejection. The process felt iterative: apply, observe patterns, adjust CV, refine positioning, repeat. External constraints like company requirements, in-person interview requirements, and schedule alignment had a significant impact on outcomes. > A few practical notes from my experience: keep applying even if you don’t care much about location, role type, or schedule. The more you apply, the more interviews you get, which helps you spot weaknesses in your CV and improve it over time. Frequent interviews also make you stronger at answering questions and explaining experience. Even if you get an offer, it’s fine to decline if it doesn’t fit your priorities. By doing this, you get a clear signal of your market value and can be confident that your skills and experience meet industry standards. ### Kubernetes Logging with OpenObserve on K3s URL: https://anantafatur.dev/kubernetes-logging-with-openobserve-on-k3s/ Last updated: 2026-02-24T13:49:12.000Z Last month, my senior asked me to explore Kubernetes logging using the ELK or EFK stack. I did try, but I struggled quite a bit. Not because ELK is bad, but because the ecosystem is wide and deep, and my own skill level wasn’t there yet. I spent more time figuring out components and configs than actually looking at logs. When the task was followed up again this week, I revisited the problem. While researching alternatives, I came back to OpenObserve. I had actually bookmarked it a long time ago when searching for “New Relic alternative”, but never really had time to explore it properly. This time, I did. This post documents that exploration. It’s not a benchmark and not a production claim. Just what I ran, why I chose it, and what I observed. ## Environment and Goal The setup ran on a **multi-node k3s cluster**. The whole exploration took about **half a day**. Logs and Kubernetes events were collected using default OpenTelemetry behavior. OpenObserve was exposed using **NodePort** for simplicity. For staging or production, **Ingress** would make more sense, but here the goal was to reduce moving parts and get visibility quickly. ## Architecture Overview The data flow is simple. Kubernetes sends logs and events to the OpenTelemetry Collector. The collector forwards them to OpenObserve. OpenObserve stores logs in **Amazon S3** and metadata in **PostgreSQL**. Inside the cluster, OpenObserve runs **Ingester, Querier, Alertmanager, and NATS**. Externally, it depends only on S3 for log storage and **RDS PostgreSQL** for metadata. The bundled PostgreSQL from the Helm chart was disabled. ## Storage Decisions Logs are stored in S3\. Early on, I noticed many objects were **smaller than 128 KB**. This matters when thinking about Glacier, because Glacier has a **128 KB minimum billable object size**. Moving small log objects there can increase cost instead of reducing it. ![](https://anantafatur.dev/content/images/2026/02/image.png) aws s3 ls s3://poc-openobserve --recursive --human-readable --summarize Because of that, Glacier lifecycle rules were intentionally deferred. This needs longer observation time. Even with S3, OpenObserve still needs **PVCs for caching and buffering**. Default sizes from the Helm chart were kept. Shrinking them too early usually creates more confusion than insight. ![](https://anantafatur.dev/content/images/2026/02/image-1.png) k get pvc -n openobserve ## Deploying OpenObserve First, I pulled the default Helm values and edited them locally. ```bash curl https://raw.githubusercontent.com/openobserve/openobserve-helm-chart/main/charts/openobserve/values.yaml -o values.yaml ``` S3 credentials were configured in `values.yaml`. ```yaml ZO_S3_ACCESS_KEY: "AKIAIOSFODNN7EXAMPLE" ZO_S3_SECRET_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" ZO_S3_BUCKET_NAME: "openobserve-logs" ZO_S3_REGION_NAME: "ap-southeast-1" ``` Metadata storage was switched to PostgreSQL, and the bundled PostgreSQL was disabled. ```yaml config: ZO_META_STORE: postgres auth: ZO_META_POSTGRES_DSN: "postgres://postgres:password@rds-endpoint:5432/openobserve" postgres: enabled: false ``` Then OpenObserve was installed using Helm. ```bash helm repo add openobserve https://charts.openobserve.ai helm repo update kubectl create namespace openobserve helm install o2 openobserve/openobserve \ -n openobserve \ -f values.yaml ``` For access, I exposed the router service using NodePort. ```bash kubectl -n openobserve edit svc o2-openobserve-router ``` Change `spec.type` to `NodePort`. ## Connecting Kubernetes Logs Logs and events were collected using OpenTelemetry. This part has some dependencies that are easy to miss. First, cert-manager: ```bash kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.0/cert-manager.yaml ``` Then Prometheus Operator CRDs, required by the OpenTelemetry operator even if Prometheus itself is not installed. ```bash kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/heads/main/example/prometheus-operator-crd/monitoring.coreos.com_scrapeconfigs.yaml kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/heads/main/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml ``` After that, the OpenTelemetry operator was installed. ```bash kubectl apply -f https://raw.githubusercontent.com/openobserve/openobserve-helm-chart/refs/heads/main/opentelemetry-operator.yaml ``` Then the OpenObserve collector was deployed in a separate namespace. ```bash kubectl create namespace openobserve-collector ``` ```bash helm upgrade --install o2c openobserve/openobserve-collector \ -n openobserve-collector \ --set k8sCluster=cluster1 \ --set exporters.'otlphttp/openobserve'.endpoint=http://o2-openobserve-router.openobserve.svc.cluster.local:5080/api/default \ --set exporters.'otlphttp/openobserve'.headers.Authorization='Basic cm9vdEBleGFtcGxlLmNvbTpCSXVETFZwdnNXbWZyMW9U' \ --set exporters.'otlphttp/openobserve_k8s_events'.endpoint=http://o2-openobserve-router.openobserve.svc.cluster.local:5080/api/default \ --set exporters.'otlphttp/openobserve_k8s_events'.headers.Authorization='Basic cm9vdEBleGFtcGxlLmNvbTpCSXVETFZwdnNXbWZyMW9U' ``` One important gotcha: if the Authorization header is wrong, logs are silently dropped. No errors, just empty dashboards. ## Closing Thoughts Compared to my earlier ELK and EFK exploration, OpenObserve was easier for me to reason about. Fewer components to wire together, fewer decisions before seeing value. ELK is powerful, but for me, the ecosystem complexity combined with my own skill gap slowed things down. With OpenObserve, I had logs flowing and dashboards visible within hours. There are still open questions, especially around long-term S3 object sizing and lifecycle policies. But OpenObserve is no longer just a bookmarked link. It’s now a concrete option that I’ve actually run and understood. ### Traefik Proxy for Docker Containers with Cloudflare Full Strict TLS URL: https://anantafatur.dev/traefik-proxy-for-docker-containers-with-cloudflare-full-strict-tls/ Last updated: 2026-01-23T09:54:35.000Z My senior asked me to explore proxying Docker containers using Traefik. At that time, there was no existing setup yet, so everything had to start from zero. I also didn’t really ask too many questions at first. Turns out, a few simple apps were still running on Kubernetes, and that’s the main reason he wanted to migrate them to plain Docker instead. He had already tried Nginx Proxy Manager before, so my task was to explore Traefik and Kong. Personally, for simple container setups, I usually just use Cloudflare Tunnel. It’s fast, less things to configure, and usually just works. If tunnel cannot work, then I fall back to plain Nginx. The Nginx approach is simple and works fine for personal projects. Spin up, proxy, okay already. But production-grade? Not really lah. Below is an example of the kind of Nginx setup I often use when I just want something up quickly. ```yaml version: "3.9" services: nginx: image: nginx:stable-alpine container_name: nginx-proxy network_mode: host volumes: - ./conf.d:/etc/nginx/conf.d:ro restart: always ``` ```nginx server { listen 80; server_name api.anantafatur.dev; location / { proxy_pass http://127.0.0.1:9001; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ``` --- ## PoC Goal This PoC demonstrates routing Docker containers using Traefik with Cloudflare Full Strict TLS. Traefik was chosen simply because that was the task assigned. The goal here is not benchmarking or performance tuning. I’m not trying to squeeze every last bit of CPU. The goal is very straightforward: make sure routing works properly, TLS terminates correctly, and nothing weird happens along the way. Once this part is confirmed, only then it makes sense to go further. --- ## Traefik Setup For this PoC, MinIO was used as the example application. Nothing special here, but realistic enough to test real-world routing and TLS termination. Traefik runs inside Docker and handles Cloudflare Full Strict TLS using an origin certificate. A dedicated Docker network is used so Traefik and application containers can talk to each other internally. This way, not every container needs to expose ports directly to the host, which keeps things cleaner. ### Folder Structure ```text traefik/ ├── docker-compose.yml ├── traefik.yml ├── dynamic.yml ├── certs/ ├── origin.crt └── origin.key ``` Folder Structure Configuration is split into static (`traefik.yml`) and dynamic (`dynamic.yml`) files. Slightly more setup work at the start, but once done, it’s easier to manage when things grow bigger. ### Docker Network and Traefik Compose ```bash docker network create traefik-net ``` ```yaml version: "3.9" services: traefik: image: traefik:v3.6.7 container_name: traefik restart: unless-stopped networks: - traefik-net ports: - "80:80" - "443:443" - "127.0.0.1:8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - ./certs:/certs:ro - ./traefik.yml:/traefik.yml:ro - ./dynamic.yml:/dynamic.yml:ro command: - "--configFile=/traefik.yml" networks: traefik-net: external: true ``` docker-compose.yaml --- ## Traefik Configuration The `certs/origin.crt` and `certs/origin.key` files are Cloudflare Origin Certificates, which you can generate and download from [Cloudflare Origin CA](https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/#deploy-an-origin-ca-certificate). Traefik’s configuration is split into **static** and **dynamic** files: the dynamic file points Traefik to these certificates and enforces a minimum TLS version, while the static file sets up entry points, the dashboard, and Docker/file providers so Traefik can watch containers and TLS settings. ```yaml tls: certificates: - certFile: /certs/origin.crt keyFile: /certs/origin.key options: default: minVersion: VersionTLS12 ``` dynamic.yaml ```yaml log: level: INFO accessLog: {} api: dashboard: insecure: true entryPoints: web: address: ":80" http: redirections: entryPoint: to: websecure scheme: https websecure: address: ":443" providers: docker: exposedByDefault: false network: traefik-net file: filename: /dynamic.yml watch: true ``` traefik.yml --- ## MinIO Setup For this PoC, MinIO was used to test Traefik routing. MinIO exposes two ports: `9001` for the console and `9000` for the S3 API. Traefik routes traffic based on host rules. Console traffic goes to one hostname, API traffic goes to another. Very clear separation, easy to reason about when something cannot work. There’s also a helper container that initialises buckets and access keys after a short delay. This is just to make sure MinIO is fully up before running any setup commands. Data persists using Docker volumes, and credentials are set using environment variables. ```yaml version: "3.9" services: minio: image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z container_name: minio restart: always networks: - traefik-net environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: supersecret command: ["server", "--console-address", ":9001", "/data"] labels: - "traefik.enable=true" # MinIO Console - "traefik.http.routers.minio-console.rule=Host(`s3.anantafatur.dev`)" - "traefik.http.routers.minio-console.entrypoints=websecure" - "traefik.http.routers.minio-console.tls=true" - "traefik.http.routers.minio-console.service=minio-console-svc" - "traefik.http.services.minio-console-svc.loadbalancer.server.port=9001" # MinIO API - "traefik.http.routers.minio-api.rule=Host(`s3-api.anantafatur.dev`)" - "traefik.http.routers.minio-api.entrypoints=websecure" - "traefik.http.routers.minio-api.tls=true" - "traefik.http.routers.minio-api.service=minio-api-svc" - "traefik.http.services.minio-api-svc.loadbalancer.server.port=9000" minio-buckets: image: quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z container_name: minio-buckets depends_on: - minio networks: - traefik-net entrypoint: > /bin/sh -c " sleep 5; mc alias set localminio http://minio:9000 minioadmin supersecret; mc mb localminio/mybucket || true; mc admin accesskey create localminio/ --access-key accesskey123 --secret-key secretkey1234567890 || true; " networks: traefik-net: external: true ``` docker-compose.yaml --- ## Architecture Diagram ``` ┌─────────────────────┐ │ Cloudflare CDN │ │ (Full Strict TLS) │ └─────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Traefik │ │ Docker Container │ │ EntryPoints: 80/443│ └─────┬─────────┬─────┘ │ │ ┌───────▼──────┐ │ │ MinIO Console│ │ │ port 9001 │ │ └──────────────┘ │ │ ┌───────▼───────┐ │ MinIO API │ │ port 9000 │ └───────────────┘ ``` Traefik terminates Cloudflare TLS using the Cloudflare origin certificate. All traffic to MinIO is routed internally over the `traefik-net` Docker network. Simple flow, no magic. --- ## Resource Usage (Idle State) The following Docker stats were captured while the environment was idle, with no active traffic routed through Traefik. This is included only to show baseline resource usage, not as a performance benchmark. ``` NAME CPU % MEM USAGE / LIMIT MEM % minio 0.02% 84.37MiB / 1.866GiB 4.42% traefik 0.00% 17.15MiB / 1.866GiB 0.90% ``` docker stats At idle, Traefik uses very little CPU and memory. Nothing shocking here. Which is good. Means it’s not burning resources while just sitting there doing nothing. --- ## Conclusion So far, this PoC works as expected. Routing is correct, TLS terminates cleanly with Cloudflare Full Strict mode, and everything stays neatly inside the Docker network. No benchmarking or load testing done yet. That one can come later. For now, this setup is purely to confirm that Traefik can do its job without giving unnecessary headache. For that purpose, it’s good enough already. ### Monitoring Home Network Latency using SmokePing with Docker Compose URL: https://anantafatur.dev/monitoring-home-network-latency-using-smokeping-with-docker-compose/ Last updated: 2026-01-20T04:16:24.000Z After moving back home, I noticed my home internet was inconsistent. Speeds sometimes hit \~100 Mbps, sometimes dropped to \~3 Mbps. Latency also felt unpredictable. I had no historical data, so I couldn’t tell whether the problem was the ISP, Wi-Fi interference, or devices — only that it felt unstable. To reduce local overhead, I added a new router, Ruijie RG-EW1200G PRO, behind the ISP ONT (Huawei EG8145V5) and disabled the ONT Wi-Fi. All home devices — laptops, phones, etc. — connect through the Ruijie AP. The MikroTik router is used only for the Proxmox homelab, isolating it from normal home traffic. There’s no QoS, traffic shaping, or advanced routing applied. The homelab runs on a Dell OptiPlex with Proxmox. For visibility, I deployed SmokePing to monitor latency over time. I’m not comparing before/after or drawing conclusions; the goal is purely observation. SmokePing runs in a Docker container inside an LXC on Proxmox. I chose the linuxserver/smokeping image because it comes with all monitoring probes preconfigured by default. I considered Uptime Kuma as a second option, but it requires manually adding each target. Here’s the Docker Compose I used: ``` services: smokeping: image: lscr.io/linuxserver/smokeping:2.9.0 container_name: smokeping hostname: prod-monitoring environment: - PUID=1000 - PGID=1000 - TZ=Asia/Jakarta - CACHE_DIR=/tmp volumes: - /root/smokeping/config:/config - /root/smokeping/data:/data ports: - 80:80 restart: unless-stopped ``` Everything runs with default settings: probes, intervals, and targets are unchanged. SmokePing has only been running for about a day, so there’s no trend or analysis yet. The data is for personal monitoring — to have a baseline reference and see connectivity patterns over time. Here’s a simple view of the network layout: ```yaml [ISP ONT - Huawei EG8145V5] │ (WAN) ▼ [Ruijie RG-EW1200G PRO Router + AP] │ ├─> [Home Devices / Laptop / Phone] │ └─> [MikroTik → Proxmox Homelab] │ ▼ [LXC → Docker → SmokePing] ``` Home devices connect through the Ruijie AP while MikroTik isolates the Proxmox homelab running SmokePing for latency monitoring Here’s a look at the SmokePing dashboard after a day of running: ![](https://anantafatur.dev/content/images/2026/01/image-2.png) SmokePing dashboard showing continuous latency to selected Internet sites over time ![](https://anantafatur.dev/content/images/2026/01/image-3.png) SmokePing dashboard showing latency to DNS servers, helping observe DNS resolution stability And here are the resource stats for the SmokePing container on Proxmox: ``` NAME CPU % MEM USAGE / LIMIT MEM % smokeping 0.01% 122.3 MiB / 2 GiB 5.97% ``` SmokePing container stats from docker stats: very low CPU and memory usage In conclusion, the setup demonstrates a low-overhead method to continuously monitor home network latency using SmokePing in Docker on Proxmox, with the Ruijie RG-EW1200G PRO handling routing and Wi-Fi, and MikroTik isolating the homelab; key takeaways are that preconfigured probes simplify deployment, resource usage is minimal (\~0.01% CPU, \~123 MiB RAM), and continuous ICMP monitoring provides baseline visibility without impacting home devices; future work includes bridging the ONT for full end-to-end visibility, adding internal hop targets, correlating latency with time-of-day usage, and longer-term trend analysis once sufficient data is collected. ### How I Saved ~77% Docker Memory by Migrating MySQL to MariaDB URL: https://anantafatur.dev/how-i-saved-77-docker-memory-by-migrating-mysql-to-mariadb/ Last updated: 2026-01-06T20:14:13.000Z > This is not a benchmark and not a recommendation for high-traffic workloads. All numbers here are real observations from a low-resource VPS running Ghost CMS. I’m running Ghost CMS on a very small VPS: 1 vCPU, 2 GB RAM, zero swap because it’s OpenVZ. At some point the server was already sitting at around 1.75 GB RAM usage, and I still needed to host another PHP + MariaDB app for a college assignment. So yeah, memory was getting dangerously tight. To understand what was eating my RAM, I checked the system using `ps aux --sort=-%mem | head` and `docker stats`. One process immediately stood out: ``` 30215 0.4 19.7 1304564 404200 ? Ssl 01:45 0:05 mysqld ``` Docker confirmed it. The Ghost MySQL container alone was consuming more than 400 MB of memory, roughly 20% of my total RAM. ``` NAME CPU % MEM USAGE / LIMIT MEM % ghost-cms 0.03% 168.8 MiB / 1.953 GiB 8.44% ghost-mysql 0.41% 405.0 MiB / 1.953 GiB 20.25% ``` For a personal blog, this was ridiculous. I don’t care about database performance here. This is not a high-traffic production system. It’s just Ghost. So the goal was very simple: **make the Ghost database container use as little memory as possible**. After some research, I realized that there is no official Alpine-based image for either MySQL or MariaDB. Both projects no longer publish Alpine variants on Docker Hub, so using an ultra-minimal base image was not an option here. Instead, I started comparing the available MariaDB tags directly on Docker Hub, focusing on compressed image size as a rough indicator of baseline resource usage. After scrolling through the tags, I found `mariadb:10.6.24-jammy`, which stood out as one of the smallest stable options, with a compressed size of 98.45 MB. For comparison, the commonly used `mariadb:10.6.24-ubi9` image weighs around 153.7 MB, which is significantly larger. The migration itself was intentionally simple. No Kubernetes, no replicas, no fancy stuff — just plain Docker. The downtime was **around one minute**, and it actually happened, not an estimate. I prepared the new database container first, completed the data import, and only then updated Ghost’s database configuration to point to the new MariaDB instance. Once the Ghost container was restarted, the site was back online almost immediately. --- First, this was my original Docker Compose setup using MySQL: ```yaml version: "3.9" services: ghost: image: ghost:6.10.3-alpine3.23 container_name: ghost-cms restart: always ports: - 8088:2368 environment: database__client: mysql database__connection__host: db database__connection__user: ${MYSQL_USER} database__connection__password: ${MYSQL_PASSWORD} database__connection__database: ${MYSQL_DATABASE} url: ${GHOST_URL} volumes: - ghost:/var/lib/ghost/content depends_on: - db db: image: mysql:8.0-bookworm container_name: ghost-mysql restart: always environment: MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} MYSQL_DATABASE: ${MYSQL_DATABASE} MYSQL_USER: ${MYSQL_USER} MYSQL_PASSWORD: ${MYSQL_PASSWORD} volumes: - db:/var/lib/mysql volumes: ghost: db: ``` The first step was adding a new MariaDB service **without touching the existing MySQL container**, so rollback was always possible. ```yaml mariadb: image: mariadb:10.6.24-jammy container_name: ghost-mariadb restart: always environment: MARIADB_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} MARIADB_DATABASE: ${MYSQL_DATABASE} MARIADB_USER: ${MYSQL_USER} MARIADB_PASSWORD: ${MYSQL_PASSWORD} volumes: - mariadb:/var/lib/mysql - ./my.cnf:/etc/mysql/conf.d/my.cnf:ro ``` Before migrating anything, I exported the database from the old MySQL container. I already had a daily backup script, so I reused it: ```bash #!/bin/bash DATE=$(date +%F_%H-%M) DIR=/root/ghost/db-backups docker exec ghost-mysql \ mysqldump -u root -p'PASSWORD' ghost \ > ${DIR}/ghost-${DATE}.sql ``` Because my VPS only has 2 GB RAM and I genuinely don’t care about database performance, I aggressively tuned MariaDB memory usage using a custom `my.cnf`: ```ini [mariadb] character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci innodb_buffer_pool_size=64M innodb_log_buffer_size=8M max_connections=20 performance_schema=OFF skip-name-resolve ``` Yes, 64 MB buffer pool. And yes, it’s perfectly fine for a blog. When importing the SQL dump into MariaDB, the first error I hit was: ``` ERROR 1046 (3D000): No database selected ``` That happened because the dump didn’t include a `USE ghost;` statement. The fix was simply specifying the database explicitly: ```bash docker exec -i ghost-mariadb \ mysql -u root -pPASSWORD ghost \ < ghost-2026-01-07_00-00.sql ``` Immediately after that, another error appeared: ``` ERROR 1273 (HY000): Unknown collation 'utf8mb4_unicode_ci' ``` This was expected. MySQL 8 uses `utf8mb4_unicode_ci`, which MariaDB does not support. I fixed it by replacing the collation inside the SQL file: ```bash sed -i \ 's/utf8mb4_unicode_ci/utf8mb4_unicode_ci/g' \ ghost-2026-01-07_00-00.sql ``` Because the import had already failed halfway, I dropped and recreated the database to start clean: ```bash docker exec -it ghost-mariadb \ mysql -u root -pPASSWORD \ -e "DROP DATABASE ghost; CREATE DATABASE ghost CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" ``` Then I re-imported the SQL dump: ```bash docker exec -i ghost-mariadb \ mysql -u root -pPASSWORD ghost \ < ghost-2026-01-07_00-00.sql ``` Once the database was successfully imported, the last step was pointing Ghost to the new MariaDB container. I changed this line in `docker-compose.yml`: ```yaml database__connection__host: db ``` to: ```yaml database__connection__host: mariadb ``` After restarting Ghost, the site loaded perfectly with no issues. Only after confirming everything was stable did I delete the old MySQL container and volume. Here’s the final Docker Compose setup: ```yaml version: "3.9" services: ghost: image: ghost:6.10.3-alpine3.23 container_name: ghost-cms restart: always ports: - 8088:2368 environment: database__client: mysql database__connection__host: mariadb database__connection__user: ${MYSQL_USER} database__connection__password: ${MYSQL_PASSWORD} database__connection__database: ${MYSQL_DATABASE} url: ${GHOST_URL} volumes: - ghost:/var/lib/ghost/content depends_on: - mariadb mariadb: image: mariadb:10.6.24-jammy container_name: ghost-mariadb restart: always environment: MARIADB_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} MARIADB_DATABASE: ${MYSQL_DATABASE} MARIADB_USER: ${MYSQL_USER} MARIADB_PASSWORD: ${MYSQL_PASSWORD} volumes: - mariadb:/var/lib/mysql - ./my.cnf:/etc/mysql/conf.d/my.cnf:ro volumes: ghost: mariadb: ``` The result was honestly better than I expected. The MariaDB container now sits at around 93 MB of RAM, while Ghost itself uses around 167 MB. ``` NAME CPU % MEM USAGE / LIMIT MEM % ghost-cms 0.01% 167.8 MiB / 1.953 GiB 8.39% ghost-mariadb 0.01% 93.21MiB / 1.953 GiB 4.66% ``` Compared to the original \~405 MB MySQL usage, that’s roughly an **77% reduction in database memory usage**, achieved purely by switching to MariaDB and tuning it aggressively. ![](https://anantafatur.dev/content/images/2026/01/image-1.png) Finally, the impact was also clearly visible from my monitoring system. Right after the ghost-mysql container was stopped, there was a sharp and immediate drop in overall RAM usage, with the memory graph showing a steep decline that confirmed MySQL as the biggest memory consumer on this VPS. There are no benchmarks and no performance claims here — this setup isn’t designed for high-traffic workloads, and that’s completely fine. The only goal was to keep Ghost running comfortably on a tiny VPS, and for that purpose, it worked perfectly. ### Backing Up Azure Managed Disks to Blob Storage with S3 Replication URL: https://anantafatur.dev/backing-up-azure-managed-disks-to-blob-storage-with-s3-replication/ Last updated: 2026-01-03T08:25:16.000Z A few days ago, my senior asked me to check how **Azure Managed Disks** could be backed up to **Azure Blob Storage** and to write the steps clearly. There was no requirement for migration, no architectural redesign, and no Azure Backup policies involved. The scope was intentionally small: disk-level backup only. Instead of stopping at documentation or copying commands from the Azure docs, I wanted to validate that the process actually worked end to end. I ran every step myself, verified the outputs, and automated the parts that made sense. This post documents that process from start to finish. --- ## The Problem, Reduced to Its Core Once stripped of context and tooling preferences, the problem became very simple. A managed disk exists inside Azure, the backup should not be tied to the VM lifecycle, and the backup should live in object storage. That naturally leads to a straightforward flow: ``` Azure Managed Disk → Snapshot → Azure Blob Storage ``` Snapshots provide a crash-consistent copy of the disk, and Blob Storage provides durable and inexpensive object storage. Nothing else is required to satisfy the task. --- ## Why Disk-Level Backup Was the Right Choice The task explicitly focused on managed disks rather than virtual machines or Azure Backup policies. Working at the disk level keeps the process simple and predictable. Disk-level backups work for both OS disks and data disks, produce portable VHD files, and avoid introducing long-running backup services or policy management. For ad-hoc or one-time backups, this level of abstraction is easier to reason about and easier to document. --- ## Making the Environment Reproducible Before touching the backup logic, I created a small lab environment using Terraform. The environment includes a lightweight Linux VM, an OS disk, an attached managed data disk, and a storage account with a private Blob container. Nothing is oversized and everything is disposable. The goal was not to build realistic infrastructure, but to eliminate guesswork. With a consistent environment, the backup flow can be validated properly and repeated without manual setup. All Terraform and automation scripts are available here: --- ## Discovering Managed Disks Automatically Instead of hardcoding disk names, the backup script dynamically discovers all managed disks in the target resource group. ```bash az disk list \ --resource-group rg-vm-backup-poc \ --query "[].name" -o tsv ``` This mirrors how the script would behave in a real environment. Any disk that exists in the resource group is included automatically, whether it is an OS disk or a data disk. No manual tracking or updates are required as disks are added or removed. --- ## Creating Snapshots (The Actual Backup Step) For each discovered disk, the script creates a snapshot. ```bash az snapshot create \ --resource-group "$SOURCE_RG" \ --name "$SNAPSHOT_NAME" \ --source "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$SOURCE_RG/providers/Microsoft.Compute/disks/$DISK" \ --sku Standard_LRS \ --incremental false \ --tags backup=true type=crash-consistent source_disk="$DISK" ``` This is the moment where the disk state is captured. Snapshots are fast to create, relatively inexpensive, and provide crash-consistent copies, which are sufficient for most disk-level backup use cases. At this stage, Azure already holds a safe copy of the disk, but the snapshot still lives inside Azure’s managed disk ecosystem. --- ## Granting Temporary Access to the Snapshot Snapshots are private by default, so Azure requires a temporary SAS token to export them. ```bash az snapshot grant-access \ --resource-group "$SOURCE_RG" \ --name "$SNAPSHOT_NAME" \ --duration-in-seconds "$SNAPSHOT_TTL" \ --access-level Read \ --query accessSAS -o tsv ``` This command grants read-only access for a limited duration. Once the SAS token expires, the snapshot becomes inaccessible again. This keeps the export process secure by ensuring that access exists only for as long as it is needed. --- ## Exporting the Snapshot to Azure Blob Storage With the SAS URI available, the snapshot can be copied directly into Azure Blob Storage as a VHD file. ```bash az storage blob copy start \ --account-name "$TARGET_STORAGE" \ --account-key "$STORAGE_KEY" \ --destination-container "$TARGET_CONTAINER" \ --destination-blob "$BLOB_NAME" \ --source-uri "$SNAPSHOT_URI" ``` After this step completes, the backup is no longer tied to the virtual machine or the snapshot lifecycle. It exists as a plain object in Blob Storage, which means it can be retained, replicated, or deleted independently of the original disk. At this point, the original requirement is already fulfilled. --- ## Optional: Replicating the Backup to Amazon S3 The task mentioned comparing Blob Storage and S3 pricing, but did not require cross-cloud replication. Since the backup already exists as an object, adding replication is trivial. I included an optional step using `rclone` to copy the VHD from Blob Storage to Amazon S3. ```bash rclone copy \ blob:$TARGET_CONTAINER/$BLOB_NAME \ "$S3_OBJECT" \ --progress \ --transfers 3 \ --buffer-size 64M \ --ignore-size ``` This is not a migration. It is closer to a simple disaster recovery idea. Once a backup exists in object storage, copying it elsewhere becomes a straightforward operation. This step can be removed entirely without affecting the core backup flow. --- ## Automation Strategy All steps are automated, but responsibilities are kept separate. Terraform handles infrastructure provisioning, while a Bash script handles the backup logic, including disk discovery, snapshot creation, SAS handling, Blob export, and optional S3 replication. --- ## Cleanup Because everything is provisioned using Terraform, cleanup is straightforward. ```bash terraform destroy ``` This removes all resources created for the lab environment and ensures there are no orphaned disks, storage accounts, or unexpected costs left behind. --- ## Closing Thoughts This setup focuses on doing one thing well: backing up Azure Managed Disks into durable object storage in a way that is simple, verifiable, and easy to repeat. There is no overengineering, no hidden services, and no unnecessary abstraction. It is just a clean disk-level backup flow that can be validated end to end and explained clearly. ### Exploring Azure Database for PostgreSQL to AWS Aurora PostgreSQL Migration URL: https://anantafatur.dev/exploring-azure-database-for-postgresql-to-aws-aurora-postgresql-migration/ Last updated: 2025-12-30T10:16:53.000Z # > This write-up was originally prepared as an internal report to my senior, but I’m publishing it here because database migration is a common engineering challenge, and sharing the technical approach may help others facing similar situations. --- Our team planned a migration from **Azure Database for PostgreSQL** to **AWS Aurora PostgreSQL**. The motivation was **operational familiarity**: - The team is more confident troubleshooting Aurora vs. Azure PostgreSQL. - One-time migration, downtime is acceptable. - Both databases are in **private subnets**; no public exposure. **Approach:** Backup and restore using `pg_dump` → S3 → `pg_restore` via bastion hosts. Managed migration services (DMS) were avoided due to VPN/tunnel overhead and the one-time nature of this migration. --- ### **High-Level Architecture** ``` [Azure PostgreSQL] (Private Subnet) | v [Azure Bastion VM] -- Temporary Disk Mounted | v [Amazon S3] -- Object Storage | v [AWS Bastion EC2] -- Temporary Disk Mounted | v [AWS Aurora PostgreSQL] (Private Subnet) ``` --- ### **Scope** - **Source:** Azure Database for PostgreSQL single instance - **Target:** AWS Aurora PostgreSQL (Primary) - **Data size:** \~45 GB **logical database** (from Azure monitoring metrics) - **Method:** `pg_dump` → S3 → `pg_restore` via bastion - **Access:** Bastion-only - **Region:** Azure Jakarta → AWS Jakarta --- ### **Step 1 – Prepare Temporary Disk on Azure Bastion** ```bash lsblk sudo mkfs.ext4 /dev/sdc sudo mkdir /mnt/pgdump sudo mount /dev/sdc /mnt/pgdump ``` - The OS disk cannot be decreased in size, so we attach a temporary disk to safely store the database dump. - Disk size ≥ 100 GB (safety margin for \~45 GB logical database). - Using a separate disk avoids risk to the system and simplifies cleanup after migration. --- ### **Step 2 – Dump Database Using `pg_dump`** ```bash pg_dump \ -h \ -U \ -d \ -F c \ -Z 6 \ --no-owner \ --no-acl \ -f /mnt/pgdump/appdb.dump ``` **Flags explained:** - `-F c` → Custom format (supports parallel restore) - `-Z 6` → Compression (balance CPU and dump size) - `--no-owner --no-acl` → Avoid ownership/permission issues on Aurora - Expected dump size: \~15–25 GB from a 45 GB source, as gzip compression typically reduces logical dumps; actual size may vary based on data composition. > Tip: Temporarily revoke write access to ensure consistent snapshot. --- ### **Step 3 – Upload Dump to S3** ```bash aws s3 cp /mnt/pgdump/appdb.dump s3:///migration/appdb.dump ``` - Use `screen` or `tmux` to prevent session termination. - S3 acts as intermediate storage for cross-cloud transfer. --- ### **Step 4 – Prepare Temporary Disk on AWS Bastion** ```bash lsblk sudo mkfs.xfs /dev/nvme1n1 sudo mkdir /mnt/pgdump sudo mount /dev/nvme1n1 /mnt/pgdump ``` - Attach ≥100 GB EBS volume. --- ### **Step 5 – Download Dump from S3** ```bash aws s3 cp s3:///migration/appdb.dump /mnt/pgdump/ ``` --- ### **Step 6 – Restore to AWS Aurora PostgreSQL** ```bash pg_restore \ -h \ -U \ -d \ --no-owner \ --no-acl \ -j 8 \ /mnt/pgdump/appdb.dump ``` **Aurora-specific notes:** - Parallel restore (`-j 8`) speeds up ingestion. - Ensure the target database is empty/new. - Restore during low-traffic window. --- ### **Step 7 – Post-Restore Validation** ```sql ANALYZE; ``` **Checks:** - Row counts match source - Indexes exist - Sequences aligned --- ### **Step 8 – Cleanup** **Azure Bastion** ```bash sudo umount /mnt/pgdump # Delete temporary disk ``` **AWS Bastion** ```bash sudo umount /mnt/pgdump # Delete temporary EBS volume ``` --- ### **Cost Considerations (Jakarta Region)** | Component | Approx Cost | | ------------------------- | ----------- | | Azure temporary disk | \~$0.29/day | | AWS temporary disk (EBS) | \~$0.32/day | | S3 storage (\~25–30 GB) | \~$0.05 | | Azure egress (\~25–30 GB) | \~$3.0–3.6 | | **Total** | \~$4–5 USD | > Note: Cross-cloud transfer cost is minimal for a one-time migration. --- ### **Conclusion** This PoC shows a bastion-based backup and restore approach for migrating Azure PostgreSQL (\~45 GB) to AWS Aurora. Temporary disks store the dump since OS disks cannot shrink. `pg_dump` → S3 → `pg_restore` allows secure transfer over private subnets, with parallel restore (`-j`) accelerating ingestion. The method avoids VPN/tunnel setup, requires minimal downtime, and is cost-efficient for one-time migrations. ### Deploy MinIO with Docker Compose and Preconfigured Buckets & Access Keys URL: https://anantafatur.dev/deploy-minio-with-docker-compose-and-preconfigured-buckets-access-keys/ Last updated: 2026-02-15T16:48:59.000Z > Prefer watching instead of reading? > The video version is available [here](https://youtu.be/w%5FN8nMXFrrE) and you can check out the GitHub repo [here](https://github.com/anantafaturdev/yt-anantafatur/tree/main/minio). I run Ghost using Docker volumes. At that point, backups are mandatory. Docker volumes are stateful and not portable, so losing the volume means losing the site. After some digging, I ended up using Duplicati because it can back up Docker volumes and push them to S3-compatible storage. Instead of relying solely on managed S3, I wanted to apply the 3-2-1 backup principle properly. That required a self-hosted object storage as one of the backup targets. This is where MinIO fits. Below is the Docker Compose setup I use to run MinIO and bootstrap it automatically. The goal is simple: start MinIO, create a bucket, and provision application credentials without touching the web UI. The setup uses two containers. The first one runs the MinIO server and persists data using a Docker volume. The second container runs the MinIO client (`mc`) and acts as a bootstrapper. It waits until MinIO is healthy, then creates a bucket and generates an access key and secret key for application use. Everything is done programmatically, so the setup is reproducible. MinIO exposes two endpoints. Port **9001** is used for the web console and administrative access. Port **9000** is the S3-compatible API endpoint and must be used by applications such as Duplicati. ```docker-compose.yaml version: "3.9" services: minio: image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z container_name: minio restart: always ports: - "9000:9000" - "9001:9001" command: ["server", "--console-address", ":9001", "/data"] volumes: - data-minio:/data environment: MINIO_ROOT_USER: ${MINIO_ROOT_USER} MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD} healthcheck: test: ["CMD", "curl", "-f", "${MINIO_ENDPOINT}/minio/health/live"] interval: 5s retries: 5 timeout: 2s minio-buckets: image: quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z container_name: minio-buckets depends_on: minio: condition: service_healthy restart: "on-failure" entrypoint: > /bin/sh -c " set -e; echo '>> Setting MinIO alias'; mc alias set localminio ${MINIO_ENDPOINT} ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}; echo '>> Creating bucket'; mc mb localminio/${MINIO_BUCKET_NAME} || true; echo '>> Creating app access key'; mc admin accesskey create localminio/ --access-key ${MINIO_ACCESS_KEY} --secret-key ${MINIO_SECRET_KEY} || true; echo '>> Done'; " volumes: data-minio: ``` The `.env` file supplies credentials and names. `MINIO_ENDPOINT` points to the Docker service name, not `localhost`. The generated access key and secret key are intended for applications like Duplicati, not for administrative use. ```.env MINIO_ROOT_USER= MINIO_ROOT_PASSWORD= MINIO_BUCKET_NAME= MINIO_ENDPOINT=http://minio:9000 #openssl rand -hex 12 | cut -c1-20 MINIO_ACCESS_KEY= #openssl rand -base64 48 | tr -d '\n' | cut -c1-40 MINIO_SECRET_KEY= ``` --- After the stack is running, I checked the resource usage using `docker stats` to get a rough idea of its footprint. ```text CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 7ecc9b2d2d0c minio 1.99% 85.2MiB / 2GiB 4.16% 38.9MB / 42MB 0B / 116MB 9 ``` At idle, MinIO stays under 100 MB of memory with minimal CPU usage. This makes it reasonable to run on small VPS or homelab setups. Resource usage increases during backup or restore operations, but idle overhead remains low. --- Recent MinIO versions no longer expose access key creation cleanly through the web UI. Creating application credentials via `mc admin accesskey create` keeps the setup fully automated. This also makes it easier to rebuild the stack or rotate credentials later without manual steps. Once MinIO is running, Duplicati can be configured to use it as an S3-compatible backend by pointing to the API endpoint on port 9000 and supplying the generated bucket name, access key, and secret key. If MinIO needs to be exposed remotely, it can be placed behind Cloudflared. On the free tier, Cloudflare Tunnel enforces a 100 MB upload size limit per request. If backup chunks exceed that limit, either reduce Duplicati’s chunk size or use a VPN-based approach such as WireGuard or Tailscale instead. References: ### How to Receive Emails from Your Own Domain (for Free) URL: https://anantafatur.dev/how-to-receive-emails-from-your-own-domain-for-free/ Last updated: 2025-12-25T22:43:24.000Z After buying a domain, it’s always exciting to think about having your own email like `hi@anantafatur.dev`. It just feels… professional, right? 😎 But the question is: how do you actually receive emails on your own domain without spending a lot? Good news: there are quick and free solutions that work really well. I’ll share the two easiest ways I know. --- ## 1\. Cloudflare Email Routing (Free & Easy) If you’re already using Cloudflare for your domain, this is probably the simplest option: - First, **add your domain to Cloudflare** by changing the name servers at your domain registrar. - Then, go to **Email Routing** in Cloudflare. - Set it up so that all emails like `*@yourdomain.com` get forwarded to your personal email. Basically, any email sent to your domain will end up in your normal inbox automatically. The best part? It’s **totally free**, no limits for regular personal use. > ⚠️ Note: This method **only allows you to receive emails**, not send them from your custom domain. --- ## 2\. ImprovMX (No Need to Move Domain) Not a fan of moving your domain to Cloudflare? No problem. There’s [ImprovMX](https://improvmx.com/) — a service that lets you forward emails from your domain to your personal inbox. Here’s how it works: 1. Register with your domain + your personal email. 2. Follow the setup instructions on their dashboard. They will give you **DNS records** that you need to add at your domain provider. 3. Once DNS propagates, emails to your domain start arriving in your personal inbox automatically. ImprovMX is also free for basic usage, but they have limits if you need advanced features. Check their pricing if you want to scale. > ⚠️ Note: Just like Cloudflare Email Routing, this method **only receives emails**. You won’t be able to send emails from `hi@yourdomain.com` with this setup. --- ## Why This Works With both methods, you don’t need to manage your own mail server. No Postfix, no Dovecot, no headaches with spam filters. You just get the professional-looking email you wanted, and it lands in your inbox reliably. ![](https://anantafatur.dev/content/images/2025/12/image-1.png) --- 💡 **Pro tip:** If you’re lazy like me, I didn’t include screenshots here. For visual step-by-step guides, you can check YouTube, or maybe I’ll make a video tutorial on my channel later 😎. --- By following either of these methods, you can finally have emails from your own domain, looking professional and without paying a dime. Just keep in mind — **this is for receiving emails only, not sending**. ### How to Get a Free SMTP for Your Personal Homelab Using SMTP2GO URL: https://anantafatur.dev/how-to-get-a-free-smtp-for-your-personal-homelab-using-smtp2go/ Last updated: 2025-12-25T22:44:24.000Z If you’ve been running your own homelab, you know that many self-hosted apps need SMTP for sending email notifications, registration confirmations, or password resets. Paying for third-party services is always an option, but if your usage is small, why not try a free tier? In this article, we’ll show you how to set up **SMTP for free using SMTP2GO**—no credit card required! ![](https://anantafatur.dev/content/images/2025/12/image-2.png) --- ## Why SMTP2GO? SMTP2GO is a simple, beginner-friendly solution for sending emails from your own apps. Here’s why it’s perfect for personal homelab usage: - **Free plan available** (0 cost, no credit card) - **200 emails/day** (1,000 emails/month) - **5 days of email reporting** - **Ticket support** Perfect for testing or low-volume email sending. > Note: You need a domain that’s at least \~1 month old. This delay helps prevent spam registrations. --- ## Step 1: Register Your Free Account 1. Go to [SMTP2GO signup page](https://www.smtp2go.com/pricing/) 2. Input your email with your own domain (you can set up custom domain emails with services like Cloudflare or ImprovMX—more on that in a future article). 3. Confirm your email. Some accounts may require phone verification. --- ## Step 2: Verify Your Domain Once logged in: 1. Go to [**Verified Senders**](https://app-us.smtp2go.com/sending/verified%5Fsenders) 2. Add your sender domain. 3. Add the **CNAME record** to your domain provider service. 4. Wait \~15 minutes for DNS propagation. 5. Once verified, your domain status will show as “verified.” > I know screenshots make things easier… but I’m too lazy to include them here 😅. For a visual step-by-step, kindly check YouTube :D, or I might create a video on my own channel in the future. --- ## Step 3: Create SMTP Users Now that your domain is verified: 1. Go to [**SMTP Users**](https://app-us.smtp2go.com/sending/smtp%5Fusers/) 2. Click **Add SMTP User**, set a username and password. \- Tip: Split users per app to track usage easily. And… voila! Your SMTP is ready to use **completely free**. 🎉 --- ## Step 4: Configure Your Apps Use the credentials you just created to configure your apps. You can test the SMTP with [SMTPer.net](https://www.smtper.net/) to ensure it works. You’ll find all necessary details (SMTP server, ports, etc.) in the **SMTP User menu** on SMTP2GO. --- ## Summary Using SMTP2GO, you can: - Send emails from your homelab apps without spending a dime - Easily manage multiple apps with separate SMTP users - Verify your domain and ensure your emails are delivered properly For personal or testing purposes, this setup is more than enough. No credit card, no hassle, just simple, free SMTP for your homelab. ### Set Up Ghost with Docker Compose – My Minimal Blogging Setup URL: https://anantafatur.dev/set-up-ghost-with-docker-compose-my-minimal-blogging-setup/ Last updated: 2026-02-22T16:35:40.000Z > Prefer watching instead of reading? > The video version is available [here](https://youtu.be/qJV4spRXBPg) and you can check out the GitHub repo [here](https://github.com/anantafaturdev/yt-anantafatur/tree/main/ghost). So, I want to break into DevOps. Sounds cool, right? But here’s the catch: it’s **not entry-level**. Most people start from dev or ops roles, gain experience, then move into DevOps. Me? I wanted to jump straight in. 😅 To prove I can do it (with a senior supervisor watching over me, of course), I decided to start writing **technical docs on my own website**. Years ago, I tried Ghost for blogging. And, well… perfectionism struck hard. Every draft felt “not perfect yet,” and I’d leave it there forever. Drafts piled up like unfinished code. This time, I said, **screw it**—messy writing is better than no writing. Nobody reads my blog anyway… yet. --- ## Finding a Cheap VPS (because my homelab is noisy af) My homelab lives in my bedroom. Fun fact: it **shuts down every night** because it’s louder than a rocket launch 😅. So, I needed a cheap VPS to run Ghost 24/7. Enter [hostdata.id NAT VPS](https://hostdata.id/nat-vps-indonesia/) — **30K/month**, 2GB RAM, 1 vCPU, 25GB SSD. Small but mighty enough for my Ghost instance. Bench.sh says: ``` CPU Model : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz CPU Cores : 1 @ 1572.582 MHz CPU Cache : 35840 KB AES-NI : ✓ Enabled VM-x/AMD-V : ✓ Enabled Total Disk : 24.5 GB (3.5 GB Used) Total Mem : 2.0 GB (747.4 MB Used) System uptime : 1 days, 20 hour 24 min Load average : 0.14, 0.09, 0.07 OS : Debian GNU/Linux 11 Arch : x86_64 (64 Bit) Kernel : 4.19.0 TCP CC : Virtualization : OpenVZ IPv4/IPv6 : ✓ Online / ✓ Online Organization : AS141968 PT Industri Kreatif Digital Location : Jakarta / ID Region : Jakarta ---------------------------------------------------------------------- I/O Speed(1st run) : 50.5 MB/s I/O Speed(2nd run) : 65.8 MB/s I/O Speed(3rd run) : 74.2 MB/s I/O Speed(average) : 63.5 MB/s ---------------------------------------------------------------------- Node Name Upload Speed Download Speed Latency Speedtest.net 4924.37 Mbps 412.77 Mbps 1.19 ms Paris, FR 505.69 Mbps 443.70 Mbps 157.49 ms Shanghai, CN 181.38 Mbps 342.45 Mbps 338.07 ms Hong Kong, CN 870.14 Mbps 394.31 Mbps 43.04 ms Singapore, SG 864.16 Mbps 413.19 Mbps 13.36 ms Tokyo, JP 406.46 Mbps 425.12 Mbps 124.63 ms ``` Honestly, I don’t care about SLA, speed, or fancy specs. This is **for starting**, not for winning any awards. --- ## Why Docker Compose? Because I’m lazy (and smart 😎) I might migrate Ghost to a bigger VPS later. Docker Compose is perfect: move the containers + volumes + configs anywhere with minimal headache. No “install everything from scratch” nightmares. --- ## Minimal Docker Compose Setup Here’s what I actually use: ```yaml services: ghost: image: ghost:6.10.3-alpine3.23 restart: always ports: - 8087:2368 environment: database__client: mysql database__connection__host: db database__connection__user: root database__connection__password: 96WlUp5b/p2VbBx9 database__connection__database: ghost url: https://anantafatur.dev mail__transport: SMTP mail__options__host: mail.smtp2go.com mail__options__port: 587 mail__options__auth__user: XXXXXXXXXXXX mail__options__auth__pass:XXXXXXXXXXXX mail__from: XXXXXXXXXXXX volumes: - ghost:/var/lib/ghost/content db: image: mysql:8.0-bookworm restart: always environment: MYSQL_ROOT_PASSWORD: 96WlUp5b/p2VbBx9 volumes: - db:/var/lib/mysql volumes: ghost: db: ``` Why this works for me: - Minimalist: no Caddy/Nginx, no analytics, no nonsense. Just **writing first**. - Using volumes to persist Ghost content—your posts won’t vanish if the container restarts. I’ve got plans to cover backups and S3 in another article. - Ghost is live at `http://:8087`. Easy. - I expose it via **Cloudflared**, so I can access it with my domain without touching reverse proxy configs. --- ## Lessons Learned 1. **Start now, imperfectly.** Drafts piling up are just “art in progress.” 2. **Cheap VPS + Docker Compose** \= fast way to get your tech blog online. 3. **Perfectionism is overrated.** Seriously, just write. --- Honestly, this blog isn’t for views or awards. It’s my **proof to myself** that I can start, finish, and publish technical work. Plus, it’s a safe place to experiment, mess up, and maybe laugh at my old drafts later. ### Hello, world 👋 URL: https://anantafatur.dev/hello-world/ Last updated: 2025-12-23T10:50:38.000Z Welcome to my documentation blog! This blog exists for a very simple reason: **I’m forgetful… and a bit lazy to search for the same solution again.** I’ve fixed a problem before. I *know* I have. But somehow, I still end up googling it like it’s the first time 😅 So instead of repeating the pain, I decided to write everything down. ## From YouTube to written docs ✍️ Last year, I started a **YouTube channel** to document things I learned about: cloud computing, Linux, system administration, and DevOps. This year, I’m doing the **written version**. Why? Because written notes are: - faster to scan - easier to search - perfect for quick recall when things break again This blog is basically my **personal knowledge base** for real problems I’ve already faced. ## Why the Roblox banner? 🤔 Good question. Short answer: I was confused about what image to use… so I picked Roblox 😆 No hidden meaning. Just vibes. ## What you’ll find here Mostly simple and practical notes about: - Cloud computing - Linux & system administration - DevOps tools and workflows - Troubleshooting and “future me, don’t forget this” moments If this blog helps you solve a problem faster, that’s awesome. If not, at least it saves *me* a lot of time.