MariaDB Galera Cluster on hybrid cloud benchmark

I needed a database that could survive a node failure. Not just survive, but keep working without anyone noticing. That means synchronous replication, multi-master, active-active. The obvious choice was MariaDB Galera Cluster [5], [6], [7]. But I did not want to put everything in one place. If AWS us-east-1 goes down, I wanted a node on my own hardware that could keep things running.

The problem? Nobody had published real benchmarks for running Galera across a hybrid cloud setup. Plenty of papers look at Galera in homogeneous environments [5], [8]. Plenty of papers talk about hybrid cloud in general [9], [10], [11], [12]. But nobody had put the two together. So I did.

This blog post is adapted from the conference paper version I presented at 8th ISRITI 2025: same data, less formal.

This was actually my skripsi, my final project during college. Is it a simple study? Yes. I really did not care about the challenge. I just wanted to finish my degree as fast as possible so I could start getting real work experience. This was also my first paper. Even though my advisor helped proofread it, I am sure it still has some consistency issues. Sorry about that.

The setup

I built a three-node cluster. Two nodes on AWS EC2 (t3.medium: 2 vCPUs, 4 GB RAM, 32 GB gp3 SSD). One node on my own Proxmox VE server with the same specs. All three nodes ran MariaDB 10.11.8 with Galera synchronous replication.

To connect the AWS and Proxmox nodes, I used WireGuard VPN. This kept the traffic encrypted and put all three nodes on the same private network. I also deployed MariaDB MaxScale 24.02.5 on one of the AWS nodes to handle load balancing, read-write splitting, and automated failover [8].

Honestly, the WireGuard tunnel was the biggest unknown. I knew it would add latency. I just did not know how much it would hurt.

For benchmarking, I used SysBench 1.0.20 with three OLTP workloads: read-only, write-only, and mixed read-write. Each test ran for 120 seconds, with thread counts from 1 to 192. I measured transactions per second (TPS) and 95th percentile response time [13], [14], [15], [16].

I tested three configurations: hybrid (AWS + Proxmox), AWS-only, and Proxmox-only. The single-cloud tests gave me the baseline. The hybrid test told me what I was actually paying for the cross-cloud redundancy.

Read-only: AWS dominates

The read-only workload was the clearest story. AWS-only averaged 2,131.92 TPS. Proxmox-only averaged 1,303.27 TPS. That is a 63.5% gap. AWS peaked at 2,663.53 TPS at 16 threads. Proxmox peaked later, at 1,662.84 TPS at 144 threads, but never got close.

The hybrid setup sat in the middle: 1,203.06 TPS average. That is 43.6% lower than AWS-only. The WireGuard overhead is real. Every read that hits the Proxmox node has to cross the VPN, and that shows up in the numbers.

Latency told the same story. AWS-only had a minimum 95th percentile of 4.82 ms at 1 thread. Hybrid started at 19.90 ms. By 192 threads, AWS was at 116.80 ms, hybrid at 170.34 ms.

Write-only: everyone suffers

Write workloads are where synchronous replication really hurts. The Galera protocol requires every node to certify every write before it commits. That means the slowest node becomes the bottleneck.

AWS-only averaged 104.42 TPS. Not great, but the best of the three. Proxmox-only averaged 76.29 TPS. The hybrid setup managed 91.74 TPS, which is only 12.1% lower than AWS-only. That surprised me. The write penalty for the hybrid setup was smaller than I expected.

But here is the real headline: latency exploded for all three configurations. At 64 threads, the 95th percentile crossed 1,200 ms for every setup. At 192 threads, Proxmox-only hit 9,624.59 ms. AWS-only, the best case, still hit 6,247.39 ms. That is over 200 times higher than the minimum latency.

This is not a cloud problem. It is a Galera problem. Synchronous replication just does not scale for writes. The consensus protocol caps your throughput no matter how good your hardware is.

Mixed read-write: the practical workload

The mixed workload is probably the most realistic test. It is a mix of reads and writes, similar to what a typical web application would generate.

AWS-only averaged 579.99 TPS and peaked at 653.48 TPS. Proxmox-only averaged 439.72 TPS. The hybrid setup averaged 419.95 TPS. That is actually slightly lower than Proxmox-only on average throughput, but the hybrid had better latency consistency: 333.16 ms vs 359.11 ms for the 95th percentile.

So the hybrid setup was not the fastest, but it was more predictable. For a workload that mixes reads and writes, the cross-cloud latency penalty is noticeable but not crippling.

The write bottleneck is universal

The single most important thing I learned from this whole experiment is the write scalability ceiling. It does not matter which configuration you pick. Once you push past 64 concurrent threads doing writes, the 95th percentile latency crosses 1.2 seconds. By 192 threads, it is measured in seconds, not milliseconds.

This is the cost of synchronous replication. Galera uses a certification-based replication where every write must be certified by all nodes before it commits. More threads mean more contention. More contention means more waiting. The latency curve is exponential.

If you are building a write-heavy application on Galera, this is the wall you will hit. It is not a cloud problem. It is a protocol problem.

What this means

If you want raw performance, put everything in AWS. The AWS-only setup gave 63.5% more read throughput and 36.9% more write throughput than Proxmox-only. The internal AWS networking is fast, and not having to cross a VPN helps a lot.

But if you want high availability across multiple locations, the hybrid setup is a reasonable compromise. You lose 43.6% read throughput and 12.1% write throughput compared to AWS-only. But you gain a node that sits outside AWS. If the AWS region has an outage, your Proxmox node is still running.

The hybrid model is best suited for read-heavy applications. Things like content delivery, e-commerce catalogs, or internal dashboards. The read penalty is significant but the writes are actually not that much worse than AWS-only. For write-heavy workloads like financial transactions or real-time analytics, I would not recommend hybrid. The write bottleneck is already bad enough without adding cross-cloud latency on top.

What I would recommend (with caveats)

This is not a universal truth. It is what I observed with one specific setup, one benchmark tool, and one database engine. But here is what I would say:

If you need a Galera cluster and you care about performance above all else, go cloud-native. Put all your nodes in the same AWS region. You will get the best throughput and the lowest latency.

If you need geographic redundancy and can tolerate a throughput hit, the hybrid model works. The read penalty is around 40%, the write penalty is around 10%. For many applications, that is an acceptable trade-off for having a node outside the cloud provider.

If you are building a write-heavy application, think hard about whether synchronous replication is actually what you need. The write bottleneck is severe, and it applies to every configuration. You might be better off with asynchronous replication, or a different architecture entirely.

What I did not test

I did not measure RTT between the AWS and Proxmox nodes. I know the latency is there, but I cannot say exactly how much it contributes. I did not tune any Galera parameters for the hybrid network conditions. Maybe there are configs that reduce the cross-cloud penalty. I did not test semi-synchronous or asynchronous replication models, which could address the write scalability bottleneck.

I only tested one instance type (t3.medium) on one cloud provider (AWS) with one on-premise setup (Proxmox VE). Different hardware would give different numbers. I did not test with real application workloads, only SysBench OLTP patterns. I did not test failover timing or measure how quickly MaxScale redirects traffic during a node failure, though I did confirm it works.

References

[1] P. Memarzia, H. Zhang, K. Ho, R. Grosman, and J. Wang, "GaussDB-Global: A Geographically Distributed Database System," in 2024 IEEE 40th International Conference on Data Engineering (ICDE), May 2024, pp. 5111–5118. doi: 10.1109/ICDE60146.2024.00383.

[2] S. Ferreira, J. Mendonca, B. Nogueira, W. Tiengo, and E. Andrade, "Benchmarking Consistency Levels of Cloud-Distributed NoSQL Databases Using YCSB," IEEE Access, vol. 13, pp. 63428–63438, 2025, doi: 10.1109/ACCESS.2025.3558923.

[3] P. Aryan, R. Khatri, and V. Balakrishnan, "An Experimental Framework for Implementing Decentralized Autonomous Database Systems in Rust," Dec. 06, 2024, arXiv: arXiv:2412.05078. doi: 10.48550/arXiv.2412.05078.

[4] L. Zhang, T. Jia, M. Jia, and Y. Li, "LogDB: Multivariate Log-based Failure Diagnosis for Distributed Databases (Extended from MultiLog)," May 03, 2025, arXiv: arXiv:2505.01676. doi: 10.48550/arXiv.2505.01676.

[5] R. Shrestha and T. Tandel, "An Evaluation Method and Comparison of Modern Cluster-Based Highly Available Database Solutions," in Proceedings of the 13th International Conference on Cloud Computing and Services Science, Prague, Czech Republic: SCITEPRESS, 2023, pp. 131–138. doi: 10.5220/0011714400003488.

[6] A. A. E. Alflahi, M. A. Y. Mohammed, and A. Alsammani, "Enhancement of database access performance by improving data consistency in a non-relational database system (NoSQL)."

[7] V. B. F. Gomes, M. Kleppmann, D. P. Mulligan, and A. R. Beresford, "Verifying Strong Eventual Consistency in Distributed Systems," Proc. ACM Program. Lang., vol. 1, no. OOPSLA, pp. 1–28, Oct. 2017, doi: 10.1145/3133933.

[8] R. Shrestha, "High Availability and Performance of Database in the Cloud - Traditional Master-slave Replication versus Modern Cluster-based Solutions," in Proceedings of the 7th International Conference on Cloud Computing and Services Science, Porto, Portugal: SCITEPRESS, 2017, pp. 413–420. doi: 10.5220/0006294604130420.

[9] F. Ullah, S. Dhingra, X. Xia, and M. A. Babar, "Evaluation of Distributed Data Processing Frameworks in Hybrid Clouds," Jan. 06, 2022, arXiv: arXiv:2201.01948. doi: 10.48550/arXiv.2201.01948.

[10] M. Waseem et al., "Containerization in Multi-Cloud Environment: Roles, Strategies, Challenges, and Solutions for Effective Implementation," July 08, 2025, arXiv: arXiv:2403.12980. doi: 10.48550/arXiv.2403.12980.

[11] M. M. Kondappan, "Cloud computing and its role in insurance platform integration," World J. Adv. Res. Rev., vol. 26, no. 1, pp. 3700–3712, Apr. 2025, doi: 10.30574/wjarr.2025.26.1.1506.

[12] A. Gurram, "The role of cloud architecture in modernizing enterprise platforms," World J. Adv. Res. Rev., vol. 26, no. 1, pp. 3501–3511, Apr. 2025, doi: 10.30574/wjarr.2025.26.1.1436.

[13] N. Husufa and I. Prihandi, "Optimizing JMeter on Performance Testing Using the Bulk Data Method," vol. 4, no. 2, 2022.

[14] H. Li et al., "Online Timestamp-based Transactional Isolation Checking of Database Systems (Extended Version)," Apr. 02, 2025, arXiv: arXiv:2504.01477. doi: 10.48550/arXiv.2504.01477.

[15] D. Landau, J. Barbosa, and N. Saurabh, "eBPF-Based Instrumentation for Generalisable Diagnosis of Performance Degradation," May 19, 2025, arXiv: arXiv:2505.13160. doi: 10.48550/arXiv.2505.13160.

[16] D. Liyanage et al., "A Benchmark for Databases with Varying Value Lengths," Aug. 11, 2025, arXiv: arXiv:2508.07551. doi: 10.48550/arXiv.2508.07551.