Benchmark: Longhorn performance on Kubernetes on Hetzner Cloud

I was always wondering how Longhorn performed on Hetzner Cloud (VM) on Kubernetes. So, I started measuring and here are my results.

I was always wondering how Longhorn performed on Hetzner Cloud (VM) on Kubernetes. So, I started measuring and here are my results.

Results

Write performance graph.
Read performance graph.
Read and write performance measurements in tabular format.

My setup

Infra
3x Hetzner Cloud VM
VM specs: CAX11– 4GB RAM, 2vCPU, 40GB Storage
CAX11: is ARM64 based VM which uses shared CPU with other VM’s.

Platform

Microk8s: on all 3 nodes (VM’s). Microk8s is a Kubernetes implementation.
Storage: Longhorn on all 3 nodes. Longhorn is an open-source cloud-native block storage provider. It makes distributed storage management like, horizontal scaling, replication, backup and snapshotting very easy.

Test approach

Kubernetes objects:
Test PVC with longhorn as storageClass
Test-pod mounted on the Test PVC, with dd

Testing tools and setup:
I used linux’s dd utility (cli) to do some manual testing. This is how my commands for writing and reading looked like, with different variables.

#create and (over)write a file of GB. Use 1GB buffer size and repeat it 1 time. 
#dd if=/dev/zero of=./testfile bs=1G count=1 oflag=dsync
#read one time 1GB of the testfile
#dd if=./testfile of=/dev/null bs=1G count=1 iflag=direct

I run each variable 3 times and noted the lowest result. No average, no median, no max.

Testing variables:
File sizes for writing and reading:
100KB, 1MB, 10MB, 50MB, 100MB, 200MB, 1GB

Storage types:
Longhorn 3x replica: refers to the storage backend that is mounted to the pod which get replicated to all three nodes transparently by Longhorn. Longhorn guarantees that the data is written to all nodes before it gives the success signal.

Longhorn 2x replica: same as 3x, but data is only replicated to 2 nodes. This is less data replication/availibility, but because there will be less data copying, acknowledging, theretically, it should perform better.

/tmp The fast local mounted storage directly on the node where the pod is mounted without longhorn in-between.

Conclusion

When it comes to Longhorn, it depends heavily on the network performance and bottlenecks, like any other storage provider of this architecture (Cephs Rook is on example).

The amount of data replication does not only cost money, but also performance. The trade-off is once again confirmed.

Last but not least: Local storage remains the king of performance.