Achieving sub-10ms latency and 94% cost savings with Diskless Kafka using AutoMQ and Amazon FSx for NetApp ONTAP

This article demonstrates how AutoMQ achieves sub-10ms latency and 94% cost savings for Diskless Kafka using Amazon FSx for NetApp ONTAP as a Write-Ahead Log (WAL) layer.

• AutoMQ combines FSx for ONTAP WAL with S3 storage for low-latency, cost-effective Kafka
• FSx for ONTAP provides Multi-AZ shared storage without cross-AZ data transfer charges
• Benchmarks show 5.98ms average write latency and 7.79ms end-to-end latency
• Monthly cost of $18,345 versus $317,092 for traditional Apache Kafka (94% savings)
• Zone-routing interceptor localizes writes to same Availability Zone, avoiding cross-AZ traffic
• Fixed-size FSx for ONTAP WAL stores only hot log data; S3 handles long-term retention
• Eliminates need for three-replica replication, reducing storage and network costs
• AutoMQ BYOC available on AWS Marketplace with free trial option

FSx for ONTAP enables Diskless Kafka to achieve local disk-class latency while maintaining S3 cost benefits and Multi-AZ high availability for production workloads.