Running diskless Apache Kafka with Amazon S3 Files

This article demonstrates how to run Apache Kafka without local disk storage by using Amazon S3 Files, which mounts S3 buckets as NFS file systems for seamless integration.

• S3 Files eliminates disk capacity planning and manual log offloading by providing scalable, durable storage backed by S3
• Decouples compute and storage lifecycles, enabling broker replacement without data loss or restoration
• Supports full file system semantics (append, rename, locking) requiring no Kafka configuration changes
• Multi-region architecture spans producer and consumer regions with cross-region replication via MirrorMaker 2
• Broker acknowledgement latency averages 10-15ms with replication latency around 155-165ms across regions
• S3 Files automatically exports closed segments to S3 after 60 seconds of inactivity
• Includes step-by-step setup: create S3 bucket, mount file system on EC2, configure Kafka to use S3 Files mount

This architecture simplifies Kafka operations by leveraging S3's durability and scalability while maintaining full file system compatibility.