This article describes Emumba's experience building a streaming IoT data ingestion system handling 250,000 events/second at scale on AWS.

• Initial PoC with 100,000 devices scaled poorly to production requirements of 22 billion daily events
• Architecture used Apache Kafka, Kafka Streams, and a real-time data warehouse for ingestion and querying
• Simple interpolation scaling failed; KStreams consumption rate periodically collapsed to zero
• Memory tuning and custom suppress functionality improved performance but disk I/O remained problematic
• Discovered instance burst limits: i3en.xlarge had only 30-minute sustained disk I/O capability
• Final solution used i3.8xlarge instances providing 10 Gbps sustained network and 875 Mbps disk throughput
• One instance had hardware issues requiring replacement; managed services like Amazon MSK recommended for production

Key lesson: Scaling streaming infrastructure requires deep understanding of distributed systems architecture, sustained throughput calculations, and proper instance selection rather than simple resource interpolation.