Improving throughput of serverless streaming workloads for Kafka

This article provides guidance on optimizing AWS Lambda for processing high-volume Apache Kafka streams with focus on throughput and scaling.

• Use Provisioned Mode for bursty workloads to ensure predictable, fast scaling instead of on-demand mode
• Apply ESM filtering to drop irrelevant records before Lambda invocation, reducing cost and concurrency
• Configure batch window and batch size to process more records per invocation and improve efficiency
• Optimize handler code by reducing per-record work and increasing memory allocation for better CPU
• Monitor OffsetLag, Duration, Concurrency, and Errors metrics to detect issues and guide tuning
• Single provisioned poller can process up to 5 MB/s of Kafka data
• Follow iterative optimization loop: baseline, filter, batch, speed up, test spikes, alert, re-evaluate

The article emphasizes that effective Kafka-Lambda optimization requires understanding the poll-filter-batch-invoke workflow and using observability metrics to drive configuration decisions.