Home icon

Introducing queue consumption mode for Apache Kafka in AWS Lambda

Compute Blog

AWS Lambda introduces queue consumption mode for Apache Kafka event source mappings, enabling queue-style event-driven workloads with high parallelism decoupled from partition count.

  • Queue mode uses Kafka Share Groups (KIP-932) to allow multiple consumers to read from same partitions concurrently
  • Eliminates head-of-line blocking and removes partition count as ceiling on parallelism
  • Requires Apache Kafka 4.2 or later with share groups enabled
  • Available in on-demand mode (up to 5 pollers, 5 MB/s) and provisioned mode (up to 2,000 pollers)
  • Stream mode remains default for ordered per-partition processing; Queue mode for independent records
  • Configure via Lambda console, AWS CLI, CloudFormation, SAM, or CDK with ConsumptionMode parameter
  • Supports failure handling with retry attempts, batch bisection, and on-failure destinations

Queue mode enables independent record processing on Kafka without additional infrastructure, ideal for image rendering, fraud detection, notifications, and telemetry ingestion workloads.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 2
2026
AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources
Oct 9
2026
AWS Lambda supports OAuth authentication for self-managed Apache Kafka event sources
Nov 21
2025
AWS Lambda announces new capabilities to optimize costs up to 90% for Provisioned mode for Kafka ESM
Jul 17
2025
AWS Lambda announces low latency processing for Kafka events

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.