Optimize consumer rebalancing on Amazon MSK with next generation protocol

This article explains how to optimize consumer rebalancing on Amazon MSK using the KIP-848 next generation protocol, which moves rebalancing logic from clients to brokers for improved performance.

• KIP-848 eliminates global synchronization barriers, allowing only affected consumers to rebalance while others continue processing
• Available on Amazon MSK with Apache Kafka 4.0+ on both Standard and Express brokers
• Enable by setting group.protocol=consumer in client configuration and verifying client library support
• Server-side parameters (heartbeat interval, session timeout, assignors) are now broker-controlled instead of client-configured
• Diagnose issues using broker logs to confirm protocol is active and check for session timeout or graceful shutdown problems
• Implement graceful shutdown and static membership (group.instance.id) to prevent unnecessary rebalances during restarts
• Monitor improvements using CloudWatch metrics and kafka-consumer-groups.sh to verify faster rebalance times and stable membership

The consumer protocol significantly improves rebalance performance for large groups, high-availability applications, and frequently scaling environments by eliminating rebalance storms and enabling continuous message processing.