On-demand and scheduled scaling of Amazon MSK Express based clusters

This article demonstrates how to implement dynamic scaling for Amazon MSK Express clusters using intelligent rebalancing, with two approaches: on-demand and scheduled scaling.

• Express brokers enable 20x faster scaling and 90% faster recovery than provisioned clusters
• Intelligent rebalancing automatically redistributes partitions across brokers without manual intervention
• On-demand scaling tracks CloudWatch metrics and adjusts broker count when thresholds are exceeded
• Scheduled scaling uses EventBridge cron expressions to scale clusters at predetermined times
• Solution uses Lambda functions triggered by EventBridge to manage scaling operations
• Decoupled compute-storage architecture enables rapid horizontal scaling within minutes
• Dynamic broker bootstrapping allows seamless client connections during scaling operations
• Deploy using AWS CDK with configurable thresholds and monitoring frequency
• Avoid running both scaling solutions simultaneously on same cluster to prevent conflicts

The solution provides automated capacity management for dynamic Kafka workloads, eliminating manual partition rebalancing while maintaining performance and optimizing costs.