How Bazaarvoice modernized their Apache Kafka infrastructure with Amazon MSK

This article details how Bazaarvoice migrated their self-hosted Apache Kafka infrastructure to Amazon Managed Streaming for Apache Kafka (Amazon MSK), addressing operational complexity and scaling challenges.

• Self-hosted Kafka required extensive manual management of scaling, configuration, and infrastructure across multiple dimensions
• Amazon MSK provided automated operations, multi-AZ reliability, simplified upgrades, and built-in ISO 27001 compliance
• Migration strategy involved deploying parallel Kafka Streams applications to avoid state loss and downtime exceeding 30-minute SLA
• Used MirrorMaker2 for input topic replication while custom solution handled state hydration and validation
• Consumer migration required tailored approaches for standard consumers, Kafka Connect sinks, and Spark/Flink applications
• Post-migration: security patching automated, broker recovery fully managed, authentication simplified via IAM integration
• Kafka version upgrades now automated versus weeks of planning previously required
• Key lessons: validate strategy with experts, verify data consistency, start with small migrations, plan granularly, optimize throughput

Bazaarvoice successfully transformed Kafka operations from resource-intensive self-management to streamlined managed service, improving efficiency and reliability while reducing operational overhead.