How VMware Tanzu CloudHealth migrated from self-managed Kafka to Amazon MSK
Big Data Blog
This article discusses how VMware Tanzu CloudHealth migrated their self-managed Apache Kafka workloads (running version 2.0) to Amazon Managed Streaming for Apache Kafka (Amazon MSK) running version 2.6.2.
Specifically, the article covers:
- Overview of Kafka cluster and its role in VMware Tanzu CloudHealth's architecture
- Reasons for migrating to Amazon MSK, including simplified operations, deprecated legacy pipelines, cost savings, and better support
- Pre-migration steps like assessment, Amazon MSK setup with Terraform, and network configuration
- Implementation details like automated deployment pipelines, improved observability, cost savings, and security enhancements
- Kafka topic migration strategy using MirrorMaker and concurrent replication
- Challenges faced during migration, such as offset discrepancies, slow data transfer, and lack of documentation
- Final architecture and future improvement plans like using AWS Graviton instances and tiered storage
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Feb 19
2024
2024
How VMware Tanzu CloudHealth modernized container workloads from self-managed Kubernetes to Amazon Elastic Kubernetes Service
Mar 27
2024
2024
How Cornerstone Built a Full-Fledged Apache Kafka Replication Using Amazon MSK Connect
Mar 21
2024
2024
Build an end-to-end serverless streaming pipeline with Apache Kafka on Amazon MSK using Python
Apr 20
2026
2026
Migrate third-party and self-managed Apache Kafka clusters to Amazon MSK Express brokers with Amazon MSK Replicator
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.