How Heroku reduced their operational overhead by migrating their 30 TB self-managed database from Amazon EC2 to Amazon DynamoDB

The article discusses how Heroku, a platform as a service (PaaS) solution, migrated the storage backend for their application metrics and alerting features from self-managed Apache Cassandra clusters to Amazon DynamoDB.

Specifically, the article covers:

• Heroku's 'Metrics as a Service' (MetaaS) architecture, which collected and processed application metrics using Apache Kafka and Apache Cassandra on Amazon EC2
• The reasons for migrating to DynamoDB, including reducing operational overhead and leveraging a managed service
• The migration process, which involved writing data to both Cassandra and DynamoDB in parallel, validating data consistency, and incrementally switching reads to DynamoDB
• The benefits realized after the migration, including reduced costs by up to 50%, improved query performance with consistent low latency, and a 90% reduction in operational overhead
• Lessons learned, such as using VPC endpoints, tuning the DynamoDB SDK client, and compressing large records to save on write capacity units