How CrescoNet Optimized Their Architecture and Reduced Their AWS Bill by Over 40%

This article details how CrescoNet, a smart metering solutions integrator, reduced their AWS bill by over 40% while maintaining performance and reliability for their meter-to-cash data pipeline processing 4.5 billion readings daily.

• Transitioned from AWS Glue to Amazon EMR on EKS with Spot Instances, saving over $50,000/month on ETL compute
• Adopted Apache Iceberg with compaction to reduce S3 GET/PUT costs by 27 times and improve query performance
• Optimized Lambda runtime from Python 3.8/3.9 to 3.11, reducing execution times by 30% and enabling memory reduction
• Applied DynamoDB Time To Live to reduce stored data by 66%, saving $3,000/month in storage costs
• Implemented Amazon MSK tiered storage and consolidated non-production instances for cost and performance gains
• Batched SQS messages to reduce request costs by over 20 times
• Purchased size-flexible RDS Reserved Instances for significant database savings
• Automated non-production workload shutdown outside business hours using Instance Scheduler

Through systematic identification, assessment, and implementation of cost optimizations, CrescoNet achieved an 80% reduction in meter-to-cash pipeline costs while improving performance and scalability.