How BMW Group built a serverless terabyte-scale data transformation architecture with dbt and Amazon Athena
Big Data Blog
BMW Group developed a serverless data transformation architecture using dbt and Amazon Athena to address challenges with data processing and scalability in their Cloud Efficiency Analytics (CLEA) team.
- Replaced complex, rigid data infrastructure with a modular serverless pipeline
- Uses GitHub Actions for workflow automation and deployment
- Implemented a three-stage data transformation process: Source, Prepared, and Semantic layers
- Leverages Athena for efficient querying of large-scale Parquet data
- Integrated dbt for SQL-based transformations with about 400 models and 100 data tests
Key benefits include improved development agility, cost-efficiency, and the ability to quickly modify data models while maintaining high performance and data quality. The architecture processes around 300 GB daily from a 15 TB data lake.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2024
2026
2024
2024
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.