How Amazon GTTS runs large-scale ETL jobs on AWS using Amazon MWAA
Big Data Blog
This article discusses how Amazon's Global Transportation Technology Services (GTTS) team runs large-scale ETL jobs on AWS using Amazon Managed Workflows for Apache Airflow (Amazon MWAA).
Specifically, the article covers:
- The legacy orchestration platform used by GTTS and its challenges like maintainability, scalability, and multi-tenancy
- The benefits of migrating to Amazon MWAA, including improved maintainability, cost-effectiveness, scheduling capabilities, user experience, security, monitoring and alerting, and scalability
- The architecture of the new Amazon MWAA-based orchestration platform, including components like DAG updates, infrastructure as code, authentication and permissions, UI and DAG runs, Airflow workers, data stores and external compute services, and logging and alerting
- How the new solution aligns with the AWS Well-Architected Framework pillars, such as operational excellence, performance efficiency, security, reliability, and cost optimization
- Conclusion and resources for learning more about Amazon MWAA
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Apr 25
2024
2024
Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA
May 16
2024
2024
Introducing Amazon MWAA support for the Airflow REST API and web server auto scaling
Feb 9
2026
2026
Orchestrate end-to-end scalable ETL pipeline with Amazon SageMaker workflows
May 1
2026
2026
A guide to Airflow worker pool optimization in Amazon MWAA
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.