Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA
Big Data Blog
This blog post demonstrates how to orchestrate an end-to-end extract, transform, and load (ETL) pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon Managed Workflows for Apache Airflow (Amazon MWAA).
Specifically, the article covers:
- Solution overview using multiple AWS accounts for enhanced security and data governance
- Prerequisites to set up required resources like S3 buckets, Glue jobs, and Redshift Serverless databases
- Setting up cross-account access between accounts for S3 buckets
- Configuring Amazon MWAA connection with AWS Secrets Manager to securely store database credentials
- Creating and running an Apache Airflow DAG to orchestrate the ETL pipeline
- Verifying the DAG run and results in Redshift and S3
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Feb 9
2026
2026
Orchestrate end-to-end scalable ETL pipeline with Amazon SageMaker workflows
Oct 6
2026
2026
Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service
Apr 10
2024
2024
Achieve near real time operational analytics using Amazon Aurora PostgreSQL zero-ETL integration with Amazon Redshift
May 23
2024
2024
Get started with AWS Glue Data Quality dynamic rules for ETL pipelines
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.