Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA
Big Data Blog
This blog post demonstrates how to orchestrate an end-to-end extract, transform, and load (ETL) pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon Managed Workflows for Apache Airflow (Amazon MWAA).
Specifically, the article covers:
- Solution overview using multiple AWS accounts for enhanced security and data governance
- Prerequisites to set up required resources like S3 buckets, Glue jobs, and Redshift Serverless databases
- Setting up cross-account access between accounts for S3 buckets
- Configuring Amazon MWAA connection with AWS Secrets Manager to securely store database credentials
- Creating and running an Apache Airflow DAG to orchestrate the ETL pipeline
- Verifying the DAG run and results in Redshift and S3
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Feb 9
2026
2026
Orchestrate end-to-end scalable ETL pipeline with Amazon SageMaker workflows
Aug 6
2024
2024
How Amazon GTTS runs large-scale ETL jobs on AWS using Amazon MWAA
Jul 16
2025
2025
AWS Glue now supports zero-ETL integrations from Amazon DynamoDB and eight applications to S3 Tables
May 1
2025
2025
Build end-to-end Apache Spark pipelines with Amazon MWAA, Batch Processing Gateway, and Amazon EMR on EKS clusters
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.