How Kaplan, Inc. implemented modern data pipelines using Amazon MWAA and Amazon AppFlow with Amazon Redshift as a data warehouse
Big Data Blog
This article discusses how Kaplan, Inc. implemented modern data pipelines using Amazon MWAA and Amazon AppFlow with Amazon Redshift as a data warehouse.
Specifically, the article covers:
- Solution overview: Using Amazon MWAA as an orchestrator, Amazon AppFlow to migrate data from Salesforce to Amazon S3, and Amazon Redshift as a data warehouse.
- Challenges faced: Automating incremental data loading and synchronization between pipeline stages.
- Implementation details: Creating a custom Airflow operator to control Amazon AppFlow functionalities, and orchestrating the entire data flow within a single DAG in Amazon MWAA.
- Conclusion: The modern data integration platform enabled robust, automated data pipelines, reducing pipeline creation time by 50% and cost per table by 80-90%.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Aug 15
2024
2024
Seamless integration of data lake and data warehouse using Amazon Redshift Spectrum and Amazon DataZone
Feb 25
2025
2025
Building and operating data pipelines at scale using CI/CD, Amazon MWAA and Apache Spark on Amazon EMR by Wipro
Apr 25
2024
2024
Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA
Jan 22
2026
2026
Modernize your data warehouse by migrating Oracle Database to Amazon Redshift with Oracle GoldenGate
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.