Introducing Amazon MWAA support for the Airflow REST API and web server auto scaling
Big Data Blog
The article introduces two new features in Amazon Managed Workflows for Apache Airflow (Amazon MWAA): Airflow REST API support and web server auto-scaling.
Specifically, the article covers:
- Airflow REST API support: This feature enables programmatic interaction with Airflow resources like connections, DAGs, DAGRuns, and Task instances. It demonstrates how to authenticate with the REST API, invoke a DAG run, and validate the result.
- Web server auto-scaling: This feature allows Amazon MWAA to automatically scale the number of web servers based on CPU utilization and active connection count, accommodating increased demand. It shows how to set up auto-scaling and trigger it programmatically for load testing.
- Conclusion: These features enable building robust, scalable, and flexible data orchestration pipelines, streamlining data engineering operations and unlocking new possibilities with Amazon MWAA.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.