Optimize cost and performance for Amazon MWAA

This article provides guidance on how to optimize performance and save costs when using Amazon Managed Workflows for Apache Airflow (Amazon MWAA). It covers best practices for right-sizing Amazon MWAA environments, automating worker autoscaling, configuring logging and CloudWatch metrics, managing AWS Secrets Manager invocations, optimizing DAG code, and selectively stopping and starting environments based on workload demand.

Specifically, the article covers:

• Right-sizing Amazon MWAA environments by monitoring resource utilization, analyzing workload patterns, choosing the appropriate environment class, and fine-tuning configuration parameters.
• Automating worker and web server autoscaling to dynamically adjust resources based on workload demands.
• Configuring appropriate log levels, retention policies, and CloudWatch metrics to balance visibility and cost.
• Using lookup patterns and caching with AWS Secrets Manager to reduce API calls and costs.
• Optimizing DAG code by removing unnecessary imports, writing efficient DAGs, using dynamic DAGs, staggering schedules, optimizing folder parsing, using deferrable operators, and dynamic task mapping.
• Stopping and starting Amazon MWAA environments based on workload requirements to reduce costs when not in use.