Introducing job queuing to scale your AWS Glue workloads
Big Data Blog
This article discusses a new AWS Glue job queuing feature that helps scale data integration workloads by automatically managing concurrency and retrying jobs when service limits are reached.
Specifically, the article covers:
- Use cases where job queuing is beneficial, such as processing many data sources in parallel or handling event-driven workloads with spikes
- How to enable job queuing for an AWS Glue job via the AWS Glue console
- How job queuing works, moving jobs to a "Waiting" state and retrying them when resources become available, rather than failing immediately
- The service limits covered by job queuing (concurrent job runs, DPUs, IP address exhaustion)
- Considerations like lack of support for Flex jobs and MaxRetries being non-configurable when queuing is enabled
- Conclusion highlighting how job queuing simplifies workload scaling and improves success rates
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Sep 3
2024
2024
AWS Glue now provides job queuing
Jul 17
2025
2025
AWS Glue now supports new workers for larger and memory intensive workloads
Jul 17
2025
2025
Scale your AWS Glue for Apache Spark jobs with R type, G.12X, and G.16X workers
Mar 19
2024
2024
Scale AWS Glue jobs by optimizing IP address consumption and expanding network capacity using a private NAT gateway
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.