Home icon

Orchestrating big data processing with AWS Step Functions Distributed Map

Compute Blog

AWS Step Functions has introduced a new Distributed Map feature for processing big data with AWS Athena and Parquet files, enabling developers to orchestrate large-scale data processing workflows.

  • Supports direct processing of Athena data manifest and Parquet files
  • Can run up to 10,000 parallel child workflow executions
  • Provides new Amazon CloudWatch metrics for tracking workflow performance
  • Enables processing of IoT sensor data with anomaly detection
  • Integrates with AWS Lambda, Amazon Kinesis Data Firehose, and S3

The new feature allows developers to streamline big data dataset transformations by processing data entries in parallel, with improved visibility and control over distributed data processing workflows.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 4
2025
Optimizing nested JSON array processing using AWS Step Functions Distributed Map
Oct 24
2025
Processing Amazon S3 objects at scale with AWS Step Functions Distributed Map S3 prefix
Nov 26
2025
Orchestrating large-scale document processing with AWS Step Functions and Amazon Bedrock batch inference
Sep 18
2025
AWS Step Functions expands data source options and improves observability for Distributed Map

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.