Orchestrating large-scale document processing with AWS Step Functions and Amazon Bedrock batch inference

This article demonstrates a scalable, serverless pipeline for large-scale document processing using AWS services to extract text, generate metadata, and build searchable knowledge bases.

• Uses Step Functions Distributed Map for parallel Amazon Textract PDF text extraction
• Amazon Bedrock batch inference (50% discount) extracts structured metadata like code availability
• Processes 500 research papers automatically with callback patterns for asynchronous job handling
• Amazon Bedrock Knowledge Bases with OpenSearch Serverless enables searchable document repositories
• Metadata filtering allows targeted queries on extracted attributes (reproducibility, datasets, code repos)
• Complete AWS CDK implementation provided with EventBridge automation and SNS notifications

This solution enables organizations to transform static document collections into intelligent, searchable RAG repositories cost-effectively while handling enterprise-scale volumes.