Bulk Executor for Amazon DynamoDB, Part 2: Lessons we learned the hard way

This article shares design lessons and technical solutions learned while building Bulk Executor, an open-source tool for running bulk operations against Amazon DynamoDB tables using AWS Glue.

• Selected AWS Glue for serverless parallelism with built-in DynamoDB connector and Spark DataFrame capabilities
• Implemented cost estimation for reads and writes using table metadata and pricing APIs with heuristics for provisioned mode
• Solved job termination by catching interrupt signals to cleanly stop both client and remote AWS Glue jobs
• Required point-in-time recovery (PITR) on tables for mutating commands to enable rollback of unintended changes
• Switched to CloudWatch Live Tail for real-time output but handled 500 events/second limit by writing large results to S3
• Built distributed rate limiting for direct API calls to prevent throttling across multiple workers
• Designed loosely-coupled command architecture enabling community contributions like diff, sql, and load commands
• Used AI tooling to generate stress tests comparing AWS Glue 4 and 5 connectors, enabling confident version upgrades
• Created AI linter to verify consistency across three different formats of IAM permission requirements

The tool demonstrates how addressing small usability issues—Live Tail sampling, incomplete error messages, unbounded rates—significantly improves user experience when building on AWS Glue.