How Moeve standardized dbt runs across data lakes with Amazon Athena

This article describes how Moeve built a centralized, serverless dbt launcher on Amazon Athena to standardize data transformations across multiple AWS accounts and teams.

• Decoupled dbt runs from orchestration using AWS Step Functions, AWS Fargate, and Amazon EventBridge for loose coupling
• Chose Amazon Athena as the default processing engine for serverless, pay-per-scan pricing with no infrastructure management
• Stored run parameters in Amazon DynamoDB instead of pipeline code, enabling configuration changes without deployments
• Implemented idempotent incremental processing using Apache Iceberg with dynamic partition predicates to reduce query costs
• Reduced new project onboarding from days to 15 minutes with configuration-only setup and local CI validation using DuckDB
• Achieved per-project and per-run cost attribution by running transformations in data lake accounts via cross-account IAM roles

The solution demonstrates how clear architectural boundaries—parameters in, events out—enable standardization while maintaining flexibility for independent evolution of orchestration and processing layers.