Building a scalable personalized recommendation system on AWS: From batch to real-time

This article describes Amazon's scalable personalized recommendation system architecture that processes millions of customer signals and serves tens of thousands of requests per second across global marketplaces.

• Centralized data lake with Lake Formation provides governed cross-account access to golden datasets, eliminating per-pipeline data extraction and ensuring consistency.
• Amazon MWAA orchestrates complex multi-step workflows with reusable custom operators for Athena, Glue, and SageMaker, reducing pipeline development from weeks to days.
• AWS Glue handles distributed data transformations with auto-scaling worker pools; Amazon Athena queries the data lake; SageMaker trains models, runs batch inference, and performs vector search.
• Amazon DynamoDB serves pre-computed recommendations at single-digit millisecond latency with TTL for automatic cleanup of stale data.
• Amazon MemoryDB extends the system with real-time vector similarity search paired with SageMaker endpoints for on-demand embeddings, capturing fast-moving signals without replacing batch foundation.
• Batch handles slow-moving signals while real-time handles fast-moving ones; both share models, data lake, and serving infrastructure.

The architecture demonstrates how batch and real-time approaches complement each other, with batch providing efficiency and real-time providing freshness for personalized product discovery at scale.