How to stream PostgreSQL changes to Amazon S3 with AWS Fargate

This article demonstrates how to build an event-driven change data capture (CDC) pipeline that streams PostgreSQL row-level changes to Amazon S3 in near real time using AWS Fargate, EventBridge, SQS, and Lambda.

• Pipeline uses PostgreSQL's Write-Ahead Log (WAL) and logical replication to capture changes without polling
• Fargate CDC reader continuously polls replication slot and publishes events to EventBridge
• EventBridge routes events to SQS for buffering; Lambda consumes batches and writes partitioned JSON to S3
• Supports private subnet deployments with VPC endpoints and no internet gateway exposure
• Deploy entire pipeline with single CloudFormation template; costs approximately $9/month for Fargate at low volume
• Includes operational guidance for monitoring replication lag, handling Fargate restarts, and managing Lambda failures
• Supports three output plugins: pgoutput (binary), wal2json (structured JSON), and test_decoding (plain text)

This serverless CDC approach eliminates Kafka dependencies, provides pay-per-use pricing, and simplifies operational complexity compared to traditional batch jobs or polling mechanisms.