How Smartsheet built Real-time Dynamic Filtering on Apache Flink reducing $40K/month in messaging costs

This article describes how Smartsheet built a Real-time Dynamic Filtering (RDF) system on Amazon Managed Service for Apache Flink to reduce messaging costs and improve latency for live collaboration features.

• Smartsheet processes hundreds of thousands of events per second for live collaboration, workflows, and notifications
• Previous SNS/SQS architecture discarded over 90% of events after expensive per-event DynamoDB lookups
• SNS filter policy changes took up to 15 minutes to propagate, making dynamic filtering impractical
• RDF uses Flink's KeyedCoProcessFunction to join filter policies from DynamoDB Streams with event data streams
• Filter policies stored in RocksDB state backend; "subscribe all" policies use broadcast state across tasks
• Reduced monthly costs by $40,000+ (SNS transfer, SQS, DynamoDB, Lambda combined)
• Improved live collaboration latency by 1.8x; filter propagation now sub-second (p95 ≤1s)
• Architecture designed as shared platform for other teams (workflows, notifications)

Smartsheet's RDF demonstrates how stream processing can replace costly fan-out architectures, achieving significant cost savings and latency improvements through intelligent state management and dynamic filtering.