This article details how Buildkite uses Amazon MSK and Apache Flink to power Test Engine, their analytics platform for CI/CD test data at massive scale.

• Buildkite processes 50 billion test executions monthly, 500K events/second peak ingestion
• Original Rails/PostgreSQL architecture couldn't sustain growth; required complete re-architecture
• Amazon MSK handles high-throughput ingestion (5-100 MB/sec normally, up to 1 GB/sec peak)
• Apache Flink performs stateful stream processing: flaky test detection, metadata enrichment, data routing
• Dual-write strategy enabled safe migration from legacy to new streaming architecture
• Operational improvements: 60% Flink workload reduction, retired key-value store, halved PostgreSQL capacity
• Customers now get interactive analytics across 70 billion records in seconds, not hours
• Real-time log streaming enables developers to diagnose failures before builds complete

Buildkite demonstrates that streaming infrastructure is essential, not optional, for multi-tenant SaaS platforms operating at scale. Managed services eliminate operational complexity while enabling real-time, flexible analytics.