46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service
Big Data Blog
This article describes how Confluent migrated its Amazon OpenSearch Service logging infrastructure from general-purpose r6gd instances to purpose-built oi2 optimized instances, achieving significant efficiency gains.
- Reduced hot tier from 168 r6gd nodes to 90 oi2 nodes (46% reduction) while handling identical ingest volume
- Halved indexing latency from ~0.18-0.19 ms to ~0.08-0.10 ms across all domains
- Eliminated write queue backpressure; queued operations dropped from 8-38 to near zero
- oi2 instances use local NVMe storage with synchronous S3 replication and physical segment replication for efficiency
- Achieved ~18% cost savings at on-demand pricing despite higher per-node cost through consolidation
- Used measure-first approach: Phase 1 swapped instance types at constant node count, Phase 2 iteratively rightsized based on heap pressure
The migration demonstrates that optimizing efficiency per unit of work, rather than instance unit price, drives both performance and cost improvements for ingest-bound OpenSearch workloads.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2026
2025
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.