Home icon

46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service

Big Data Blog



This article describes how Confluent migrated its Amazon OpenSearch Service logging infrastructure from general-purpose r6gd instances to purpose-built oi2 optimized instances, achieving significant efficiency gains.

  • Reduced hot tier from 168 r6gd nodes to 90 oi2 nodes (46% reduction) while handling identical ingest volume
  • Halved indexing latency from ~0.18-0.19 ms to ~0.08-0.10 ms across all domains
  • Eliminated write queue backpressure; queued operations dropped from 8-38 to near zero
  • oi2 instances use local NVMe storage with synchronous S3 replication and physical segment replication for efficiency
  • Achieved ~18% cost savings at on-demand pricing despite higher per-node cost through consolidation
  • Used measure-first approach: Phase 1 swapped instance types at constant node count, Phase 2 iteratively rightsized based on heap pressure

The migration demonstrates that optimizing efficiency per unit of work, rather than instance unit price, drives both performance and cost improvements for ingest-bound OpenSearch workloads.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 10
2026
Amazon ElastiCache Serverless now supports vector, full-text and hybrid searches plus aggregations
Oct 12
2026
Amazon OpenSearch Service now supports index-level encryption
Oct 23
2026
Build RAG-ready indexes from Amazon S3 with Amazon OpenSearch intelligent ingestion
Dec 2
2025
Amazon OpenSearch Service adds GPU-accelerated and auto-optimized vector indexes

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.