Scaling cluster manager and admin APIs in Amazon OpenSearch Service
Big Data Blog
The article discusses performance improvements for cluster management and admin APIs in Amazon OpenSearch Service, specifically for scaling large clusters up to 1,000 nodes with 500,000 shards.
- Identified and resolved four major bottlenecks in cluster state communication and admin API processing
- Implemented a lightweight transport API to check cluster state freshness, reducing leader node CPU usage by 50%
- Introduced local aggregation and filtering of statistics to reduce network throughput and memory pressure
- Added transport layer cancellation for long-running admin API requests
- Created paginated list APIs for metadata retrieval to handle large cluster sizes
These optimizations improve performance and stability for OpenSearch Service clusters of all sizes, ensuring efficient metadata management and reduced load on cluster manager nodes.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2025
2025
2024
2025
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.