Optimize storage costs in Amazon OpenSearch Service using Zstandard compression

This article discusses the performance benefits of using the Zstandard compression algorithm, which was introduced in OpenSearch v2.9, to optimize storage costs in Amazon OpenSearch Service.

Specifically, the article covers:

• The importance of compression in OpenSearch, including storage efficiency, reduced I/O operations, and environmental impact
• Core concepts related to compression algorithms, including compression ratio, compression speed, and decompression speed
• Details about the Zstandard codec and its two modes: zstd and zstd_no_dict
• How to create an index with the Zstd codec and update codec settings for an existing index
• A benchmark comparing the performance of Zstandard against LZ4 and Zlib codecs, showing that Zstandard provides a better compression ratio than Zlib with a slight gain in write throughput and similar read latency as LZ4
• Limitations of the Zstandard codec, such as potential increased latency for certain queries and incompatibility with k-NN and Security Analytics indexes
• Conclusion that Zstandard provides a good balance between storage size and compression speed, and can be tuned based on use case requirements