Home icon
Introducing Amazon MSK as a source for Amazon OpenSearch Ingestion

Blog



This article introduces Amazon MSK as a data source for Amazon OpenSearch Ingestion, a serverless managed service for real-time data collection and ingestion into OpenSearch Service.

  • OpenSearch Ingestion now supports Amazon MSK as a source for streaming data ingestion
  • Eliminates need for Lambda, custom code, Kafka Connect, or Logstash for MSK-to-OpenSearch pipelines
  • Requires MSK multi-VPC connectivity and appropriate IAM permissions for both services
  • Supports reading from up to four topics per pipeline with JSON or plaintext deserialization
  • Auto-scaling capability based on CPU usage and pending records in pipeline
  • Acknowledgments setting ensures data delivery before committing MSK offsets
  • Recommended OCU configuration depends on topic partition count for optimal parallel processing

This integration simplifies operational analytics workflows by providing a fully managed, serverless solution for ingesting high-volume streaming data from MSK into OpenSearch without manual consumer management.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.