A serverless pipeline with Amazon OpenSearch Ingestion efficiently routes, filters, and detects anomalies in streaming operational data, minimizing costs by storing only anomalous data in OpenSearch Service and archiving full data cost-effectively in Amazon S3 for querying with Amazon Athena.

<div>
<p>This article discusses how to build a cost-efficient infrastructure for operational analytics using Amazon OpenSearch Ingestion pipelines. It covers a solution to ingest, route, filter, and analyze high-volume time-series data from sources like VPC flow logs and perform anomaly detection before storing the data in OpenSearch Service.</p>
<p>Specifically, the article covers:</p>
<ul>
<li>Using conditional routing with OpenSearch Ingestion to separate logs with specific attributes and store in OpenSearch Service or archive in S3</li>
<li>Performing in-stream anomaly detection with OpenSearch Ingestion to identify anomalous events</li>
<li>Storing raw data in S3 for archival using the Parquet format</li>
<li>Querying archived data in S3 using Amazon Athena</li>
<li>An example solution using VPC flow logs to demonstrate the approach</li>
<li>Conclusion highlighting the cost benefits of this architecture</li>
</ul>
</div>


Related articles