A customer’s journey with Amazon OpenSearch Ingestion pipelines
Big Data Blog
The article discusses a customer's journey of using Amazon OpenSearch Service and OpenSearch Ingestion pipelines for log management and analysis.
Specifically, the article covers:
- The customer's initial solution using Logstash on EC2 for log enrichment and Amazon OpenSearch Service for log storage and analysis, and the challenges faced in maintaining this setup.
- Evaluation of alternative solutions like Splunk and the decision to go with OpenSearch Ingestion pipelines.
- Initial challenges faced with OpenSearch Ingestion pipelines due to misconfigurations and performance issues.
- Optimized design using selective log ingestion, Index State Management, and Amazon Athena for detailed log analysis.
- Performance and cost comparison of the different solutions, highlighting the benefits of the optimized OpenSearch Ingestion pipelines solution.
- Future plans to explore the new Amazon OpenSearch Service zero-ETL integration with Amazon S3.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Jun 12
2024
2024
Ingest and analyze your data using Amazon OpenSearch Service with Amazon OpenSearch Ingestion
Apr 22
2025
2025
Introducing a guided visual pipeline builder for Amazon OpenSearch Ingestion
Aug 20
2025
2025
Build enterprise-scale log ingestion pipelines with Amazon OpenSearch Service
Apr 22
2025
2025
Accelerate data pipeline creation with the new visual interface in Amazon OpenSearch Ingestion
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.