Querying raw log data using SQL and PPL with the optimized engine in Amazon OpenSearch Service

This article demonstrates how to run fast analytical queries directly against raw log and trace data in Amazon OpenSearch Service using PPL and SQL with the new optimized engine.

• Optimized engine stores data in columnar Apache Parquet format and executes queries through Apache DataFusion with Apache Calcite planning
• Query planner routes operations to columnar engine for aggregations and analytical scans, or inverted index for selective search
• Practical incident investigation walkthrough shows five queries: assessing error scope, checking host concentration, measuring latency distribution, calculating error rates, and sizing fleet footprint
• Supports both PPL and SQL languages interchangeably in Query Workbench and Discover interfaces
• Compresses observability data up to 70% more efficiently than General Purpose engine, enabling cost-effective raw data retention
• Available in OpenSearch 3.5 or later with Observability use case selection during domain setup

The optimized engine enables direct querying of raw observability data without precomputation, supporting complex analytical investigations at billion-document scale.