ReleaseBytes
aws AWS Big Data Blog ·

Amazon OpenSearch Service adds SQL and PPL querying for raw log data with optimized engine

blogobservabilityawsgaengineer
feature

Amazon OpenSearch Service now offers an optimized engine that enables direct querying of raw log and trace data using SQL and Piped Processing Language (PPL). This eliminates the need for data movement or reshaping, facilitating fast analytical queries for observability and incident investigation. The engine stores data in columnar Apache Parquet format and uses Apache DataFusion for vectorized execution, alongside an inverted index for optimal query routing. This capability is designed for Site Reliability Engineers (SREs) and others performing log analytics at billion-document scale.

Features (1) ›
  • Optimized Engine for Direct SQL/PPL Querying on Raw Log Data

    The new optimized engine in Amazon OpenSearch Service allows users to run fast analytical queries directly against raw log and trace data using SQL and PPL. It stores data in columnar Apache Parquet format and executes queries via Apache DataFusion, leveraging an inverted index alongside the columnar store to route operations optimally for analytical scans or selective searches. This enables complex aggregations, filters, and scans over billions of events without precomputed structure, streamlining log analytics and root cause analysis workflows.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/querying-raw-log-data-using-sql-and-ppl-with-the-optimized-engine-in-amazon-opensearch-service/

Related releases