ReleaseBytes
aws AWS Big Data Blog ·

Accelerate Spark Queries with Iceberg Materialized Views on AWS EMR and Glue

bloganalyticsawsgaengineerhealthcare
feature

AWS has introduced automatic query rewrite for Apache Iceberg materialized views in Spark, enabling significant reductions in query execution time and cost for analytical data lake workloads. This capability allows organizations to optimize complex queries involving multi-table joins and aggregations without modifying existing SQL from BI dashboards or legacy applications. Available with Amazon EMR 7.12.0+ and AWS Glue 5.1+, it automatically redirects matching queries to precomputed MVs to improve performance. This feature helps overcome the challenge of slow, costly queries where manual rewriting is difficult or risky.

  • →Automatic Query Rewrite for Iceberg Materialized Views
  • →How Materialized Views and Rewrites Function
  • →Prerequisites and Configuration
Features (1) ›
  • Automatic Query Rewrite for Iceberg Materialized Views

    This new capability in Apache Spark (on EMR and Glue) automatically analyzes incoming query plans and redirects them to suitable materialized views (MVs) when available, eliminating the need to manually rewrite slow or costly analytical SQL queries. It supports structural rewrites for aggregations and joins, and exact-match rewrites for more complex patterns like window functions.

Notes (2) ›
  • How Materialized Views and Rewrites Function

    Users define SQL queries for MVs, which AWS Glue Data Catalog stores as precomputed Apache Iceberg tables in Amazon S3. Automatic refresh schedules keep these MVs current, detecting source table changes. Query optimization checks MV staleness and uses two matching strategies—structural and exact-match—to find the best fit, ensuring queries run faster without returning stale results.

  • Prerequisites and Configuration

    To utilize automatic query rewrite, users need Amazon EMR 7.12.0+ or AWS Glue 5.1+, with source tables in Apache Iceberg or Parquet format registered in AWS Glue Data Catalog. It requires an S3 bucket for MV data, appropriate IAM/Lake Formation permissions, and enabling the feature via `spark.sql.optimizer.answerQueriesWithMVs.enabled=true` in the Spark session.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/accelerating-spark-queries-with-iceberg-materialized-views/

Related releases