ReleaseBytes
aws AWS News Blog ·

AWS Glue 6.0 Now Generally Available with 30% Lower Pricing and Apache Iceberg v3

blogdataawsgaengineer
feature announcement

AWS Glue 6.0 is now generally available, offering a 30% price reduction compared to previous versions and a modernized runtime based on Apache Spark 4.1. This release introduces full support for Apache Iceberg v3, including a VARIANT data type for semi-structured data, and adds native spatial processing and nanosecond-precision timestamps. It also brings Spark declarative pipelines, Arrow-native PySpark UDFs for improved performance, and a real-time streaming mode with single-digit millisecond latency for event processing. These updates aim to simplify ETL authoring, enhance performance, and enable more precise and resilient data processing for data engineers.

  • →AWS Glue 6.0 Generally Available with Lower Pricing and Modernized Runtime
  • →Full Apache Iceberg v3 Support for Data Lakes
  • →Real-time Streaming Mode with Low Latency
  • →Simplified ETL Authoring with Spark Declarative Pipelines
  • →Improved PySpark Performance with Arrow-Native UDFs
Features (3) ›
  • AWS Glue 6.0 Generally Available with Lower Pricing and Modernized Runtime

    AWS Glue 6.0 is now generally available, providing a 30% price reduction and a modernized runtime built on Apache Spark 4.1, Python 3.12, and Scala 2.13. This upgrade aims for faster performance and broader capability support.

  • Full Apache Iceberg v3 Support for Data Lakes

    This release includes complete Apache Iceberg v3 specification support, built on Iceberg 1.11.0. Key additions include the VARIANT data type with shredding for faster query performance on semi-structured data, Geometry and Geography types for GIS analytics, and nanosecond-precision timestamps for high-frequency workloads.

  • Real-time Streaming Mode with Low Latency

    A new real-time streaming mode, built on Spark 4.1's Real-Time Mode, achieves single-digit millisecond latency for stateless streaming use cases. This enables real-time event processing, low-latency data transformation, and time-sensitive data routing.

Enhancements (2) ›
  • Simplified ETL Authoring with Spark Declarative Pipelines

    AWS Glue 6.0 incorporates Spark Declarative Pipelines from Spark 4.1, simplifying ETL authoring. Data engineers can now declare transformations, allowing the engine to automatically determine execution order and optimization, reducing pipeline development complexity.

  • Improved PySpark Performance with Arrow-Native UDFs

    Arrow-native execution is introduced for Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs). This eliminates serialization overhead between Python and the JVM, significantly improving PySpark performance for complex data transformations.

Read the original announcement →

https://aws.amazon.com/blogs/aws/aws-glue-6-0-now-available-with-30-lower-price-and-full-apache-iceberg-v3-support/

Related releases