ReleaseBytes
aws AWS Big Data Blog ·

AWS Glue 6.0 Launched with Apache Spark 4.1, 30% Price Reduction, and Iceberg V3 Support

blogdataawsgaengineer
feature announcement

AWS has launched Glue 6.0, a new version of its serverless data integration service, designed to deliver faster and more cost-effective data processing. This release features a 30% price reduction and upgrades to Apache Spark 4.1.1, Python 3.13, and Scala 2.13, enhancing performance and developer experience. It also introduces comprehensive support for Apache Iceberg V3 specifications, including VARIANT data types, deletion vectors, and row lineage tracking. These improvements help organizations build modern lakehouse architectures with lower operational overhead and improved price-performance.

  • →Enhanced Apache Iceberg V3 Support
  • →New Features for Developer Experience and Performance
  • →Upgraded Runtime with Apache Spark 4.1.1
  • →30% Reduction in AWS Glue Usage Pricing
Features (2) ›
  • Enhanced Apache Iceberg V3 Support

    AWS Glue 6.0 supports the latest versions of Iceberg, Hudi, and Delta Lake, with a focus on Apache Iceberg V3 specifications for enterprise use. This includes the VARIANT data type with automatic shredding for semi-structured data, deletion vectors for efficient row-level updates, and support for geospatial data types and nanosecond-precision timestamps.

  • New Features for Developer Experience and Performance

    The release introduces new capabilities like Spark Declarative Pipelines, Real-Time Mode for streaming with sub-second latencies, and Python virtual environments with S3 caching. These additions are designed to further improve performance and enhance the developer experience for data integration workloads.

Enhancements (2) ›
  • Upgraded Runtime with Apache Spark 4.1.1

    AWS Glue 6.0 now runs on an AWS-optimized Apache Spark 4.1.1, a significant upgrade from Spark 3.5, delivering improved performance for data engineering and real-time streaming. It also updates to Python 3.13, Scala 2.13, and AWS SDK for Java 2.x, bringing interpreter changes, library updates, and security enhancements.

  • 30% Reduction in AWS Glue Usage Pricing

    AWS Glue 6.0 reduces the usage rate by 30% per DPU-hour compared to Glue 5.1. Combined with performance improvements, this offers substantial effective cost savings for large-scale ETL and batch workloads, demonstrating up to 36% better price performance in benchmarks.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/introducing-aws-glue-6-0-for-apache-spark/

Related releases