ReleaseBytes
aws AWS Big Data Blog ·

AWS Glue 6.0 Now Available with Apache Spark 4.1 and AI-Powered Upgrade Tool

blogaiawsgaengineer
feature

AWS has released Glue 6.0, which upgrades to Apache Spark 4.1 and Python 3.13, offering up to 36% better price performance and new features like Iceberg v3 and real-time mode. To address breaking changes when migrating existing PySpark ETL jobs, AWS Glue now provides a generative AI-powered upgrade analysis tool. This tool automatically identifies and resolves incompatibilities, such as removed configurations or Python library issues, and validates the changes with data quality checks. It streamlines the migration process for developers, ensuring correctness and efficiency while moving to the latest Glue runtime.

  • →AWS Glue 6.0 Enhancements and Runtime Upgrade
  • →AI-Powered Upgrade Tool for PySpark Job Migration
Features (2) ›
  • AWS Glue 6.0 Enhancements and Runtime Upgrade

    AWS Glue 6.0 is now available, featuring Apache Spark 4.1 and Python 3.13, and delivering up to 36% better price performance compared to Glue 5.1. It introduces Apache Iceberg v3 with VARIANT Shredding, Spark Declarative Pipelines, Real-Time Mode for low-latency streaming, and Arrow-native Python UDFs for improved performance.

  • AI-Powered Upgrade Tool for PySpark Job Migration

    A new generative AI-powered analysis tool helps automate the upgrade of existing PySpark ETL jobs to AWS Glue 6.0 by iteratively identifying and resolving incompatibilities. It handles issues such as removed Spark 4.1 configurations, Python module version conflicts, and stricter type casting due to ANSI SQL mode being enabled by default. The tool applies fixes, validates job execution, and performs data quality checks to ensure successful migration.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/upgrade-aws-glue-jobs-to-glue-6-0-with-ai-powered-spark-upgrades/

Related releases