ReleaseBytes
aws AWS Big Data Blog ·

AWS Glue 6.0 Adds Support for Iceberg v3 Geospatial and Variant Types

blogdataawsgaengineer
feature announcement

AWS Glue 6.0, powered by Apache Spark 4.1, now supports Apache Iceberg v3, introducing new column-level capabilities for data lakes. Key additions include native geospatial types (GEOMETRY and GEOGRAPHY), nanosecond-precision timestamps, and a VARIANT type for semi-structured data. These features eliminate common workarounds for diverse data types, improving query performance, reducing complexity, and easing maintenance burdens. This update primarily benefits organizations building data lakes with high-frequency event streams and heterogeneous payloads that require precise event sequencing or spatial queries.

  • →Native geospatial data types
  • →Nanosecond-precision timestamps
  • →VARIANT type for semi-structured data
  • →DEFAULT column values
  • →Demonstrating new capabilities with a vehicle telemetry pipeline
Features (4) ›
  • Native geospatial data types

    AWS Glue 6.0 now supports GEOMETRY (with spatial predicates) and GEOGRAPHY types for Iceberg v3 tables, enabling direct spatial queries and eliminating the need for separate coordinate columns or manual spatial math on raw latitude/longitude doubles.

  • Nanosecond-precision timestamps

    The TIMESTAMP_NTZ(9) type is now supported in Iceberg v3 tables via AWS Glue 6.0, allowing for sub-microsecond event ordering. This ensures deterministic sequencing and prevents ties for high-frequency event streams in data lakes.

  • VARIANT type for semi-structured data

    Iceberg v3 in AWS Glue 6.0 introduces a VARIANT type, allowing heterogeneous payloads with varying JSON schemas to be stored in a single column with automatic shredding. This simplifies the extraction of typed metrics from diverse semi-structured data without schema unions or separate tables.

  • DEFAULT column values

    Iceberg v3 tables now support DEFAULT column values in AWS Glue 6.0. This feature automatically applies specified defaults when ingestion writers omit fields, which is useful for tables with multiple data producers that may not populate every column.

Notes (1) ›
  • Demonstrating new capabilities with a vehicle telemetry pipeline

    The post includes a walkthrough of a connected vehicle fleet monitoring pipeline on AWS Glue 6.0, showcasing how to use geospatial, nanosecond, and variant types to detect geofence violations, sequence events precisely, and extract metrics from heterogeneous payloads.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/build-with-geospatial-and-variant-types-in-iceberg-v3-on-aws-glue-6-0/

Related releases