ReleaseBytes
aws AWS Big Data Blog ·

AWS Glue 6.0 empowers real-time event pipelines with Spark RTM and Variant columns

blogdataawsgaengineer
feature

AWS Glue 6.0, built on Apache Spark 4.1 with Apache Iceberg v3 support, now introduces Spark Real-Time Mode (RTM), Variant columns with shredding, and Arrow-native user-defined functions (UDFs). These new capabilities address challenges like schema heterogeneity, high JSON parsing overhead, and the need for mixed real-time and batch processing speeds within a single serverless pipeline. Engineers and architects building event-driven systems can leverage these features to process multi-schema streams efficiently and perform high-speed anomaly detection alongside deep historical analysis. The article demonstrates building a market risk pipeline for financial services, with a CloudFormation template provided for deployment.

  • →Spark Real-Time Mode (RTM) for sub-second latency
  • →Variant columns and shredding with Apache Iceberg v3
  • →Arrow-native User-Defined Functions (UDFs) for vectorized processing
  • →CloudFormation template for real-time event pipeline example
Features (3) ›
  • Spark Real-Time Mode (RTM) for sub-second latency

    AWS Glue 6.0 introduces Spark Real-Time Mode, which eliminates micro-batch boundaries to allow records to flow continuously through pipelines. This enables sub-second latency for critical tasks like flagging anomalies or high-risk events.

  • Variant columns and shredding with Apache Iceberg v3

    The platform now supports Iceberg v3 Variant columns, allowing heterogeneous semi-structured data to be stored in a single column without predefined schemas. Variant shredding automatically extracts observed fields into typed Parquet columns at write time, optimizing downstream analytical queries.

  • Arrow-native User-Defined Functions (UDFs) for vectorized processing

    AWS Glue 6.0 incorporates Arrow-native UDFs that process data as vectorized columnar batches. This eliminates traditional row-by-row serialization overhead between JVM and Python, significantly accelerating Python-based computations like complex risk calculations.

Notes (1) ›
  • CloudFormation template for real-time event pipeline example

    A CloudFormation template is provided to deploy a multi-layer financial market risk pipeline that demonstrates the new capabilities of AWS Glue 6.0. The template provisions an MSK cluster, S3 bucket, Glue database, IAM roles, VPC networking, and several Glue jobs.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/build-a-real-time-event-pipeline-with-spark-real-time-mode-on-aws-glue-6-0/

Related releases