ReleaseBytes
databricks Databricks Blog ·

Apache Spark 4.2 Enhances AI Analytics, Data Pipelines, and Usability

blogaidatabricksengineer
feature

Apache Spark 4.2 introduces significant updates, including metric views for governed business definitions, Spark Connect for remote execution, and enhanced Python integration with Arrow. These changes aim to provide AI-native analytics, improve data freshness through features like Auto CDC and Real-Time Streaming, and simplify Spark's use across various applications and services. The release is now available in Databricks Runtime 19 Beta, benefiting engineers and architects working with large-scale data and AI workloads.

  • →Metric Views for Governed Business Definitions
  • →Spark Connect for Remote Execution
  • →AI-Native Analytics with New SQL Primitives
  • →Declarative Pipelines and Auto CDC for Data Movement
  • →Improved PySpark and Python Integration
Features (4) ›
  • Metric Views for Governed Business Definitions

    Spark 4.2 introduces metric views, a native semantic layer in Spark SQL, allowing teams to define business metrics once for consistent use across dashboards, reports, applications, and AI tools. This feature ensures correct aggregation semantics and provides a single source of truth for AI applications and analytics platforms.

  • Spark Connect for Remote Execution

    The introduction of Spark Connect separates the client from the Spark server using gRPC and Arrow, enabling Spark to be embedded in notebooks, services, developer tools, and AI applications without requiring a full Spark runtime on the client. Applications can call Spark from their own runtime, with analysis, optimization, and execution managed on the server.

  • AI-Native Analytics with New SQL Primitives

    Spark 4.2 adds new SQL primitives for AI-native analytics, including vector similarity search, ranking, and time-series analysis functions like NEAREST BY. It also introduces built-in GEOMETRY and GEOGRAPHY types with ST_* functions for location-aware analytics, enabling richer data analysis directly within Spark SQL.

  • Declarative Pipelines and Auto CDC for Data Movement

    Spark 4.2 enables SQL scripts to DECLARE, OPEN, FETCH, and CLOSE cursors for row-by-row processing. The release also introduces Auto CDC, Data Source V2 enhancements like row-level operations and transactions, and Real-Time Mode in Structured Streaming to simplify reliable processing of continuously changing data.

Enhancements (2) ›
  • Improved PySpark and Python Integration

    Spark 4.2 enhances PySpark compatibility with improvements to RDD API, DataFrame inputs, debuggability, and YARN cluster-mode support. Arrow-optimized Python UDF execution is enabled by default, and support for Pandas 3 and Arrow UDFs reduces friction for Python workloads. Python Data Sources allow custom batch and streaming connectors, with improved profiling for tuning and operation.

  • Enhanced SQL Function and View Resolution

    Spark 4.2 provides unambiguously invoked built-in functions by qualifying them with SYSTEM.BUILTIN and allows fully qualified temporary views with SYSTEM.SESSION to disambiguate from user-defined functions. SQL search path support with SET PATH simplifies the resolution of tables, functions, and variables across namespaces.

Read the original announcement →

https://www.databricks.com/blog/introducing-apache-spark-42

Related releases