Databricks Variant data type is now generally available for semi-structured data

The Variant data type, developed with the Delta and Spark communities and extended to Parquet and Iceberg, is now generally available on Databricks. It enables flexible ingestion of semi-structured data without updating pipelines and maintains structured-like query performance.

Variant Shredding, now also generally available, is a performance optimization that uses Databricks' Predictive Optimization to improve query performance on Variant data. It automatically identifies critical shredded fields and collects statistics to enhance file skipping, leading to significantly faster reads.

Variant can be ingested using Auto Loader for incremental processing of files from object storage or Zerobus, a managed ingestion service. Data written using Variant is compatible with Delta and Iceberg, allowing interoperability within the lakehouse. Genie Code in Lakeflow Pipelines Editor can generate Auto Loader pipelines using natural language.

Variant is broadly integrated into Databricks, including Auto Loader and Spark Declarative Pipelines for data workloads, and Agent Bricks and AI Functions for AI workloads. This allows for seamless use across various aspects of the platform.

Over 5,000 teams use Variant to ingest data from streaming sources, API JSON payloads, and schemaless databases. It is particularly useful for managing unpredictable schema changes from upstream sources without breaking downstream pipelines.

Databricks users execute over 500 million Variant queries per month on more than 160 TB of data. Variant enables query speeds up to 30x faster than storing JSON as a string and nearly 4x faster than unshredded Variant, as demonstrated by its use in querying security logs at petabyte scale.