Databricks Introduces Real-Time Mode for Spark Structured Streaming
Databricks has launched Real-Time Mode (RTM) for Apache Spark Structured Streaming, enabling sub-second latency for operational workloads like fraud detection and IoT monitoring. This feature simplifies the tech stack by eliminating the need for separate real-time processing engines, leveraging existing Spark expertise and APIs with a simple trigger configuration change. RTM is available now, offering a unified platform for both analytical and operational data processing to reduce complexity and costs.
- →Sub-second latency with Spark Structured Streaming Real-Time Mode
- →Simplified stack and reduced operational complexity
- →Developer-friendly integration with existing Spark expertise
- →Operational Guardrail Pipeline pattern demonstrated for anomaly detection
- →At-least-once delivery guarantees for Kafka sink
Features (1) ›
- Sub-second latency with Spark Structured Streaming Real-Time Mode
Apache Spark Structured Streaming now offers Real-Time Mode (RTM), achieving millisecond-level latency for operational workloads by processing events as they arrive, enabling sub-second response times previously requiring specialized engines.
Enhancements (2) ›
- Simplified stack and reduced operational complexity
RTM eliminates the need for separate technology stacks for real-time processing, allowing organizations to manage, monitor, and troubleshoot a single Spark-based platform for both analytical and operational workloads.
- Developer-friendly integration with existing Spark expertise
Enabling RTM requires only a change in the trigger configuration, allowing developers familiar with Structured Streaming to unlock millisecond-level latency without code rewrites or complex migrations.
Notes (2) ›
- Operational Guardrail Pipeline pattern demonstrated for anomaly detection
A reusable pattern for operational workloads is established, demonstrating anomaly detection on Ethereum blockchain transactions to flag invalid data quality and potential data leakage in real-time.
- At-least-once delivery guarantees for Kafka sink
Real-Time Mode with a Kafka sink provides at-least-once delivery guarantees, requiring downstream consumers to handle potential duplicates through idempotent writes or deduplication logic.
https://www.databricks.com/blog/ultra-fast-anomaly-detection-using-apache-spark-real-time-mode
Related releases
- Lakebase default storage quota increased to 32 TB Databricks Release Notes ·
- Databricks Lakeflow Connect: Zerobus Ingest to Streaming Tables GA Databricks Release Notes ·
- Databricks Introduces Genie Code for AI Runtime in Public Preview Databricks Release Notes ·
- Databricks Genie One Introduces Desktop App, Enhanced Collaboration, and Expanded Context Databricks Blog ·
- Databricks Lakeflow Designer August Release Adds Unique Operator and Canvas Enhancements Databricks Release Notes ·
- Databricks Introduces Unified Trace Table for AI Gateway Monitoring (Beta) Databricks Release Notes ·