ReleaseBytes
databricks Databricks Blog ·

Databricks Introduces Real-Time Mode for Spark Structured Streaming

bloginfradatabricksengineer
feature announcement

Databricks has launched Real-Time Mode (RTM) for Apache Spark Structured Streaming, enabling sub-second latency for operational workloads like fraud detection and IoT monitoring. This feature simplifies the tech stack by eliminating the need for separate real-time processing engines, leveraging existing Spark expertise and APIs with a simple trigger configuration change. RTM is available now, offering a unified platform for both analytical and operational data processing to reduce complexity and costs.

  • →Sub-second latency with Spark Structured Streaming Real-Time Mode
  • →Simplified stack and reduced operational complexity
  • →Developer-friendly integration with existing Spark expertise
  • →Operational Guardrail Pipeline pattern demonstrated for anomaly detection
  • →At-least-once delivery guarantees for Kafka sink
Features (1) ›
  • Sub-second latency with Spark Structured Streaming Real-Time Mode

    Apache Spark Structured Streaming now offers Real-Time Mode (RTM), achieving millisecond-level latency for operational workloads by processing events as they arrive, enabling sub-second response times previously requiring specialized engines.

Enhancements (2) ›
  • Simplified stack and reduced operational complexity

    RTM eliminates the need for separate technology stacks for real-time processing, allowing organizations to manage, monitor, and troubleshoot a single Spark-based platform for both analytical and operational workloads.

  • Developer-friendly integration with existing Spark expertise

    Enabling RTM requires only a change in the trigger configuration, allowing developers familiar with Structured Streaming to unlock millisecond-level latency without code rewrites or complex migrations.

Notes (2) ›
  • Operational Guardrail Pipeline pattern demonstrated for anomaly detection

    A reusable pattern for operational workloads is established, demonstrating anomaly detection on Ethereum blockchain transactions to flag invalid data quality and potential data leakage in real-time.

  • At-least-once delivery guarantees for Kafka sink

    Real-Time Mode with a Kafka sink provides at-least-once delivery guarantees, requiring downstream consumers to handle potential duplicates through idempotent writes or deduplication logic.

Read the original announcement →

https://www.databricks.com/blog/ultra-fast-anomaly-detection-using-apache-spark-real-time-mode

Related releases