Building Durable Agents with Temporal and Lakebase on Databricks
Databricks details an architecture for building durable, long-running cloud agents using Temporal for resilient execution and Lakebase Postgres for queryable operational state. This approach addresses challenges like worker failures, long waits, and reliable retries, ensuring agents preserve progress and state independently of their execution environment. The solution integrates with Unity Catalog for governed policy and audit trails, making it relevant for engineers and architects developing resilient AI agents on Databricks. A personal-loan underwriting agent serves as a reference implementation, showcasing operational visibility and runtime governance.
- →Architecture for Durable, Long-Running AI Agents
- →Integration with Unity Catalog for Governed Data and Audit
- →Reference Implementation: Personal-Loan Underwriting Agent
- →Complementary Roles of Temporal and Lakebase
Notes (4) ›
- Architecture for Durable, Long-Running AI Agents
This article presents an architecture combining Temporal for durable control flow and Lakebase Postgres for queryable application state, enabling the creation of resilient cloud agents. This design ensures agents can recover from worker failures, manage long waits for external systems or human input, and safely retry operations without duplicating side effects.
- Integration with Unity Catalog for Governed Data and Audit
The solution facilitates seamless integration with Databricks Unity Catalog, allowing agents to read governed policy via synced tables in Lakebase. Additionally, Lakebase Change Data Feed can publish operational changes and decisions back to Unity Catalog-managed Delta history tables, providing a comprehensive audit trail for agent activities.
- Reference Implementation: Personal-Loan Underwriting Agent
A personal-loan underwriting agent is provided as a reference implementation, demonstrating how the architecture handles complex, multi-step workflows. This example showcases the system's ability to gather evidence, apply policy, produce recommendations, and durably wait for human review, even across worker restarts and policy changes.
- Complementary Roles of Temporal and Lakebase
Temporal manages the agent's durable control flow and event history, enabling deterministic replay and recovery after failures. Lakebase stores the application's current state, messages, evidence, and metrics in a queryable Postgres database, providing essential operational visibility to users and downstream applications.
https://www.databricks.com/blog/build-durable-agents-temporal-and-lakebase
Related releases
- Verkada Connector Now in Beta for Databricks Lakeflow Connect Databricks Release Notes ·
- Databricks Unity Catalog Gains ABAC DENY Policies (Beta) Databricks Release Notes ·
- Anysphere Audit Logs Connector for Databricks Lakeflow Connect (Beta) Databricks Release Notes ·
- Databricks Terraform Provider v1.131.0 Adds IAM and PostgreSQL Snapshot Resources Terraform Databricks Provider Releases ·
- Databricks Java SDK v0.153.0 Introduces API Enhancements and Breaking Changes Databricks Java SDK Releases ·
- Databricks SDK for Python v0.136.0 Adds New Domain and ML Features Databricks Python SDK Releases ·