Databricks: Data-Native AI Agents Offer Integrated Governance and Security

Databricks promotes running AI agents within its Data Intelligence Platform, treating them as native workloads. This approach unifies governance, security, and observability by integrating agents with Unity Catalog, AI Search, MLflow tracing, Lakebase state management, and AI Gateway.

Production agents require robust state and memory management for conversation history, task progress, and cached results. Running these within the governed platform ensures end-to-end auditability and prevents sensitive agent memory from residing in untrusted external systems.

Running AI agents in separate stacks from enterprise data leads to significant penalties. These include fragmented governance, increased egress costs, higher latency due to multiple hops, and observability gaps, making production deployment risky.

Retroactive governance controls, such as redacting sensitive fields after an agent has accessed data, are insufficient. When agents compute over data, governance must be enforced at query planning time, not after the fact, to prevent data from being shaped by unauthorized access.

Architecting agents to run within the data platform, rather than pulling data out, minimizes latency, reduces costs associated with egress and duplicate storage, simplifies lifecycle management, and provides unified observability and governance.