Top 5 Databricks System Table Queries for Cost Management
Databricks highlights five SQL queries utilizing system tables like `system.billing.usage` to help users understand and manage their platform costs. These queries provide insights into daily spend by product, SQL warehouse cost trends, and tag-based cost attribution. They also enable daily cost anomaly detection and future spend forecasting using SQL AI Functions, empowering platform administrators to monitor and optimize their Databricks expenditure effectively.
- →Query Daily Spend by Product
- →Analyze SQL Warehouse Cost Trends
- →Attribute Costs Using Custom Tags
- →Detect Daily Cost Anomalies
- →Forecast Future Databricks Spend
Notes (5) ›
- Query Daily Spend by Product
This query joins `system.billing.usage` and `system.billing.list_prices` to provide a daily breakdown of spend by product. It helps users understand platform usage, product growth, and total expenditure.
- Analyze SQL Warehouse Cost Trends
This query shows daily usage trends for Databricks SQL Warehouses, which correlates to their consumption and cost. It helps identify warehouses for further optimization of scaling and autostop settings.
- Attribute Costs Using Custom Tags
This query leverages custom tags, enforced via compute and serverless usage policies, to attribute costs across users, projects, or teams. It helps in understanding spend allocation and identifying untagged usage.
- Detect Daily Cost Anomalies
This query identifies unexpected spikes in spending by comparing daily spend against a rolling 14-day average. It categorizes elevated spend (one standard deviation) or anomalies (two standard deviations) for quick detection and alerting.
- Forecast Future Databricks Spend
Utilizing the `AI_FORECAST` SQL AI Function, this query extrapolates recent usage data to project a 30-day spend forecast. This helps with future planning and budgeting without complex manual time-series modeling.
https://www.databricks.com/blog/top-5-system-table-queries-understanding-your-databricks-costs
Related releases
- Databricks SDK for Python v0.143.0 Adds ML Feature Fields, Fixes Logging Databricks Python SDK Releases ·
- Horizontal scaling for Databricks Apps is now generally available Databricks Release Notes ·
- Genie Code Builds Intelligent Document Processing Pipelines Databricks Release Notes ·
- DeepSeek V4 Pro (0813) Model Scheduled for Retirement Databricks Release Notes ·
- S&P Global Energy details conversational AI for structured data with Databricks Genie Agents Databricks Blog ·
- Databricks Python SDK v0.142.0 Adds AI Gateway Skill Management Methods Databricks Python SDK Releases ·