Databricks Benchmarks Coding Agents on its Codebase

The benchmark results indicate that the Pareto frontier for coding tasks includes models from OpenAI, Anthropic, and open-source options, suggesting a mixed-tool approach is optimal. Open models, like GLM 5.2, are now capable of handling high-difficulty tasks and can be cost-effective alternatives to more expensive models.

The benchmark identified three distinct capability tiers for coding agents. While high-end models excel at all problem types, they are expensive. Medium and lower-intelligence models are effective and cheaper for common tasks, suggesting a shift towards these for daily development.

Analysis shows that token price is a poor predictor of actual end-to-end task costs. Larger, more token-efficient models can incur lower overall costs despite higher per-token rates. Model efficiency varies significantly, impacting task completion time and token consumption.

The harness used to call a model dramatically influences both cost and quality. Simple harnesses, like Pi, often performed best on Databricks' workloads, demonstrating the importance of context management in task execution. Model choice is only one part of the efficiency puzzle.

Databricks has created an internal benchmark to evaluate the performance and cost-effectiveness of coding agents on real-world tasks, using its own multi-million line codebase. This benchmark aims to help engineers make informed decisions about tool selection.

The benchmark was constructed using a rigorous process, filtering recent, human-written pull requests with high-quality tests and self-contained changes. Tasks were carefully crafted to represent typical full-stack development needs across various languages and services used at Databricks.