Guide: Migrating Azure Synapse workloads to Databricks Lakehouse
This guide provides a practical playbook for migrating workloads from Azure Synapse Analytics (Dedicated SQL, Serverless SQL, and Spark pools) to a unified Databricks Lakehouse. It details how to consolidate multiple services, enable AI and ML capabilities, and improve operational efficiency, leading to simpler architecture, better performance, and lower costs. The document outlines a phased migration strategy, emphasizing discovery, assessment, and design with field-tested engineering tips for successful execution.
- →Databricks Lakehouse supports future-ready data teams with AI and ML convergence.
- →Tools like Lakebridge Profiler and Analyzer aid Synapse migration assessment.
- →Migrating Synapse to Databricks unifies data, analytics, and AI on one platform.
- →Synapse migration to Databricks can yield significant operational efficiency gains.
- →Understand the scope of Synapse migration, which involves distinct services.
Features (2) ›
- Databricks Lakehouse supports future-ready data teams with AI and ML convergence.
Modern data teams increasingly focus on ML models, real-time pipelines, and AI applications, which require a converged data architecture. Databricks is built for this convergence, unifying data, analytics, and AI, and leveraging Unity Catalog and Unity AI Gateway for consistent governance across all assets, including models and AI applications.
- Tools like Lakebridge Profiler and Analyzer aid Synapse migration assessment.
Lakebridge Profiler scans the Synapse estate for metadata on configuration, resource utilization, and query patterns to build a Total Cost of Ownership (TCO) case. Lakebridge Analyzer then evaluates the T-SQL codebase, classifying objects by complexity, flagging unsupported constructs, and mapping dependencies to assess migration timelines and prioritize assets.
Enhancements (1) ›
- Synapse migration to Databricks can yield significant operational efficiency gains.
Migrating from Azure Synapse to Databricks Lakehouse can reduce the number of systems to operate and support, leading to fewer integrations and handoffs. Companies like Casey's reduced operational data delivery times, and Italgas reported a 73% reduction in workload costs while serving both Power BI and AI analytics directly from Databricks.
Notes (3) ›
- Migrating Synapse to Databricks unifies data, analytics, and AI on one platform.
Organizations often use multiple Synapse services like Dedicated SQL, Serverless SQL, and Spark Pools, alongside ADF, leading to duplicated governance, tooling, and operational overhead. Databricks offers a unified platform for data engineering, analytics, ML, and governance, reducing complexity and integration points for easier operation at scale.
- Understand the scope of Synapse migration, which involves distinct services.
Azure Synapse comprises distinct services like Dedicated SQL Pools, Serverless SQL Pools, and Spark Pools, each with unique migration strategies. Dedicated SQL Pools, often the most complex, require significant redesign for business logic, stored procedures, and performance optimizations, while Serverless SQL and Spark Pools are generally simpler to migrate.
- Structure Synapse migrations as a phased program, not a single project.
Successfully migrating from Synapse to Databricks involves moving multiple compute models, consolidating governance, modernizing orchestration, and reworking T-SQL logic. A phased program approach, including discovery, assessment, and design, is recommended over a single workstream to manage complexity and effort effectively.
https://www.databricks.com/blog/navigating-synapse-migration-databricks