ReleaseBytes
databricks Databricks Blog ·

Databricks Offers Framework for Migrating from BigQuery

blogdatadatabricksengineer
announcement

Databricks has released a strategic framework to guide enterprises in migrating from Google BigQuery to the Databricks Lakehouse architecture. This migration aims to break down data silos, unify governance, and establish a foundation for AI innovation while reducing total cost of ownership. It is primarily aimed at organizations experiencing scaling challenges, rising costs, and fragmented governance with BigQuery, consolidating BI, ETL, and AI workloads into a single environment for predictable performance and reduced operational overhead.

  • →Phased migration approach for BigQuery to Databricks
  • →Open-source Databricks Labs toolkit for BigQuery migration
  • →Dual operation with Lakehouse Federation for validation
  • →Databricks provides a migration framework from BigQuery
  • →Unity Catalog for unified governance and security
Features (3) ›
  • Phased migration approach for BigQuery to Databricks

    The migration emphasizes a phased approach, recommending an initial assessment of the BigQuery estate to identify key workloads and usage patterns. This assessment informs the choice between a 'BI-first' or 'ETL-first' strategy for migration waves.

  • Open-source Databricks Labs toolkit for BigQuery migration

    The Lakebridge, an open-source toolkit from Databricks Labs, includes a BigQuery profiler to automate the discovery of datasets, query history, and slot consumption. This helps in planning migration waves based on actual usage.

  • Dual operation with Lakehouse Federation for validation

    Organizations can utilize Lakehouse Federation during a dual operation phase to shadow BigQuery workloads for validation. Success criteria are set to trigger the decommissioning of legacy pipelines, ensuring minimal disruption.

Enhancements (2) ›
  • Unity Catalog for unified governance and security

    Unity Catalog is presented as a key component for governance, offering a 3-tier mapping that replicates BigQuery permissions and adds automatic lineage tracking. It also supports fine-grained security features like row filters and column masks.

  • Open table formats and data migration strategies

    The framework promotes moving to open table formats and outlines workstreams for data and logic migration. Bulk history migration is recommended via BigQuery export to Google Cloud Storage, while continuously updated tables can be handled via the Storage API connector.

Notes (1) ›
  • Databricks provides a migration framework from BigQuery

    Databricks has introduced a strategic framework to assist organizations in migrating from BigQuery to their Lakehouse architecture. The framework covers process, technology, and people aspects of the migration, aiming to unify governance and enable AI innovation.

Read the original announcement →

https://www.databricks.com/blog/bigquery-databricks-strategic-framework-modern-migration

Related releases