ReleaseBytes
databricks Databricks Blog ·

Branching Lakebase databases like code for CI/CD

blogdatadatabricksengineerarchitect
feature announcement

Glaspoort implemented a CI/CD pattern for their Databricks Lakebase, treating database changes with the same rigor as application code. This involves branching every environment directly from production and using ephemeral per-PR databases, with migrations as the single source of truth. This approach aims to avoid the common "reset-from-parent trap" that causes environments to drift from production and necessitates costly rebuilds. The pattern allows for faster, more reliable database updates in production environments.

  • →Branching strategy to avoid environment drift
  • →Ephemeral per-PR database branches
  • →Migrations as the source of truth
  • →Glaspoort's CI/CD pattern for Lakebase databases
  • →Two promotion models for database changes
Features (3) ›
  • Branching strategy to avoid environment drift

    The core of the design involves branching development and acceptance environments directly from production, rather than stacking them sequentially. This strategy, termed the "reset-from-parent trap," aims to prevent development and acceptance databases from drifting away from the production state.

  • Ephemeral per-PR database branches

    Each pull request gets its own disposable Lakebase branch, copied from production. Migrations are then replayed and tested against a live application image before any changes are applied to a production environment.

  • Migrations as the source of truth

    Database migrations are treated as the definitive source of truth within this CI/CD workflow. This ensures consistency and reliability in how database changes are applied and managed across different environments.

Enhancements (1) ›
  • Two promotion models for database changes

    The article discusses a tradeoff between two promotion models: merging as soon as CI passes versus merging only after a PR has been fully promoted through acceptance. Glaspoort opted for a "velocity first" approach, implementing safeguards like stack revalidation and a crisis pipeline.

Notes (2) ›
  • Glaspoort's CI/CD pattern for Lakebase databases

    Glaspoort developed a system to manage Databricks Lakebase database changes using a CI/CD pattern similar to application code. This involves branching each environment directly from production and creating ephemeral per-pull request (PR) databases.

  • Lakebase as an OLTP database for applications

    Glaspoort utilizes Databricks Lakebase, described as serverless Postgres OLTP, running alongside their lakehouse. This setup allows curated data from the lakehouse to be synced into a production Lakebase branch for application use.

Read the original announcement →

https://www.databricks.com/blog/branching-databases-code-cicd-pattern-lakebase-production-glaspoort

Related releases