ReleaseBytes
gcp Google Cloud Blog ·

Checkout.com Migrates to Google Cloud Composer 3 for Improved Data Orchestration

bloginfragcpgaengineergcp-cloud-storagegcp-composer
feature patch announcement

Checkout.com migrated from a self-managed Apache Airflow environment to Google Cloud's Managed Service for Apache Airflow (Gen 3) to reduce operational overhead and enhance data pipeline reliability. This move offloads infrastructure maintenance, enabling dynamic scaling, cost reductions, and faster developer workflows, ultimately freeing up engineering time for innovation. The migration affects data engineering teams managing orchestration and pipeline development.

  • →Managed Airflow introduces dynamic scaling and cost efficiencies
  • →AI-powered troubleshooting with Gemini Cloud Assist
  • →Checkout.com details migration from self-managed Airflow to Google Cloud Composer 3
  • →Managed Airflow simplifies dependency management and DAG syncing
  • →Improved reliability through DAG isolation and managed operations
Features (2) ›
  • Managed Airflow introduces dynamic scaling and cost efficiencies

    Checkout.com's data platform now leverages Managed Airflow's dynamic scaling capabilities, eliminating the need for manual resource allocation for peak loads. This shift from fixed provisioning to dynamic scaling resulted in an estimated 30% reduction in monthly costs.

  • AI-powered troubleshooting with Gemini Cloud Assist

    Checkout.com can now utilize Gemini Cloud Assist directly from the Airflow DAG UI for troubleshooting failed tasks. Gemini generates scorecards evaluating hypotheses with supporting and contradictory evidence, aiming to reduce mean time to recovery.

Enhancements (3) ›
  • Managed Airflow simplifies dependency management and DAG syncing

    The migration to Managed Airflow (Gen 3) addressed challenges with complex dependency management and slow DAG sync times experienced in the self-managed environment. Dependencies are now handled at the image level, and DAG syncing to the scheduler is near-instantaneous using Cloud Storage.

  • Improved reliability through DAG isolation and managed operations

    Managed Airflow (Gen 3) provides DAG isolation, ensuring that a single problematic DAG does not impact the entire environment. Google Cloud manages patching and upgrades, while integration with Cloud Monitoring and Logging enhances visibility for engineers.

  • Faster developer workflows and modernized dbt execution

    The migration accelerates developer workflows through near-instant DAG syncing and simplified team onboarding. Containerized dbt runs in Managed Airflow (Gen 3) eliminate dependency bottlenecks and manual infrastructure overhead.

Notes (1) ›
  • Checkout.com details migration from self-managed Airflow to Google Cloud Composer 3

    Checkout.com shares their experience moving from a self-hosted Apache Airflow to Google Cloud's Managed Service for Apache Airflow (Gen 3). The company sought to reduce the burden of server management, patching, and incident response, which detracted from pipeline development.

Read the original announcement →

https://cloud.google.com/blog/products/data-analytics/how-checkout-com-tallies-data-with-cloud-composer-3/

Related releases