ReleaseBytes
gcp Google Cloud Blog ·

Checkout.com Migrates to Managed Airflow on Google Cloud

bloginfragcpgaengineergcp-cloud-storage
feature patch

Checkout.com successfully migrated its self-managed Apache Airflow to Google Cloud's Managed Service for Apache Airflow (Gen 3), significantly reducing operational overhead and improving scalability. This move addressed challenges like complex dependency management, slow DAG sync times, and manual scaling by leveraging the managed service's features. The migration resulted in enhanced reliability through DAG isolation, cost savings estimated at 30%, and faster developer workflows, including AI-powered troubleshooting with Gemini Cloud Assist.

  • →Migration to Managed Service for Apache Airflow (Gen 3) on Google Cloud
  • →AI-powered troubleshooting with Gemini Cloud Assist
  • →Checkout.com's operational challenges with self-managed Airflow
  • →Dynamic scaling and cost reduction with Managed Airflow
  • →Improved reliability through DAG isolation and managed operations
Features (2) ›
  • Migration to Managed Service for Apache Airflow (Gen 3) on Google Cloud

    Checkout.com migrated to Google Cloud's Managed Service for Apache Airflow (Gen 3) to offload infrastructure responsibilities and leverage managed scalability, aiming to improve reliability, cost efficiency, and developer velocity.

  • AI-powered troubleshooting with Gemini Cloud Assist

    Managed Airflow integrates with Gemini Cloud Assist, enabling users to initiate investigations directly from the Airflow DAG UI. Gemini generates a scorecard evaluating hypotheses with supporting and contradictory evidence, reducing mean time to recovery for failed tasks.

Enhancements (3) ›
  • Dynamic scaling and cost reduction with Managed Airflow

    Managed Airflow provides built-in dynamic scaling, automatically adjusting worker resources based on workload demands. This eliminated the need for manual resource management and led to an estimated 30% reduction in monthly costs compared to previous fixed provisioning.

  • Improved reliability through DAG isolation and managed operations

    The migration enhanced environment stability by introducing DAG isolation, preventing one DAG failure from impacting the entire system. Google Cloud now handles patching and upgrades, and integration with Cloud Monitoring and Logging improves visibility for debugging.

  • Faster developer workflows and modernized dbt execution

    Using Cloud Storage for DAGs enabled near-instant syncing, and containerized dbt runs with Managed Airflow eliminated dependency bottlenecks. This allows teams to run any dbt model with minimal setup and no manual infrastructure overhead.

Notes (1) ›
  • Checkout.com's operational challenges with self-managed Airflow

    Before migrating, Checkout.com faced significant operational friction managing a self-hosted Apache Airflow environment. This included challenges with server management, patching, upgrades, complex dependency management, slow DAG sync times, and manual scaling processes, all of which diverted focus from building data pipelines.

Read the original announcement →

https://cloud.google.com/blog/products/data-analytics/how-checkout-com-tallies-data-with-cloud-composer-3/

Related releases