Checkout.com Migrates to Managed Airflow on Google Cloud

Checkout.com migrated to Google Cloud's Managed Service for Apache Airflow (Gen 3) to offload infrastructure responsibilities and leverage managed scalability, aiming to improve reliability, cost efficiency, and developer velocity.

Managed Airflow integrates with Gemini Cloud Assist, enabling users to initiate investigations directly from the Airflow DAG UI. Gemini generates a scorecard evaluating hypotheses with supporting and contradictory evidence, reducing mean time to recovery for failed tasks.

Managed Airflow provides built-in dynamic scaling, automatically adjusting worker resources based on workload demands. This eliminated the need for manual resource management and led to an estimated 30% reduction in monthly costs compared to previous fixed provisioning.

The migration enhanced environment stability by introducing DAG isolation, preventing one DAG failure from impacting the entire system. Google Cloud now handles patching and upgrades, and integration with Cloud Monitoring and Logging improves visibility for debugging.

Using Cloud Storage for DAGs enabled near-instant syncing, and containerized dbt runs with Managed Airflow eliminated dependency bottlenecks. This allows teams to run any dbt model with minimal setup and no manual infrastructure overhead.

Before migrating, Checkout.com faced significant operational friction managing a self-hosted Apache Airflow environment. This included challenges with server management, patching, upgrades, complex dependency management, slow DAG sync times, and manual scaling processes, all of which diverted focus from building data pipelines.