Checkout.com Migrates to Managed Airflow on GCP for Reliability and Cost Savings

Checkout.com migrated to Google Cloud's Managed Service for Apache Airflow (Gen 3) to offload infrastructure responsibilities and leverage managed scalability. This resulted in improved reliability, cost efficiency, and developer velocity.

Checkout.com integrated Gemini Cloud Assist within the Airflow DAG UI for AI-powered troubleshooting. Gemini generates scorecards with evidence to hypotheses, drastically reducing mean time to recovery for failed tasks.

Managed Airflow's dynamic scaling automatically adjusts workers based on workload demands, eliminating the need for peak capacity provisioning. This shift from fixed to dynamic scaling reduced monthly costs by an estimated 30%.

The managed service provides DAG isolation, ensuring one DAG's failure doesn't impact the entire environment. Google Cloud handles patching and upgrades, and integration with Cloud Monitoring and Logging improves visibility for debugging.

Near-instant DAG syncing using Cloud Storage, simplified onboarding, and containerized dbt runs eliminate dependency bottlenecks. Environment updates are also streamlined, no longer requiring full redeployments for new roles or package updates.

Checkout.com's data engineering team faced significant operational burdens maintaining a self-managed Apache Airflow environment, including server management, patching, and incident response. This consumed valuable time that could have been spent on pipeline development.

The self-managed Airflow environment presented stability challenges during high loads, complex dependency management requiring manual upgrades, slow DAG sync times (approx. 6 minutes), and manual scaling processes. Onboarding new teams also involved manual secret and variable creation.