ReleaseBytes
gcp Google Cloud Blog ·

Pine59's Migration to Airflow 3 on Google Cloud Improves Data and MLOps Pipelines

blogmlgcpengineergcp-bigquerygcp-gkegcp-composer
announcement

Pine59, a location intelligence provider, successfully migrated its extensive data and machine learning pipelines to Managed Service for Apache Airflow (Gen 3) running Airflow 3 on Google Cloud. This modernization significantly enhanced the company's MLOps capabilities, streamlined developer workflows, and improved pipeline speed and reliability. The transition included optimizing ML inference workloads with dedicated GKE clusters and leveraging Airflow 3's developer features for custom debugging tools. This move resulted in quantifiable performance gains, including a 32% reduction in processing time for a key data-intensive pipeline.

  • →Significant Performance Improvements with Managed Airflow Gen 3
  • →Enhanced MLOps Capabilities and Integration
  • →Improved Developer Workflow and Custom Extensibility
  • →Quantifiable Speed and Reliability Gains
Notes (4) ›
  • Significant Performance Improvements with Managed Airflow Gen 3

    Pine59 observed immediate and substantial improvements in processing speed, task scheduling, and overall stability by migrating its production workloads to Managed Airflow (Gen 3) running Airflow 3. This upgrade dramatically reduced queue latency, allowing tasks to start almost instantly, preventing bottlenecks during peak processing surges.

  • Enhanced MLOps Capabilities and Integration

    The migration allowed Pine59 to refine its MLOps architecture by optimizing the orchestration of ML inference workloads. They integrated a dedicated Google Kubernetes Engine (GKE) cluster, specifically optimized for model inference, into their pipelines, creating a clear separation of orchestration and heavy ML execution compute.

  • Improved Developer Workflow and Custom Extensibility

    Airflow 3 delivered a vastly improved developer workflow and user interface, which Pine59 capitalized on by building custom plugins. These include a BigQuery Auto-linkify tool for faster debugging and a DAG Run Configuration Search form, directly integrated into the Airflow UI to enhance internal developer velocity and troubleshooting.

  • Quantifiable Speed and Reliability Gains

    The transition yielded concrete results, with a significant reduction in DAG run queue times and overall processing duration. For example, Pine59's Daily Foot Traffic pipeline, which processes data for up to 14 million locations, saw its completion time drop by nearly 32%, from 38 minutes to less than 26 minutes.

Read the original announcement →

https://cloud.google.com/blog/topics/supply-chain-logistics/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud/

Related releases