Google Cloud details agentic data pipelines for MLOps with the Data Agent Kit
This Google Cloud post details the Orchestration Pipelines framework and Data Agent Kit, first announced at Cloud NEXT '26. The open-source kit integrates into IDEs and CLIs, providing a Data Engineering tab and specialized agentic skills to simplify Apache Airflow DAG creation via natural language and a declarative YAML DSL. This approach aims to democratize MLOps capabilities and lower the barrier for data professionals by abstracting complex Python boilerplate. The article includes a step-by-step walkthrough demonstrating an MLOps use case for proactive supply chain management.
- →Overview of Google Cloud's Orchestration Pipelines and Data Agent Kit
- →Setting up the Data Agent Kit development environment
- →Authoring Apache Airflow DAGs with natural language
- →Example MLOps use case for supply chain management
- →Demonstrating an agent-generated training pipeline
Notes (5) ›
- Overview of Google Cloud's Orchestration Pipelines and Data Agent Kit
This section introduces the Orchestration Pipelines framework and the Data Agent Kit, a unified, open-source collection of data engineering and data science tools. These were first announced at Google Cloud NEXT ’26 to democratize access to MLOps capabilities.
- Setting up the Data Agent Kit development environment
The article outlines the quick setup process for the Data Agent Kit, involving installing and configuring the IDE/CLI extension and verifying that the `gcp-pipelines-orchestration` agent skill is enabled for deep contextual knowledge.
- Authoring Apache Airflow DAGs with natural language
The Data Agent Kit enables users to author, deploy, and troubleshoot production-grade Apache Airflow DAGs using natural language prompts and a declarative YAML DSL. This capability allows data professionals to bypass complex Python boilerplate.
- Example MLOps use case for supply chain management
A practical MLOps use case is demonstrated, focusing on proactive supply chain management to predict transit times and prevent customer churn. It leverages BigQuery, Managed Service for Apache Spark, Gemini Enterprise Agent Platform, and dbt.
- Demonstrating an agent-generated training pipeline
The post showcases a YAML definition generated by the Data Agent Kit that outlines a training engine pipeline. This pipeline queries BigQuery, provisions a Managed Spark serverless cluster to train a model, and pushes it to Gemini Enterprise Agent Platform Model Registry.
https://cloud.google.com/blog/products/data-analytics/build-data-pipelines-in-less-time-with-data-agent-kit/
Related releases
- Google Cloud Boosts AI Infrastructure with Filestore, GKE, and New Security Tools Google Cloud Blog ·
- Google details BigQuery Graph's GA and cross-cloud analytics for AI agents Google Cloud Blog ·
- Looker Update Brings AI Data Agents, In-Database Analytic Models, and CI Enhancements Google Cloud release notes ·
- BigQuery Enhances Compliance, Graph Processing, and XGBoost ML Capabilities Google Cloud release notes ·
- BigQuery Addresses Security Vulnerability and Adds Real-time Python UDF Logging Google Cloud release notes ·
- Terraform Google Provider v8.0.0 introduces significant breaking changes Terraform Google Provider Releases ·