Snowflake Enhances AI Data Engineering with New Pipeline Capabilities
Snowflake has introduced new AI-powered features for data engineering, aiming to make pipeline creation more robust and efficient. These capabilities integrate AI directly into workflows, offering agentic assistance, autonomous pipelines, and improved declarative options for SQL and Python developers. The updates focus on reducing manual effort, accelerating time-to-production, and enhancing governance for all data engineers working within or across lakehouses and Snowflake.
- →Agentic Workflows with Snowflake CoCo for Data Engineering
- →Autonomous Pipelines with Dynamic Iceberg Tables
- →dbt Projects Integration and Enhanced dbt DAG
- →Pipeline Builder for Programmatic Workflows
- →Performance and Interoperability Updates for Dynamic Tables
Features (4) ›
- Agentic Workflows with Snowflake CoCo for Data Engineering
Snowflake CoCo, an agentic workflow tool, operates within the user's local environment to build end-to-end data engineering solutions. Benchmarks show it outperforms generic coding agents, using fewer tokens and steps, and understands enterprise data context. CoCo is available via Snowsight, CLI, and a new desktop app in public preview, supporting models like Claude and GPT.
- Autonomous Pipelines with Dynamic Iceberg Tables
Declarative workflows allow users to define desired outcomes, with Snowflake managing the execution, making pipelines more scalable than traditional manual orchestration. Wolt uses Dynamic Iceberg Tables to enrich and refresh data in their data lake, reducing maintenance time and overhead for incremental pipelines.
- dbt Projects Integration and Enhanced dbt DAG
dbt Projects can now be managed natively within Snowflake, providing built-in observability and CI/CD integration without separate infrastructure management. dbt Fusion (GA) is included with dbt Projects, improving compilation times, and column-level lineage is now part of the enhanced dbt DAG (GA).
- Pipeline Builder for Programmatic Workflows
The new Pipeline Builder (private preview) allows data scientists and engineers to visually connect Notebooks and ML Jobs into end-to-end pipelines without custom orchestration code. This feature automates scheduling and infrastructure setup, facilitating faster iteration and easier monitoring of ML pipelines.
Enhancements (2) ›
- Performance and Interoperability Updates for Dynamic Tables
Snowflake has enhanced its native declarative workflows, including faster Dynamic Tables refresh performance (GA) which accelerates workloads by up to 2.8x on Gen2 warehouses. Public previews offer custom incrementalization using MERGE/INSERT statements and adaptive refresh that automatically selects the most efficient method.
- Snowpark Enhancements for End-to-End Development
Snowpark provides a native development experience for Python, Java, and Scala within Snowflake. It supports iterating in Notebooks, building transformations with DataFrame API, packaging logic as stored procedures/UDFs, and scheduling with Tasks, offering a complete workflow from coding to production.
https://www.snowflake.com/content/snowflake-site/global/en/blog/ai-smart-pipelines-whats-new