Snowflake CoCo adds Spark migration skill with AI
Snowflake's CoCo AI agent now includes a spark-migration skill to help users move PySpark code to Snowpark Connect with minimal changes. This addresses the complexity of migrating large codebases, offering automated assessment, targeted fixes, and functional validation. The skill is available now within CoCo, requiring no additional setup for existing CoCo users.
- →AI-powered Spark to Snowpark migration skill
- →Streamlined migration workflow with single prompt
- →Automated assessment and reporting for Spark code
- →End-to-end functional validation of migrated code
- →Broad applicability for various migration scenarios
Features (1) ›
- AI-powered Spark to Snowpark migration skill
Snowflake CoCo now features a spark-migration skill that assists in migrating Apache Spark code, including PySpark, Scala, and Java, to Snowpark Connect. This AI agent automates code assessment, applies targeted fixes, and validates functionality, aiming to reduce manual effort and accelerate migration.
Enhancements (3) ›
- Streamlined migration workflow with single prompt
Users can now migrate entire directories of Spark code using a single prompt to CoCo's spark-migration skill. The skill orchestrates an agentic workflow that includes assessing compatibility, fixing unsupported patterns, and generating reports, simplifying the migration process compared to manual methods.
- Automated assessment and reporting for Spark code
The spark-migration skill performs an in-depth assessment of Spark source files, identifying compatibility issues related to RDD operations, UDF serialization, and file formats. It generates detailed issue logs and inventory reports, providing visibility into changes and areas requiring manual attention.
- End-to-end functional validation of migrated code
An optional validation phase allows for end-to-end testing of the migrated Snowpark Connect code against the original PySpark code using synthetic data. This process compares table outputs to catch behavioral differences and can automatically commit fixes found during validation.
Notes (1) ›
- Broad applicability for various migration scenarios
The spark-migration skill supports migrating individual files, entire directories, and notebook files. It can also be prompted to assess compatibility, validate existing migrations from other tools, or complete them.
https://www.snowflake.com/content/snowflake-site/global/en/blog/migrate-spark-to-snowflake
Related releases
- Snowflake Observe on Iceberg enters private preview Snowflake Blog ·
- Migrate Apache Spark to Snowflake using CoCo skill Snowflake Blog ·
- Snowflake and Google Cloud collaborate on Apache Iceberg lakehouse Snowflake Blog ·
- Snowflake Adaptive Compute GA in Azure, GCP regions Snowflake Blog ·
- Snowflake Custom Incrementalization for Dynamic Tables Improves Performance Snowflake Blog ·
- Snowflake Enhances AI Security with Agentic Controls and Governance Snowflake Blog ·