ReleaseBytes
snowflake Snowflake Blog ·

Snowflake Custom Incrementalization for Dynamic Tables Improves Performance

blogdatasnowflakeengineer
feature patch

Thrive Learning, a UK-based enterprise learning platform, improved the cost-performance of its Snowflake Dynamic Tables by implementing custom incrementalization. This new capability allows transformations to process only changed rows rather than full refreshes, significantly reducing compute and auto-clustering costs. The solution, developed by Thrive's data engineering team, also maintains data lineage within their dbt workflows, enabling fresher data for AI use cases.

  • →Enable custom incrementalization for Snowflake Dynamic Tables
  • →Reduce transformation and auto-clustering costs with custom incrementalization
  • →Maintain data lineage and governance with custom incrementalization in dbt
  • →Thrive Learning case study on Dynamic Tables and custom incrementalization
  • →Improve data freshness for AI use cases
Features (1) ›
  • Enable custom incrementalization for Snowflake Dynamic Tables

    Snowflake Dynamic Tables now support a CUSTOM_INCREMENTAL refresh mode, allowing users to define explicit MERGE logic for processing only changed rows. This feature was utilized by Thrive Learning to resolve issues with full refreshes caused by complex joins and deduplication in their data pipelines.

Enhancements (3) ›
  • Reduce transformation and auto-clustering costs with custom incrementalization

    By implementing custom incrementalization, Thrive Learning reduced transformation compute costs by approximately 97% and auto-clustering costs by about 99%. This was achieved by processing only delta changes and eliminating the need for full table rebuilds or complex UNION operations between fast and slow data lanes.

  • Maintain data lineage and governance with custom incrementalization in dbt

    Custom incrementalization for Dynamic Tables integrates seamlessly with dbt projects, preserving data lineage and simplifying workflow management. Thrive Learning wrapped these tables in a custom dbt materialization, ensuring they remain visible and manageable within their existing dbt DAG without requiring a separate orchestration layer.

  • Improve data freshness for AI use cases

    The cost reductions and near real-time data achieved through custom incrementalization enable Thrive Learning to support more agentic AI use cases. Fresher, more accessible data positions the platform to expand its AI capabilities effectively.

Notes (1) ›
  • Thrive Learning case study on Dynamic Tables and custom incrementalization

    Thrive Learning, a UK-based learning platform, shares how they leveraged Snowflake Dynamic Tables to achieve continuous data transformation and freshness. Their journey involved optimizing a complex pipeline that initially fell back to full refreshes, leading to high costs, before adopting custom incrementalization.

Read the original announcement →

https://www.snowflake.com/content/snowflake-site/global/en/blog/thrive-dynamic-table-costs-custom-incremental

Related releases