ReleaseBytes
snowflake Snowflake Blog ·

Snowflake simplifies high-throughput data streaming to Apache Iceberg with Snowpipe Streaming

blogdatasnowflakeengineer
announcement

Snowflake showcases a streamlined method for high-throughput data streaming to Apache Iceberg tables, achieving over 1 million transactions per second (TPS) using Snowpipe Streaming and Snowpark Container Services. This approach simplifies large-scale streaming evaluations by eliminating the need for complex external infrastructure such as Kafka, EKS clusters, and IAM policy setups. Data engineers, SREs, and architects can now rapidly assess high-volume Iceberg streaming workloads within a fully Snowflake-managed environment. A companion GitHub repository provides a demo that allows setup and execution of the end-to-end streaming pipeline in half a day.

  • →Simplify 1M+ TPS Data Streaming to Apache Iceberg
  • →Leverage Snowflake's Integrated Architecture for Streaming
  • →Streamlined Setup with Snowflake-Managed Iceberg Tables
  • →Flexible Deployment Paths for Production Environments
  • →Best Practices for Optimizing Snowpipe Streaming
Notes (6) ›
  • Simplify 1M+ TPS Data Streaming to Apache Iceberg

    Snowflake demonstrates how to stream over one million transactions per second to Apache Iceberg format using Snowpipe Streaming and Snowpark Container Services. This setup allows for rapid evaluation and eliminates the need for complex external infrastructure like Kafka, EKS, or elaborate IAM configurations.

  • Leverage Snowflake's Integrated Architecture for Streaming

    The architecture uses Snowpark Container Services (SPCS) to generate synthetic load, streaming data via the Snowpipe Streaming SDK directly into Snowflake-managed Iceberg tables. This fully integrated approach facilitates quick testing and validation of high-volume streaming workloads.

  • Streamlined Setup with Snowflake-Managed Iceberg Tables

    Using Snowflake-managed Iceberg tables simplifies the initial setup by removing the need for external volume configurations, storage integrations, and cloud storage permissions. Data is stored as Parquet with Snowflake-managed Iceberg metadata, supporting immediate querying and governance.

  • Flexible Deployment Paths for Production Environments

    For production, users have options including Snowflake-managed Iceberg with external S3 volumes, Snowflake-managed Iceberg on Snowflake's internal storage, or native Snowflake tables. The streaming code remains consistent across these choices.

  • Best Practices for Optimizing Snowpipe Streaming

    Snowflake's server-side buffering and automatic compaction batch small files into larger ones for efficiency. Users can further optimize performance by batching rows client-side before calling appendRows and maintaining long-lived channels.

  • Future Outlook: Iceberg v4 and Adaptive Metadata Trees

    The article notes the potential for Iceberg v4's Adaptive Metadata Tree to further reduce metadata overhead for frequent commits in high-frequency streaming workloads. Snowflake is actively involved in shaping the future of this open format.

Read the original announcement →

https://www.snowflake.com/content/snowflake-site/global/ja/blog/snowpipe-streaming-iceberg-tables

Related releases