Build Medallion Architectures with Iceberg MVs in Amazon SageMaker
AWS details a declarative approach for building medallion architectures using Apache Iceberg materialized views in Amazon SageMaker Unified Studio. This method significantly simplifies data pipeline construction by consolidating ETL, orchestration, and incremental processing into single SQL definitions per layer. It reduces the complexity of maintaining separate code for transformations, orchestration, and change data capture. The solution utilizes AWS Glue's managed Spark compute for atomic refreshes and is supported by services like Amazon Athena Spark, AWS Glue 5.1+, and Amazon EMR 7.12+.
- →Declarative Medallion Architecture with Iceberg MVs
- →Apache Iceberg Materialized View Functionality
- →Service Support and Technical Architecture
- →Build a Medallion Pipeline in SageMaker Unified Studio
Features (1) ›
- Declarative Medallion Architecture with Iceberg MVs
AWS introduces a declarative method for building medallion architectures using Apache Iceberg materialized views within Amazon SageMaker. This approach collapses ETL, orchestration, and incremental processing into single SQL definitions, reducing pipeline complexity and maintenance overhead.
Enhancements (1) ›
- Apache Iceberg Materialized View Functionality
Apache Iceberg materialized views enable defining each layer of a medallion architecture with SQL statements, which AWS Glue's managed Spark compute then refreshes incrementally. Iceberg's row-level change tracking ensures only new or modified records are processed, building implicit dependency chains between layers.
Notes (2) ›
- Service Support and Technical Architecture
This approach is supported by Amazon Athena Spark, AWS Glue 5.1+, and Amazon EMR 7.12+. The architecture leverages Amazon S3 Tables for storage, AWS Glue Data Catalog for metadata, and Amazon SageMaker Unified Studio for authoring SQL definitions.
- Build a Medallion Pipeline in SageMaker Unified Studio
The post provides a step-by-step guide to implement a Bronze → Silver → Gold pipeline using SQL in Amazon SageMaker Unified Studio. This includes environment initialization, data ingestion, and creating Silver and Gold materialized views.
https://aws.amazon.com/blogs/big-data/building-medallion-architecture-with-iceberg-materialized-views-in-amazon-sagemaker/
Related releases
- AWS Config expands support for 60 new resource types AWS What's New ·
- Meta's Muse-Glimmer-30B and Alibaba's Qwen 3.8-27B Models Now on SageMaker JumpStart AWS What's New ·
- NVIDIA Cosmos3-Edge, Nano, and Super Physical AI Models Now on Amazon SageMaker JumpStart AWS What's New ·
- SageMaker MLflow now supports customer managed keys AWS What's New ·
- SageMaker HyperPod Enhances Ray Support with Observability, Resilient Training, and Accelerated Inference AWS What's New ·
- AWS Glue 6.0 Reduces Pricing by 30% and Adds Iceberg v3 Support AWS What's New ·