ReleaseBytes
aws AWS Big Data Blog ·

Amazon MSK Express adds Apache Iceberg table delivery for Kafka data

blogdataawsengineermediaaws-s3
feature

Amazon Managed Streaming for Apache Kafka (MSK) Express brokers now support direct delivery of streaming data to Apache Iceberg tables on Amazon S3 Tables. This feature eliminates the need for managing Kafka connectors or custom consumers, allowing data to be materialized as queryable Iceberg tables within minutes. It offers cost savings and improved query performance by optimizing file sizes, and integrates with AWS Glue Data Catalog for enhanced data discoverability and AI-assisted data analysis. The capability is available in all regions where MSK Express brokers are offered.

  • →Direct delivery of Kafka data to Apache Iceberg tables
  • →Delivery to general purpose S3 buckets for archival and training
  • →Improved cost and query performance
  • →Enhanced data discoverability and AI integration
  • →Managed end-to-end streaming analytics architecture
Features (2) ›
  • Direct delivery of Kafka data to Apache Iceberg tables

    Amazon MSK Express brokers can now directly deliver streaming data to Apache Iceberg tables on Amazon S3 Tables, simplifying the process of making streaming data available for analytics. This eliminates the need for manual deployment and management of Kafka connectors, Flink jobs, or custom consumers.

  • Delivery to general purpose S3 buckets for archival and training

    In addition to S3 Tables, Amazon MSK can deliver streaming data to general purpose Amazon S3 buckets in its source format. This is suitable for workloads such as data archival, backup, or preparing data for ML training.

Enhancements (2) ›
  • Improved cost and query performance

    Delivery to streaming tables offers up to 60% cost savings compared to self-managed solutions and reduces downstream query costs by up to 30% through optimized file sizing, without requiring custom code or infrastructure management.

  • Enhanced data discoverability and AI integration

    Data delivered to S3 Tables is registered in AWS Glue Data Catalog, enabling automatic discoverability through features like Business Context and Semantic Search (preview). This allows data stewards to enrich tables with business descriptions and glossary terms, which AI agents can use for real-time reasoning.

Maintenance (1) ›
  • Agent Skill for simplified setup

    An Agent Skill is available to provide AI-assisted guidance for setting up streaming table integrations, including schema registry and IAM role configuration, simplifying the setup process for users.

Notes (1) ›
  • Managed end-to-end streaming analytics architecture

    This capability provides a fully managed architecture from data ingestion via MSK Express brokers to storage in S3 Tables and querying with engines like Amazon Athena, Redshift, or Apache Spark, eliminating intermediate infrastructure components.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/deliver-apache-kafka-data-to-streaming-tables-for-apache-iceberg-with-amazon-msk-express-brokers/

Related releases