How United Airlines Queries Databricks Data from Amazon Redshift via AWS Glue Federation
United Airlines has implemented AWS Glue Data Catalog federation to enable Amazon Redshift Serverless to query data managed by Databricks Unity Catalog directly. This approach eliminates the need to duplicate petabytes of data from Databricks-managed S3 into Redshift, streamlining cross-platform analytics for customer engagement data. The solution leverages Amazon Redshift Serverless's Graviton-based engine, optimized for reading Apache Iceberg tables from Amazon S3, with permissions governed by AWS Lake Formation. United Airlines currently federates 30 tables in production, with plans to expand this to hundreds more across various business domains.
Notes (1) ›
- Connecting Amazon Redshift to Databricks Unity Catalog via AWS Glue Federation
United Airlines demonstrates how AWS Glue Data Catalog federation enables Amazon Redshift Serverless to query Databricks-managed data directly from S3, without requiring data duplication. The post details the architecture, which includes AWS Lake Formation for permissions and relies on Redshift's ability to read Apache Iceberg tables for cross-platform analytics.
https://aws.amazon.com/blogs/big-data/how-united-airlines-uses-amazon-redshift-and-aws-glue-data-catalog-federation-to-query-databricks-managed-data/
Related releases
- Amazon Redshift RG Instances Now Available in Europe (Zurich) Region AWS What's New ·
- AWS Launches Data Analytics Plugin for ChatGPT Work with Redshift Integration AWS Big Data Blog ·
- Amazon Redshift rg.large Instances Now Support Single-Node Clusters AWS What's New ·
- How Moovit Achieved 33% Cost Optimization Through AWS Architectural Modernization AWS Big Data Blog ·
- Amazon Redshift adds IAM Identity Center authentication with enhanced VPC routing AWS What's New ·
- Amazon Redshift adds support for Apache Iceberg v3 tables AWS What's New ·