AWS Lake Formation Enables Fine-Grained Access for S3 Tables with BigQuery
This article, part two of a series, details how to configure AWS Lake Formation for fine-grained access control when connecting Google BigQuery to Amazon S3 Tables for cross-cloud analytics. Lake Formation offers more granular, centralized permission management than IAM policies, issuing temporary, scoped credentials at query time. It is ideal for environments with multiple analytics engines or varied access requirements. The solution integrates S3 Tables, AWS Glue Data Catalog, Lake Formation, and Google Cross-Cloud Lakehouse.
- →Implement fine-grained access for S3 Tables with BigQuery using Lake Formation
- →Solution architecture for cross-cloud analytics
- →Prerequisites and Amazon S3 Tables setup
- →Configure Lake Formation integration for S3 Tables
- →Set up cross-cloud IAM access for BigQuery
Notes (6) ›
- Implement fine-grained access for S3 Tables with BigQuery using Lake Formation
This post describes using AWS Lake Formation to manage granular permissions for Amazon S3 Tables when connecting to Google BigQuery, offering an alternative to sole reliance on IAM policies for data access. Lake Formation evaluates permissions at query time and issues short-lived, scoped credentials to requesting engines, suitable for environments with diverse access needs.
- Solution architecture for cross-cloud analytics
The architecture uses the AWS Glue Iceberg REST Catalog as a bridge, allowing BigQuery's Cross-Cloud Lakehouse to sync metadata and read Iceberg data files directly from S3 Tables. Key components include S3 Tables for Apache Iceberg data, AWS Glue Data Catalog for metadata, AWS Lake Formation for permissions, and Google Cross-Cloud Lakehouse for BigQuery connectivity.
- Prerequisites and Amazon S3 Tables setup
Users require an AWS account with Amazon S3 Tables available, a Google Cloud project with BigLake API activated, and AWS CLI and gcloud CLI configured. Instructions are provided for creating an S3 table bucket, namespace, and populating a table if not already existing.
- Configure Lake Formation integration for S3 Tables
Setup involves creating a Lake Formation service IAM role, enabling S3 Table Integration in the Lake Formation console, and ensuring default permissions for IAMAllowedPrincipals are empty. This registers the S3 Tables data location with Lake Formation and creates the s3tablescatalog federated catalog in Glue.
- Set up cross-cloud IAM access for BigQuery
This step involves registering Google as an OIDC identity provider in the AWS account to validate tokens issued by Google's identity service. A cross-cloud IAM role is created with a placeholder trust policy, which is later updated with the BigLake service account ID, and attached with specific permissions for Lake Formation.
- Grant Lake Formation permissions and connect BigQuery
Lake Formation permissions must be granted at catalog, database, and table levels (DESCRIBE, SELECT) to the cross-cloud role. Before granting, default IAMAllowedPrincipals access should be revoked. Finally, Lake Formation settings for external engines are enabled, and a federated catalog is created in Google Cloud to connect BigQuery to the AWS Glue Iceberg REST Catalog.
https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-2-access-control-with-lake-formation/
Related releases
- Enable Cross-Cloud Analytics with S3 Tables and Google BigQuery via IAM AWS Big Data Blog ·
- Amazon Connect Customer adds information extraction for agent voice and chat AWS What's New ·
- Amazon Redshift now offers long-term system table retention via S3 Tables AWS What's New ·
- CloudFront OAC now natively supports S3 Multi-Region Access Points as origins AWS What's New ·
- Amazon Redshift adds long-term system table retention with S3 Tables AWS Big Data Blog ·
- SageMaker AI Studio Adds Generative AI Inference Recommendations for Optimized Deployments AWS What's New ·