Automate AWS Glue Data Catalog view creation with SDK
This post details how to automate the creation and updating of AWS Glue Data Catalog views, specifically the ATHENA dialect, using the AWS SDK. This enhancement supports data mesh architectures by enabling programmatic creation of views with both ATHENA and SPARK dialects across multiple accounts via cross-account IAM definer roles. This allows for more flexible data sharing and centralized governance without exposing underlying base tables.
- →Programmatic creation of multi-dialect Glue views
- →Cross-account definer roles for data mesh enablement
- →Key considerations for multi-dialect view creation
- →Adding ATHENA dialect to existing SPARK-only views
Notes (4) ›
- Programmatic creation of multi-dialect Glue views
AWS Glue now supports programmatic creation and updating of ATHENA-dialect Data Catalog views using the AWS SDK. This allows users to simultaneously create both ATHENA and SPARK dialects of a view using cross-account IAM definer roles, enhancing automation for data mesh use cases.
- Cross-account definer roles for data mesh enablement
The use of cross-account definer roles is highlighted for enabling enterprise data mesh architectures. This setup allows producer accounts to manage CI/CD pipelines for data assets while a central governance account hosts databases and permissions.
- Key considerations for multi-dialect view creation
Specific points are made regarding the validation of ATHENA dialects requiring a one-time cross-account Glue connection setup, while SPARK dialects do not undergo validation. Database qualifiers are required for SPARK dialect view definitions.
- Adding ATHENA dialect to existing SPARK-only views
The post also demonstrates how to update existing AWS Glue views that were initially created with only the SPARK dialect to include the ATHENA dialect programmatically via the SDK.
https://aws.amazon.com/blogs/big-data/automate-creating-aws-glue-data-catalog-views-with-aws-sdk-for-data-mesh-use-case/
Related releases
- Amazon S3 enhances access denied errors with specific policy ARNs AWS What's New ·
- AWS IAM Role Manager Automates Service Role Setup AWS What's New ·
- AWS IAM Role Manager Streamlines Identity and Access Management Setup AWS Security Blog ·
- Terraform AWS Provider v6.59.0 Adds New Data Sources and Resources Terraform AWS Provider Releases ·
- AWS IAM simplifies role assignment for workforce users with account access manager AWS What's New ·
- Amazon Bedrock expands IAM principal cost allocation for mantle endpoint AWS What's New ·