Databricks Simplifies S3 Data Connection with Delegated IAM Permissions
Databricks has introduced a streamlined, automated flow for connecting Amazon S3 buckets to Unity Catalog, reducing manual configuration from days to minutes. This new method uses AWS IAM temporary delegation, eliminating the need for complex IAM policies and CloudFormation templates, and directly benefiting users by simplifying a foundational step for data ingestion, pipelines, and analytics. The process is now a few clicks within the Databricks workspace, automatically provisioning necessary storage credentials and external locations with least-privilege permissions. Users lacking sufficient AWS access can request it from their administrator directly within the flow, and the authorization automatically expires after provisioning, enhancing security.
- →Automated S3 external location creation with IAM delegation
- →Enhanced security with temporary, expiring IAM authorization
- →Streamlined user experience for S3 connectivity
- →Enables LTAP architecture by simplifying cloud storage integration
Security (1) ›
- Enhanced security with temporary, expiring IAM authorization
Databricks uses AWS IAM temporary delegation, meaning the authorization granted to Databricks to provision resources on your behalf is time-bounded and automatically expires. The system ensures Databricks never holds standing access to your AWS account, and all provisioning actions are logged in AWS CloudTrail for auditing. This approach aligns with enterprise security standards by creating least-privilege IAM roles.
Features (1) ›
- Automated S3 external location creation with IAM delegation
Databricks now offers a simplified, automated workflow for creating external locations to connect Amazon S3 buckets to Unity Catalog. This new flow leverages AWS IAM temporary delegation, reducing the setup complexity from writing 140-line IAM policies and configuring multiple AWS resources to a few clicks within the Databricks workspace. It automatically handles storage credential provisioning with least-privilege IAM roles and configures the external location, enabling Auto Loader and File Events.
Enhancements (1) ›
- Streamlined user experience for S3 connectivity
The update significantly simplifies connecting S3 data to Databricks, which is crucial for ingestion, pipelines, analytics, and the emerging LTAP architecture. Users previously faced a complex, multi-console setup process; this change allows them to set up secure S3 connections in minutes. A new flow within Catalog Explorer guides users through the process, including the option to request necessary AWS permissions from administrators if they lack them.
Notes (1) ›
- Enables LTAP architecture by simplifying cloud storage integration
The simplified S3 connectivity removes a key infrastructure barrier for adopting the Lake Transactional/Analytical Processing (LTAP) architecture. This allows operational data in cloud storage to be instantly queryable by analytical engines without requiring separate pipelines or complex data replication.
https://www.databricks.com/blog/connect-amazon-s3-data-databricks-delegated-iam-permissions
Related releases
- Databricks Terraform Provider v1.123.0: Improved Plan Validation, Delta Sharing, MLflow, and Bug Fixes Terraform Databricks Provider Releases ·
- Databricks SDK Java v0.137.0 adds AI Gateway and deployment fields Databricks Java SDK Releases ·
- Databricks SDK for Go v0.166.0 Enhances Workspace Services Databricks Go SDK Releases ·
- NorthStar Anesthesia builds clinician scheduling app on Databricks Apps in weeks Databricks Blog ·
- Databricks AI Search adds high-QPS scaling Databricks Blog ·
- AI Applications and Best Practices in Healthcare Databricks Blog ·