AWS Glue 6.0 Simplifies ETL with Spark Declarative Pipelines
blogdataawsgaengineer
patch
AWS Glue 6.0 introduces Spark Declarative Pipelines (SDP), enabling data teams to define ETL workflows by declaring dataset contents rather than manually orchestrating jobs. This new capability, powered by Apache Spark 4.1, automates dependency resolution, checkpoint management, and execution order within a single Glue job. It significantly reduces boilerplate code, removing the need for external orchestrators for internal pipeline coordination. Data engineers can now build medallion architectures with less manual effort, with the feature now generally available.
Maintenance (1) ›
- Build the pipeline files (the spark-pipeline.ymlspecification plus the three transformation files)
Read the original announcement →
https://aws.amazon.com/blogs/big-data/build-declarative-etl-pipelines-with-aws-glue-6-0/
Related releases
- Terraform AWS Provider v6.64.0 Adds New Resources and Bedrock Agent Enhancements Terraform AWS Provider Releases ·
- AWS Lambda now supports 90-minute function timeouts on Managed Instances AWS What's New ·
- AWS Launches Deception Benchmark for AI Security Tool Evaluation AWS Security Blog ·
- AWS Lambda Managed Instances now support Graviton5-powered EC2 instances AWS What's New ·
- Amazon EMR Extends No-Cost Support for Older Releases During Migrations AWS What's New ·
- AWS Systems Manager Enhances Diagnosis for Unmanaged EC2 Instances AWS What's New ·