ReleaseBytes
aws AWS Big Data Blog ·

Amazon EMR Serverless adds larger worker option for intensive Spark workloads

blogdataawsengineerhealthcare
feature

Amazon EMR Serverless now offers a 32 vCPU / 244 GB worker configuration to support demanding Spark jobs like large-scale joins and shuffle-intensive ETL. This larger worker size, combined with shuffle-optimized disks, aims to improve price-performance for customers migrating complex workloads from traditional Spark clusters. Benchmarks show an average of 29% faster query execution and 29% lower cost compared to smaller worker configurations.

  • →New 32 vCPU / 244 GB worker configuration
  • →Shuffle-optimized disk for improved I/O
  • →Benchmark results demonstrate performance and cost gains
Features (1) ›
  • New 32 vCPU / 244 GB worker configuration

    Amazon EMR Serverless now provides a larger worker option with 32 vCPUs and 244 GB of memory, configurable with up to 2,000 GB of shuffle-optimized disk. This configuration is designed to handle shuffle-intensive, I/O-heavy, and memory-intensive workloads.

Enhancements (1) ›
  • Shuffle-optimized disk for improved I/O

    The shuffle-optimized disk offers higher throughput and IOPS, benefiting queries that scan large datasets or spill intermediate data. This is particularly advantageous for shuffle-intensive workloads that redistribute large amounts of data, reducing shuffle wait time by keeping more data local to executors.

Notes (1) ›
  • Benchmark results demonstrate performance and cost gains

    Testing across TPC-DS and TPC-H benchmarks showed that the new larger workers delivered an average of 29% faster query execution and 29% lower query-attributed cost. Improvements peaked at 45-55% for shuffle-heavy, multi-table join queries, with larger workers winning every query on both performance and cost.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/accelerate-spark-on-emr-serverless-with-larger-workers-and-shuffle-optimized-disks/

Related releases