ReleaseBytes
aws AWS Big Data Blog ·

GPU-accelerated Apache Spark on Amazon EMR with EC2 G7 instances runs up to 3.7x faster

blogdataawsgaengineerhealthcareaws-ec2aws-eks
feature

AWS now supports GPU-accelerated Apache Spark on Amazon EMR on EKS, leveraging EC2 G7 instances with NVIDIA RTX PRO 4500 GPUs. This integration provides up to 3.7x faster performance for computationally intensive AI/ML feature engineering, ETL, and real-time analytics workloads, requiring no changes to existing Spark code. Data engineers and data scientists can significantly reduce job run times and operational costs. Availability requires Amazon EMR on EKS release 6.9.0 or later and a GPU-enabled EKS cluster with G7 instances.

  • →GPU Acceleration for Apache Spark on Amazon EMR on EKS
  • →Improved Performance for AI/ML and Data Processing
  • →Underlying Technology and Benchmarking Results
  • →Prerequisites for Implementation
Features (1) ›
  • GPU Acceleration for Apache Spark on Amazon EMR on EKS

    AWS has enabled GPU-accelerated Apache Spark workloads on Amazon EMR on EKS using Amazon EC2 G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. This integration allows Spark jobs to run up to 3.7x faster than comparable CPU instances, without requiring any changes to existing Spark code.

Notes (3) ›
  • Improved Performance for AI/ML and Data Processing

    The GPU acceleration significantly speeds up computationally intensive tasks such as AI/ML feature engineering, large-scale ETL transformations, and real-time analytics workloads. This allows data science teams to iterate on models faster, compress multi-hour batch windows, and process larger data volumes within tighter latency windows.

  • Underlying Technology and Benchmarking Results

    Amazon EMR on EKS natively supports the NVIDIA cuDF plugin for Apache Spark, a result of joint engineering between AWS and NVIDIA. Benchmarking on a TPC-DS 3 TB dataset demonstrated that EC2 G7 instances with RTX PRO 4500 GPUs completed jobs in 4.7 minutes at the 64 GB memory tier, outperforming CPU baselines.

  • Prerequisites for Implementation

    To utilize GPU-accelerated Spark, users need Amazon EMR on EKS release 6.9.0 or later, an Amazon EKS cluster with a GPU-enabled node group using G7 instances, and the NVIDIA device plugin installed. The AL2023_x86_64_NVIDIA EKS optimized accelerated AMI is recommended.

Read the original announcement →

https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/

Related releases