AWS Spark Upgrade Agent simplifies Scala migration to Spark 4.x

This post demonstrates automating Spark 3.3 (Scala 2.12) to Spark 4.0 (Scala 2.13) migration on Amazon EMR 8.x using the AWS Spark Upgrade Agent. It addresses API deprecations, behavioral changes, and build configuration, transforming months of manual effort into hours.

The AWS Spark Upgrade Agent is a managed remote server that automates Spark migrations via a Model Context Protocol (MCP) interface. Its architecture emphasizes security with scoped IAM roles, local source code handling, encryption in transit, and AWS CloudTrail for auditability.

A large company with thousands of Spark jobs across Scala, PySpark, and Spark SQL migrated from Spark 3.3/3.5 to Spark 4.0 on Amazon EMR 8.x using the agent. The agent iteratively validates and transforms code, automatically updating build configurations and resolving runtime failures, significantly reducing manual engineering.

Before starting the migration, users need AWS CLI configuration with necessary permissions, an IAM role with Amazon EMR permissions, an Amazon S3 staging bucket, an IDE with MCP support (Kiro or VS Code with Cline extension), and a configured MCP server. An Amazon EMR Serverless application for Spark 4.0 is also required for validation.

Two AWS CloudFormation stacks are provided to set up the required infrastructure: one for the IAM role and S3 staging bucket, and another for the Amazon EMR Serverless target application (Spark 4.0) and its execution role. These stacks automate resource provisioning for the upgrade process.