AWS EMR on EKS Adds Apache Spark Troubleshooting Agent
AWS details how the Apache Spark troubleshooting agent, previously available for other EMR services, now supports Amazon EMR on EKS. This agent helps data engineers diagnose Spark failures by correlating logs and signals from various sources like S3, CloudWatch, and the Spark History Server. It aims to reduce mean-time-to-resolution for incidents from days to minutes by providing root cause analysis and code recommendations through natural language prompts.
- →Spark Troubleshooting Agent Extends to EMR on EKS
- →Natural Language Diagnosis and Root Cause Analysis
- →Unified Interface Across EMR Services
- →Managed Service with Enhanced Security
- →Input Gathering Specific to EMR on EKS
Notes (6) ›
- Spark Troubleshooting Agent Extends to EMR on EKS
The Apache Spark troubleshooting agent now supports Amazon EMR on EKS, allowing data engineers to diagnose Spark failures. The agent correlates signals from Spark driver/executor logs, event logs, and container termination data, which previously required manual investigation across multiple systems.
- Natural Language Diagnosis and Root Cause Analysis
Users can provide a single natural language prompt to the agent, which then retrieves Spark logs from Amazon S3 or CloudWatch and Spark event logs. It identifies the root cause and recommends fixes for code-related failures, aiming to reduce MTTR from days to minutes.
- Unified Interface Across EMR Services
The troubleshooting agent offers a single interface for diagnosing failed Spark applications across Amazon EMR on EKS, EMR on EC2, EMR Serverless, AWS Glue, and SageMaker notebooks, eliminating the need to navigate different consoles and log locations.
- Managed Service with Enhanced Security
The agent runs as a fully managed MCP server, requiring no user deployment. It uses a single-tenant design for data isolation, with read-only operations governed by AWS IAM permissions and auditability through AWS CloudTrail.
- Input Gathering Specific to EMR on EKS
For EMR on EKS, the agent automatically handles Spark driver and executor logs delivered to Amazon S3 or Amazon CloudWatch Logs. It also connects to the Amazon EMR Persistent UI for Spark History Server data, correlating information from these sources.
- LLM-Powered Analysis and Cost Details
Leveraging Amazon Bedrock and Retrieval Augmented Generation (RAG), the agent uses a large language model grounded in Spark and AWS troubleshooting expertise. This managed inference is provided at no additional cost, with users only paying for existing EMR on EKS resources.
https://aws.amazon.com/blogs/big-data/introducing-apache-spark-troubleshooting-agent-for-amazon-emr-on-eks/
Related releases
- Amazon EKS now supports advanced Kubernetes control plane configuration AWS What's New ·
- Amazon EKS 1.34+ enables forensic container checkpointing with Kubelet API AWS Containers Blog ·
- Amazon EKS introduces advanced Kubernetes control plane configuration AWS Containers Blog ·
- Amazon EKS Auto Mode automates node failure detection and repair AWS Containers Blog ·
- AWS Network Firewall adds forward proxy functionality AWS What's New ·
- CloudWatch adds managed Prometheus collectors AWS What's New ·