Amazon EKS Auto Mode automates node failure detection and repair

Amazon EKS Auto Mode automatically detects, drains, and replaces failing nodes using the Node Monitoring Agent (NMA) and Karpenter. The NMA translates low-level system signals into Kubernetes node conditions, which Karpenter consumes to initiate node replacement. On EKS Auto Mode, these components run automatically; for other EKS compute options, NMA can be installed as an add-on. The NMA runs as a systemd service baked into the AMI, ensuring continuous operation even during degraded pod scheduling.

The Node Monitoring Agent detects faults across kernel, container runtime, networking, storage, and hardware, reporting them as Kubernetes node conditions. Karpenter, the compute controller, reads these conditions and performs node replacement as its sole repair action. This architecture, where the agent reports to the Kubernetes API and Karpenter acts upon it, ensures reliability through decoupling.

The agent groups detections into five conditions: KernelReady, ContainerRuntimeReady, NetworkingReady, StorageReady, and AcceleratedHardwareReady. Faults are assigned either 'Condition-severity,' triggering a 'False' status and potential replacement, or 'Event-severity,' which logs a Kubernetes event for visibility without affecting node service. Condition-severity faults are terminal issues requiring replacement.

Terminal faults include hardware issues like GPU device-count mismatches or ECC errors, network problems such as a down VPC CNI process, and runtime failures like pods stuck in Terminating state. Event-severity issues, which remain informational, include bandwidth ceilings, EBS IOPS throttling, and GPU thermal warnings.

Karpenter's repair policies define toleration windows for node conditions before repair initiates. For example, Accelerated Hardware Ready faults have a 10-minute window, while others have 30 minutes. A fleet-wide safety gate prevents repair if more than 20% of nodes in a NodePool or cluster are unhealthy, and repair is also paused during Amazon ARC zonal shifts.

Once the toleration window is met and safety gates are passed, Karpenter taints the node, drains pods respecting Pod Disruption Budgets, terminates the instance, and launches a replacement. For stateful workloads, successful recovery relies on the workload's ability to restart from a checkpoint after the node is replaced and pods are rescheduled.