Run GPU batch inference on ECS with scale-to-zero managed instances

A reference architecture deploys a GPU batch inference pipeline on ECS using Managed Instances, SQS for job buffering, and AWS Application Auto Scaling. The service automatically scales down to zero tasks and instances when the SQS queue is empty, ensuring users pay only for active inference time.

The ECS Managed Instances capacity provider uses attribute-based instance selection to provision On-Demand GPU instances based on specified requirements like manufacturer (NVIDIA), minimum GPU memory (20 GB), and single-GPU constraint, selecting the most cost-effective instance type.

The system includes GPU health monitoring that automatically detects and replaces instances with hardware failures via DCGM. Native GPU metrics in Amazon CloudWatch Container Insights provide visibility into device-level utilization, memory, and thermal conditions.

Model weights are baked into container images during the build process, enabling offline operation and reducing cold start variability. A CUDA compatibility layer helps prevent version mismatches between the container image and host drivers, enhancing reliability during driver updates.

Amazon ECS Managed Instances now simplify GPU batch inference by handling instance provisioning, AMI updates, driver management, and security patching, removing the need for self-managed EC2 instances and complex Auto Scaling configurations.

The primary cost advantage is eliminating idle costs by scaling down to zero instances when no jobs are present. Reduced GPU management fees on ECS Managed Instances further enhance cost-effectiveness, especially for bursty job arrival patterns.