Fast Model Loading for AI Inference on Amazon EKS

AWS has identified configuration-only changes to accelerate AI model loading on Amazon EKS, specifically targeting large language models. These optimizations significantly cut cold-start times for inference workloads that load large model weights into GPU memory, impacting engines like vLLM and SGLang using Run:ai Model Streamer.

Analysis revealed that for AI models under ~100 GiB, `torch.compile` (PyTorch's model compiler) is the primary cold-start bottleneck, while for larger models, loading weights from Amazon S3 dominates. This shift occurs because `torch.compile` time is relatively constant, whereas weights loading scales linearly with model size.

AWS opted for config-only tuning and contributions to the publicly available Run:ai Model Streamer to ensure rapid deployment and broad benefit across Amazon EC2, Amazon ECS, and Amazon SageMaker. This approach avoids proprietary EKS features or complex DaemonSets, leveraging existing integrations with common inference engines.

Investigations into Run:ai Model Streamer's S3 downloading mechanism found that using a 4 GiB chunk size for model weights, which often matches SafeTensors shard sizes, provides optimal throughput. This single chunk per file approach minimizes serial requests and reduces the number of required TCP connections, simplifying configuration.

To mitigate the impact of long-tail S3 GET latencies, an aggressive timeout and retry mechanism was implemented for slow requests. If a request falls below a minimum speed threshold, it is killed and retried on a fresh connection, ensuring that a single stalled connection does not delay the entire model loading process.