SageMaker HyperPod adds model caching for faster inference autoscaling

Amazon SageMaker HyperPod now includes model caching, an inference optimization that pre-loads model weights and container images onto cluster nodes. This significantly reduces cold start times for large language model (LLM) inference workloads, allowing pods to start in seconds instead of minutes. The feature helps alleviate bottlenecks in scaling LLM deployments by speeding up container image downloads and model weight loading. It is generally available in all regions where SageMaker HyperPod operates and can be enabled via the HyperPod Inference Operator.