SageMaker HyperPod enhances LLM inference with Disaggregated Prefill and Decode
Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), an optimization that separates LLM inference prefill and decode phases onto dedicated GPU pools to improve per-token latency and throughput for production LLM workloads. This allows for independent scaling of compute and memory bandwidth resources, benefiting chat assistants, agentic pipelines, and RAG applications. DPD is enabled via the InferenceEndpointConfig custom resource and is available for SageMaker HyperPod clusters using EKS on EFA-capable instance types in all available AWS regions.
- →SageMaker HyperPod adds Disaggregated Prefill and Decode (DPD) for LLM inference
- →DPD provides consistent latency and higher throughput under concurrency
- →Intelligent router optimizes DPD usage for different prompt lengths
- →DPD configuration integrated into existing SageMaker HyperPod InferenceEndpointConfig
Features (1) ›
- SageMaker HyperPod adds Disaggregated Prefill and Decode (DPD) for LLM inference
SageMaker HyperPod now supports DPD, an optimization that separates LLM inference prefill and decode phases onto dedicated GPU pools. This uses GPU-Direct RDMA over EFA to transfer the KV cache, improving per-token latency and throughput for production LLM workloads. The optimization is available for SageMaker HyperPod clusters using EKS on EFA-capable instance types in all available AWS regions.
Enhancements (3) ›
- DPD provides consistent latency and higher throughput under concurrency
By running compute-bound prefill and memory-bandwidth-bound decode on separate GPU pools, DPD eliminates resource contention. This leads to more consistent per-token latency under sustained concurrency and higher goodput at strict latency SLOs. Customers can also independently scale prefill and decode capacity to match workload distributions.
- Intelligent router optimizes DPD usage for different prompt lengths
An intelligent router automatically directs long-context requests through the disaggregated path for DPD benefits. Shorter prompts are sent directly to the decoder without transfer overhead, ensuring efficiency for all traffic types. DPD is composable with existing KV cache offloading and intelligent routing features on HyperPod.
- DPD configuration integrated into existing SageMaker HyperPod InferenceEndpointConfig
Customers can enable DPD by adding a `pdSpec` section to the `InferenceEndpointConfig` custom resource. This integration simplifies adoption for users already familiar with configuring inference endpoints on the HyperPod Inference Operator.
https://aws.amazon.com/about-aws/whats-new/2026/7/amazon-sagemaker-hyperpod-dpd/