SageMaker HyperPod adds data capture for inference workloads

SageMaker HyperPod now records inference request and response payloads from production endpoints to Amazon S3, providing visibility into model inputs and outputs. This feature helps detect drift, troubleshoot issues, build evaluation datasets, and improve deployed models.

Captured data can be used to train speculative decoding draft models, build evaluation pipelines, feed fine-tuning jobs with real-world inputs, and maintain audit trails for compliance. This eliminates the need for custom logging pipelines outside the service.

Customers can choose to capture inference traffic at the SageMaker endpoint, load balancer, or model pod. Captured data is delivered asynchronously to Amazon S3 without blocking inference, supporting configurable sampling and customer-managed AWS KMS encryption.

This feature is available for SageMaker HyperPod clusters using the EKS orchestrator in all AWS Regions where SageMaker HyperPod is supported. It can be enabled when deploying models and used with SageMaker Model Monitor and existing workflows.