SageMaker HyperPod Inference has been extended to include NVIDIA Dynamo, enabling distributed inference for large language models. The addition matters to AI, cloud, and DevOps teams because it provides a managed path to run models that exceed a single instance’s GPU memory, while exposing controls for scaling, routing, and node health.
Integration Overview
The HyperPod add‑on installs several custom resources that together orchestrate a Dynamo deployment:
- HyperPod Inference Operator – reads an
InferenceEndpointConfig(IEC) and forwards thedynamoSpecsection to the Dynamo stack. - Dynamo Operator – creates the resources defined in a
DynamoGraphDeployment(DGD), such as the frontend service, model worker pods, replica counts, and resource requests. - Grove – groups related pods, handling multi‑node worker relationships and scaling policies.
- KAI Scheduler – performs gang scheduling on GPU resources, ensuring that all pods required for a multi‑node worker are placed together.
AWS Deep Learning Containers supply the Dynamo runtime, and the add‑on pins compatible component versions. Updating the add‑on selects a supported Dynamo release.
Deployment Options
Practitioners can choose between two configuration styles:
- Bring‑your‑own DGD – Existing Dynamo users keep their upstream manifests. The Dynamo operator handles deployment once the container image, storage, networking, and pod placement are aligned with the HyperPod cluster.
- InferenceEndpointConfig with
dynamoSpec– HyperPod users define a new Dynamo endpoint using the familiar IEC format and an OpenAI‑compatible request interface. The operator supplies defaults for KV‑aware routing, replica counts, and selects vLLM as the default engine (SGLang and TensorRT‑LLM are also supported).
Both paths support single‑instance and multi‑node topologies, and HyperPod continues to manage node health, replacement, and worker recreation.
Architectural and Operational Implications
Key changes introduced by the integration:
- Disaggregated prefill and decode – The Dynamo frontend routes prompts to a pool of prefill workers and tokens to a separate pool of decode workers. Each pool can scale independently, allowing prompt processing and token generation to be tuned separately.
- KV‑aware routing – Dynamo’s router tracks reusable prefix KV cache and current worker load, reducing recomputation for repeated prompts and balancing traffic across workers.
- Multi‑node parallelism – Workers may run on a single instance or span multiple instances using tensor and pipeline parallelism. Communication between workers, including KV cache transfer, travels over Elastic Fabric Adapter (EFA) for low‑latency networking.
- Health monitoring and recovery – HyperPod continuously checks node, GPU, and EFA health. On failure, it replaces the node and the operators recreate affected Dynamo workers, preserving endpoint availability.
- Scheduling guarantees – The KAI Scheduler’s gang scheduling ensures that all pods required for a multi‑node worker are co‑located, preventing partial deployments that could stall inference.
Operationally, teams gain a single point of control (the IEC) for endpoint lifecycle, while still being able to leverage existing Dynamo manifests if preferred. The separation of prefill and decode also introduces new scaling knobs that must be monitored to avoid over‑provisioning one pool while the other becomes a bottleneck.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
- Evaluate whether your LLM exceeds the memory of a single
ml.p5.48xlargeinstance; if so, plan a disaggregated deployment. - Choose the configuration style that matches your existing workflow – keep existing DGD files or adopt the IEC‑based
dynamoSpecfor a more integrated experience. - Monitor KV cache reuse metrics to verify that the KV‑aware router is delivering the expected reduction in prompt recomputation.
- Leverage HyperPod’s health checks and node replacement to design for high availability; ensure your monitoring stack captures node, GPU, and EFA health events.
- Adjust prefill and decode replica counts independently based on observed prompt length and token generation latency.


