Amazon SageMaker AI added 13 new capabilities to its two inference deployment models—managed endpoints and HyperPod—in 2026. The changes aim to reduce time‑to‑production, improve token‑level performance metrics, and give operators more control over scaling, observability and container handling, which are core concerns for AI, cloud, DevOps and security engineers.
SageMaker inference endpoint updates
The managed endpoint path now includes:
- Inference recommendations – an automated workflow that selects instance types, containers and optimization flags based on a model and a cost/latency/throughput goal.
- Capacity‑aware instance pools – pools that consider available GPU capacity when allocating resources.
- OpenAI‑compatible API – a drop‑in endpoint that speaks the OpenAI request format.
- Container caching – reuse of previously pulled containers to cut cold‑start latency.
- Observability extensions – token‑level metrics such as time‑to‑first‑token (TTFT) and inter‑token latency (ITL) are exposed for monitoring.
- Async inference with inline payloads – asynchronous request handling that returns results in the same payload format.
- Prefix‑aware routing – routing decisions that can consider request prefixes, useful for multi‑model serving.
SageMaker inference HyperPod updates
The Kubernetes‑native HyperPod path received:
- Simplified operator – a lighter‑weight Kubernetes operator for provisioning dedicated GPU clusters.
- Tiered KV cache – a hierarchical key‑value cache that can reduce repeated token lookups.
- Data capture – built‑in hooks to record request and response data for downstream analysis.
- Performance features – enhancements that target raw GPU throughput and latency.
- Disaggregated prefill and decode – separation of the initial token generation (prefill) from subsequent token decoding, allowing more efficient resource use.
- Model caching – persistent storage of model weights on the cluster to avoid re‑loading between invocations.
Inference recommendations workflow
The new recommendation service follows a three‑step process:
- Narrow – filters the instance catalog based on model size, architecture and memory needs.
- Optimize – applies goal‑specific techniques such as EAGLE 3.0 speculative decoding for throughput, kernel tuning for latency, and tensor parallelism for large models.
- Benchmark – runs NVIDIA AIPerf on real GPU hardware, producing statistically‑rigorous latency percentiles and throughput numbers.
The output is a SageMaker Model Package containing ready‑to‑deploy configuration and validated performance metrics. An example with a 20‑billion‑parameter model showed a two‑fold increase in token throughput without changing request latency, and the service incurs no extra charge beyond the underlying compute.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Engineers should evaluate whether the automated recommendation flow can replace manual benchmarking for new models, especially when cost or latency targets are strict. Teams using managed endpoints can adopt container caching and observability extensions to shrink cold‑start windows and gain token‑level insight for SLO tracking. HyperPod users can leverage the simplified operator and model caching to reduce cluster provisioning effort and avoid repeated weight loads. In both paths, the new routing and caching features introduce additional configuration points that must be validated in CI pipelines and monitored for resource contention. Finally, the capacity‑aware pools and disaggregated prefill/decode hint at a shift toward more granular resource scheduling, so operators should plan for updated autoscaling policies and ensure that monitoring dashboards capture the newly exposed token metrics.
