GPU‑aware inference routing with SageMaker HyperPod Inference Gateway

Amazon SageMaker now ships a GPU‑aware routing layer called the HyperPod Inference Gateway, delivered as an EKS managed addon that sits in front of your model‑serving pods. It replaces default Kubernetes load balancers with a router that inspects real‑time GPU metrics and directs each request to the pod best positioned to serve it, cutting first‑token latency by up to 82% and improving GPU utilization without any changes to your application code.

Why Traditional Load Balancing Misses the Mark

Round‑robin or least‑connections policies treat every pod as identical, ignoring the state of the GPU memory, cache, or loaded adapters. When a busy pod receives a new request, the request queues while idle capacity on other pods sits unused, leading to latency spikes and over‑provisioning.

SageMaker HyperPod Inference Gateway Architecture

The gateway follows a two‑tier design, though only the first tier is currently available.

Tier 1 – Per‑cluster gateway (EKS managed addon)

Installed as amazon-sagemaker-hyperpod-inference, it comprises three components built on the open‑source Gateway API Inference Extension:

• Envoy Gateway: an L7 proxy that terminates HTTPS traffic and exposes a single private endpoint per cluster.
• Body‑Based Router (BBR): parses OpenAI‑compatible request bodies, extracts the model field, and routes to the appropriate model pool, supporting many models behind one endpoint.
• Endpoint Picker (EPP): consumes Prometheus metrics from every model‑serving pod and applies a weighted scoring algorithm. Scoring factors include KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and number of running requests. Each factor’s weight is configurable, allowing teams to prioritize latency or throughput.

Tier 2 – Global Inference Router (planned)

The upcoming tier will coordinate routing across clusters and regions, adding cross‑cluster failover, global rate limiting, and cost‑aware traffic shaping. It builds on the per‑cluster gateway, which continues to handle local routing.

Deploying the Inference Gateway

Installation requires a single aws eks create-addon call, followed by labeling model pods and applying an InferenceGatewayConfig custom resource. No sidecars, service mesh, or application changes are needed.

aws eks create-addon \
  --cluster-name my-hyperpod-cluster \
  --addon-name amazon-sagemaker-hyperpod-inference \
  --addon-version v2.0.0-eksbuild.1 \
  --configuration-values '{"inferenceGateway": {"enabled": true}, "inferenceOperator": {"enabled": true}}'

Label your model deployments so the gateway can discover them:

spec:
  template:
    metadata:
      labels:
        app: vllm-llama  # gateway matches on this label

Define routing rules in a single InferenceGatewayConfig resource:

apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1
kind: InferenceGatewayConfig
metadata:
  name: my-gateway
spec:
  tls: {}
  bbr:
    enabled: true
  schedulers:
    - name: llama-70b
      modelName: "llama-3.1-70b"
      modelSelector:
        matchLabels:
          app: vllm-llama
      targetPort: 8000
      scheduler: llm-d

Operational and Security Considerations

Because routing decisions rely on Prometheus metrics, teams must ensure metric collection is reliable and that the scoring weights reflect their workload characteristics. Adjusting weights can shift the balance between latency‑sensitive chat use cases and batch‑oriented throughput.

The gateway terminates HTTPS at the Envoy layer and exposes only a private endpoint per cluster, keeping traffic inside the VPC. No additional sidecars or mesh components are introduced, reducing the surface area for configuration errors.

Related CloudNinjas coverage: AWS.

What This Means For Practitioners

Adopting the HyperPod Inference Gateway gives you a drop‑in routing layer that can immediately lower first‑token latency and improve GPU efficiency without refactoring model servers. Practitioners should validate metric pipelines, experiment with scoring weights, and monitor the private endpoint for any unexpected traffic patterns. Keep an eye on the announced Tier 2 Global Inference Router if you operate multi‑cluster or multi‑region fleets, as it will extend the same routing intelligence across broader deployments.