Amazon SageMaker AI now supports the G7 instance family, which uses NVIDIA Blackwell GPUs, alongside the existing G5 (A10G) and G6 (L4) families for small LLM inference. The change introduces a new hardware generation that can achieve lower latency, higher throughput, and reduced cost‑per‑token even when using fewer GPUs and less total GPU memory.
What Changed: G7 Instances and Blackwell GPUs
The G7 line replaces the prior G5 and G6 generations for inference workloads. A G7 ml.g7.12xlarge node provides two RTX PRO 4500 Blackwell GPUs with 64 GB of combined memory, whereas G5 (ml.g5.12xlarge) and G6 (ml.g6.12xlarge) each provision four GPUs (A10G or L4) with 96 GB total memory. The Blackwell architecture brings a new generation of tensor cores and memory bandwidth, which the SageMaker AI Generative AI inference recommendation engine can evaluate automatically.
Benchmark Findings Across G5, G6, and G7
Two 30‑billion‑parameter Mixture‑of‑Experts models were used to compare the families:
- AI coding assistant: Qwen3‑Coder‑30B deployed via the DJL Large Model Inference (LMI) container.
- Enterprise AI assistant: NVIDIA Nemotron‑3‑Nano‑30B evaluated with vLLM and the Generative AI inference recommendation workflow.
Across both use cases, the G7 configuration consistently showed:
- Reduced time‑to‑first‑token and overall latency compared with G5 and G6.
- Higher request‑per‑second throughput despite having half the GPU count.
- Lower cost‑per‑token when measured against the on‑demand pricing of the older families.
These outcomes were observed even though G7 provides less aggregate GPU memory, indicating that the Blackwell GPUs’ efficiency outweighs raw memory capacity for the tested workloads.
Architectural and Operational Implications
Practitioners need to consider several concrete impacts when adopting G7 for LLM inference:
- Instance sizing: Because G7 uses two GPUs, deployment scripts that assume four‑GPU layouts (e.g., environment variables for
CUDA_VISIBLE_DEVICES) may require adjustment. - Memory planning: Models that rely on the full 96 GB memory pool of G5/G6 must be verified against the 64 GB limit of G7, possibly using model quantization (FP8) or off‑loading techniques.
- Cost modeling: The recommendation engine can now suggest G7 as the optimal choice for latency‑sensitive or cost‑constrained services, simplifying capacity planning.
- Automation workflow: The two‑step approach—direct benchmarking with LMI for known endpoints and automated recommendation with vLLM for new deployments—provides a repeatable path to validate G7 suitability before production rollout.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
When evaluating small LLM inference on SageMaker AI, start by running the built‑in Generative AI inference recommendation for your model and workload. If the recommendation includes a G7 option, verify that your model fits within the 64 GB memory envelope and update any multi‑GPU orchestration logic to handle a two‑GPU topology. Use the LMI container to perform a quick latency/throughput benchmark on a G7 endpoint; compare the results against existing G5/G6 deployments to confirm the expected gains. Finally, incorporate the cost‑per‑token metric into your budgeting process, as the G7’s efficiency can translate into measurable savings without sacrificing performance.
