The latest addition to AWS’s AI model catalog is the Nemotron 3.5 Super VL, a 120B‑parameter mixture‑of‑experts (MoE) model that activates only 12B parameters per token. It supports text, image, document, and video inputs and produces text output, making it a true multimodal engine for agentic workflows. For engineers, the change means a single model can replace separate text‑only and vision models, but the deployment footprint and runtime considerations differ from traditional dense models.
Model characteristics and hardware baseline
Nemotron 3.5 Super VL combines a hybrid Mamba‑2, attention, and expert layer stack. Sparse routing limits active compute to 12B parameters per token, preserving memory while still holding the full weight set on the GPU. The Mamba‑2 layers keep a constant‑size state, avoiding KV‑cache growth for long contexts, and the model supports up to 1 million tokens per request. Checkpoints are available in BF16 and NVFP4 formats.
Benchmark results reported by NVIDIA show 63.44 % on Hermes, 87.05 % on LangChain agent evaluations, and 80.8 % on the CharXiv chart‑reasoning benchmark. These numbers illustrate strong reasoning and visual‑understanding capabilities relative to prior Nemotron releases.
Minimum GPU configurations for inference with the vLLM server are:
- B200/GB200: 8 GPUs for
BF16, 2 GPUs forNVFP4 - B300/GB300: 4 GPUs for
BF16, 1 GPU forNVFP4 - H200: 8 GPUs for
BF16, 4 GPUs forNVFP4 - H100: 16 GPUs for
BF16, 8 GPUs forNVFP4
Running on Amazon EKS and EKS Auto Mode
Deploying the model on Amazon EKS follows the standard container‑native inference pattern. A Kubernetes Deployment runs the vLLM server, exposing a REST endpoint for downstream agents. EKS Auto Mode can provision the required node groups automatically, matching the GPU count to the selected model format. Practitioners should monitor node‑level GPU utilization because the MoE routing can cause uneven load across GPUs, especially when batch sizes vary.
Operational considerations include:
- Ensuring the node group uses a GPU‑compatible AMI that supports the chosen
BF16orNVFP4runtime. - Configuring pod resource requests to reflect the minimum GPU count; under‑provisioning leads to pod scheduling failures.
- Leveraging horizontal pod autoscaling based on request latency rather than CPU metrics, as GPU usage dominates.
Using SageMaker HyperPod EKS for large‑scale serving
SageMaker HyperPod EKS extends the EKS model by providing a managed, high‑throughput networking fabric optimized for multi‑GPU workloads. The same container image used on vanilla EKS can be launched as a SageMaker model endpoint, benefitting from built‑in health checks, versioning, and automatic scaling across HyperPod clusters.
Key implications for platform teams:
- HyperPod clusters simplify the networking topology required for 16‑GPU H100 deployments, reducing the need for custom CNI tuning.
- Model version rollout can be orchestrated through SageMaker’s endpoint update API, allowing zero‑downtime swaps while retaining the same VPC security posture.
- Observability integrates with SageMaker Model Monitor, but custom metrics for MoE routing efficiency may still need to be exported from the vLLM container.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Nemotron 3.5 Super VL offers a unified multimodal engine that can replace separate LLM and vision models, but the cost and complexity of the required GPU clusters remain significant. Engineers should evaluate whether the MoE efficiency offsets the larger memory footprint on the chosen GPU family. When using EKS, plan for explicit GPU node sizing and autoscaling based on latency. For large‑scale, production‑grade deployments, SageMaker HyperPod EKS provides managed networking and lifecycle features that reduce operational overhead, though custom monitoring of MoE routing patterns is advisable. Early testing with the minimum GPU configurations listed above will reveal the practical throughput and scaling characteristics before committing to a full‑scale rollout.