Amazon SageMaker AI now includes a built‑in concurrency sweep capability that automates the process of finding the optimal instance type and request load for a generative AI endpoint. By programmatically increasing simultaneous requests and measuring throughput and latency, engineers can determine the exact point where adding more traffic stops improving performance and starts violating latency targets.
How a concurrency sweep works
The sweep sends a series of controlled traffic bursts to a SageMaker endpoint and records two key metrics at each step:
- Throughput: tokens generated per second.
- Latency: time taken per request.
Increasing the concurrency—e.g., from 64 up to 1,024 simultaneous calls—produces a performance curve that reveals three actionable data points: the optimal concurrency level, the breaking point where latency exceeds the service‑level agreement, and the number of instances required to handle peak load.
Deploying the model for a sweep
The workflow starts by deploying the target model to an endpoint using the native vLLM container. In the example, the NVIDIA Nemotron‑3 Nano 30B model runs on an ml.g7e.2xlarge instance, which provides a Blackwell GPU suited for inference workloads. Configuration is supplied through environment variables prefixed with SM_VLLM_. A minimal snippet looks like:
VLLM_IMAGE = f"763104351884.dkr.ecr.{region}.amazonaws.com/vllm:0.19.1-gpu-py312-cu129-ubuntu22.04-sagemaker"
environment = {
"SM_VLLM_MODEL": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
"SM_VLLM_ENFORCE_EAGER": "true",
"SM_VLLM_TENSOR_PARALLEL": "..."
}
Deploying with the vLLM container ensures the endpoint can handle the token‑wise streaming mode required for generative workloads.
Running and analyzing the sweep
Once the endpoint is live, the sweep is launched via the CreateAIBenchmarkJob API, which is part of SageMaker AI Inference Recommendations. The service orchestrates the traffic pattern, collects metrics, and stores results in Amazon S3. Engineers then review the generated report to locate the concurrency level that maximizes throughput while keeping latency within the defined SLA. The report also provides a per‑instance capacity figure that can be multiplied by the desired instance count to meet expected traffic peaks.
Operational and security considerations
From an operations standpoint, the sweep replaces manual load‑testing loops, reducing the time to reach a production‑ready capacity plan. It also makes it easier to adjust the fleet when traffic patterns change, as a new sweep can be rerun with a single API call.
Security‑wise, the process requires an IAM execution role that grants SageMaker AI and S3 permissions. Practitioners should verify that the role follows the principle of least privilege and that service quotas for the chosen instance family (e.g., ml.g7e.2xlarge) are sufficient before starting a sweep.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopt the built‑in concurrency sweep to replace ad‑hoc load tests, use the vLLM container for efficient model serving, and let the generated capacity profile drive instance count decisions. Ensure IAM roles are scoped correctly and monitor service quotas to avoid deployment failures. Re‑run sweeps whenever model versions, token lengths, or traffic patterns shift to keep the endpoint right‑sized and cost‑effective.
