Deploying the NVIDIA CUDA Multi‑Process Service (CUDA MPS) alongside the Triton Inference Server on Amazon EC2 reduced the GPU fleet needed for real‑time ASR from sixteen L40S instances to four, while keeping sub‑second latency and a throughput of 92.1 requests per second per GPU. Practitioners who run inference workloads with low per‑request GPU utilization can cut infrastructure spend and simplify operations by consolidating workloads onto fewer GPUs.
Why the change matters
ASR inference on the Parakeet TDT 0.6B V2 model consumes only about 15‑20 % of an L40S GPU’s streaming multiprocessors per request. The default CUDA time‑slicing model gives each process exclusive access, forcing the remaining SM capacity to sit idle and limiting a single GPU to roughly 62 RPS. By enabling concurrent kernel execution, CUDA MPS fills the idle SMs, allowing four inference streams to share a GPU and achieve the reported 92.1 RPS. The result is a 75 % reduction in required GPU instances, directly lowering EC2 costs and the operational surface area.
GPU sharing mechanisms compared
- Time‑slicing (default): full context switch, sequential execution, suited for a few large models.
- MIG (Multi‑Instance GPU): hard physical partitions, concurrent execution with fixed resource slices, ideal for strict multi‑tenant isolation.
- CUDA MPS: soft SM limits, shared context, concurrent kernels, minimal code changes, best for many small models on a single GPU.
The Heidi Health deployment chose CUDA MPS because it required high concurrency without redesigning the model serving stack.
Implementation sketch
Key steps, distilled from the source, are:
- Launch an EC2 instance with an NVIDIA L40S GPU.
- Install the NVIDIA driver, CUDA toolkit, and Triton Inference Server.
- Start the MPS daemon on the host (e.g.,
nvidia-cuda-mps-control -d). - Set
CUDA_MPS_ACTIVE_THREAD_PERCENTAGEto allocate the desired SM share per client process. - Deploy the ASR model in ONNX or TensorRT format to Triton, which handles request scheduling across the shared GPU.
Because MPS presents a single CUDA context, existing inference code runs unchanged. The only operational addition is managing the MPS daemon lifecycle and monitoring the active thread percentage to avoid over‑commitment.
Operational and security considerations
Consolidating four inference streams onto one GPU reduces instance count, but introduces new responsibilities:
- Resource monitoring: Track SM utilization and request latency to ensure the configured thread percentage does not become a bottleneck.
- Daemon reliability: The MPS control process must be kept running; a crash would revert to default time‑slicing and degrade performance.
- Memory isolation: While MPS separates address spaces, all processes share the same GPU memory pool, so runaway allocations in one process can affect others.
- Scaling strategy: Evaluate whether additional GPUs are needed for peak traffic spikes beyond the 92.1 RPS per GPU ceiling.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopting CUDA MPS on EC2 allows AI engineers and platform teams to pack more inference workloads onto fewer GPUs, delivering cost savings and a smaller attack surface. The trade‑off is added operational complexity around daemon management and careful capacity planning. Teams should prototype MPS with their own models, measure SM utilization, and establish monitoring alerts for latency and memory pressure before committing to a production‑scale reduction.
