GPU sharing on Kubernetes no longer benefits from the automatic per‑container accounting that CPUs and memory enjoy. The kernel supplies usage counters for CPU and RAM, but GPUs expose only device‑level data, so teams must construct their own metrics, rightsizing logic, guardrails, and chargeback models to keep costs and performance predictable.
Metrics are no longer automatic
The default NVIDIA DCGM exporter reports statistics at the physical GPU level. When a single pod owns a GPU the numbers are useful, but as soon as time‑slicing, MPS, or MIG is used to share a card, every pod receives the same aggregate values. A three‑pod workload will appear to consume 300 % of a GPU that physically exists, inflating dashboards with misleading data. Operators need to supplement DCGM with per‑process information from NVML or custom instrumentation that can differentiate memory footprints and, where possible, compute slices.
Rightsizing and scheduling differ from CPU
CPU requests are expressed in millicores, allowing overcommit and throttling; GPU requests are expressed as whole integers (nvidia.com/gpu: 1) with matching limits, and the scheduler does not support fractional allocation. Fractional usage is achieved only after the pod lands on a node, via driver‑level time‑slicing or MIG, which the scheduler cannot account for. Consequently, the classic rightsizing loop—compare request vs. actual use and reclaim excess—does not apply directly. Teams must devise their own conversion from observed memory or compute signals into an estimated GPU share before they can discuss waste.
Guardrails can be bypassed
The NVIDIA container runtime respects the NVIDIA_VISIBLE_DEVICES environment variable, which base CUDA images set to all. Normally the device plugin overwrites this variable with the GPUs allocated to the pod, but if the plugin is missing, misconfigured, or the runtime is instructed to honor the container‑provided value, a pod can see and claim every GPU on the node regardless of the scheduler’s assignment. This divergence between the scheduler’s view and the actual hardware usage can lead to silent contention and unexpected cost spikes. Verifying that the plugin correctly injects the variable is essential.
Utilization and attribution are ambiguous
GPU utilization reported by DCGM or nvidia‑smi reflects the fraction of time the kernel was active, not the proportion of compute work or memory bandwidth consumed. A card can sit at 100 % utilization while doing little useful work, or memory can be fully allocated at startup (e.g., by vLLM) even if compute is idle. NVML can provide per‑process memory usage, which maps cleanly to pods and teams, but compute time is only a binary “was something running?” flag. When multiple pods truly run side‑by‑side, current tooling cannot attribute compute percentages accurately. For chargeback, memory usage is the most reliable metric; compute metrics should be treated as a supplemental signal.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Operators should adopt a multi‑layer monitoring approach: use DCGM for overall health, augment with NVML per‑process memory data, and, where possible, enable accounting mode or PID‑level sampling to approximate compute shares. Validate that the device plugin correctly overrides NVIDIA_VISIBLE_DEVICES to prevent accidental full‑device exposure. Build rightsizing calculations that translate observed memory footprints into an estimated GPU fraction before reclaiming capacity. When designing billing models, prioritize memory‑based attribution and treat compute percentages as a trend indicator rather than a precise charge. Finally, evaluate the sharing paradigm (time‑slicing, MPS, MIG) that best matches workload characteristics and operational maturity, and document the assumptions behind any custom metrics or guardrails you introduce.