Engineers can now run multiple isolated Kubernetes clusters that share a single physical host and still allocate GPU memory and compute in slices, thanks to a combination of Kamaji and HAMi running on a simulated GPU environment. The change removes the need to spin up a dedicated control plane for each team and avoids the coarse‑grained GPU allocation of whole cards.
Why Combine Kamaji and HAMi
Kamaji solves the “many clusters” problem by running each tenant’s control plane as a pod inside a management cluster, eliminating a separate control‑plane VM per tenant. HAMi addresses the “GPU granularity” problem by exposing gpumem and gpucores resources, allowing pods to request a fraction of a GPU’s memory or compute. Because the two projects operate at different layers—control plane vs. node‑level scheduling—they can be stacked without overlap.
Architecture Overview
The management cluster hosts Kamaji, cert‑manager, and MetalLB. Each tenant control plane runs as a pod in the tenants namespace and includes the API server, scheduler, controller‑manager, and a Konnectivity tunnel. Tenant workers are separate containers (based on kindest/node:v1.36.4) that join the tenant cluster via kubeadm. HAMi’s mock device plugin runs on these workers, advertising fake GPUs through node annotations. The only shared components are the management cluster and the Kamaji etcd datastore, which keep tenant data isolated.
Implementation Steps
- Create the management cluster and supporting services. Use
kind create cluster --name kamaji, install cert‑manager versionv1.21.2, and deploy MetalLB versionv0.16.1. Configure an IP address pool that matches the kind network subnet. - Deploy Kamaji. Add the Clastix Helm repo and install the chart with
helm upgrade --install kamaji clastix/kamaji --namespace kamaji-system --create-namespace --set 'resources=null' --version 0.0.0+latest. - Provision tenant control planes. Apply a
TenantControlPlanemanifest fortenant-aandtenant-busing the tenant versionv1.36.4. After the pods are ready, extract each tenant’s admin kubeconfig from the secret${t}-admin-kubeconfig. - Start a worker node per tenant. Run a privileged
kindest/nodecontainer, join it to the tenant cluster with the token generated bykubeadm token create --print-join-command, and mount the host’s/lib/modulesread‑only. - Install HAMi on each worker. Deploy the mock device plugin, webhook, scheduler extensions, and quota controller. Annotate the node with the simulated GPUs, e.g., one
NVIDIA A100 80 GBfor tenant‑a and twoNVIDIA L4 24 GBfor tenant‑b.
After completing the steps, each tenant can schedule pods that request, for example, gpumem: 4096Mi and gpucores: 25%. The lab reports that all eight validation tests passed, confirming that the Konnectivity tunnel, HAMi webhook, and fractional GPU scheduling work together.
Operational and Security Implications
Running control planes as pods reduces infrastructure overhead but introduces a shared etcd datastore; operators must ensure that Kamaji’s namespace isolation and etcd access controls remain correctly configured. Because tenant workers are separate containers, a compromise of one worker does not automatically grant access to another tenant’s API server, preserving a degree of isolation. However, the only shared network path is the Konnectivity tunnel, so monitoring tunnel health and access logs is advisable.
HAMi’s mock plugin does not interact with real NVIDIA drivers, so the pattern is safe for development and CI pipelines. When moving to physical GPUs, the same manifests apply, but the mock plugin must be replaced with the real device plugin, and node annotations must reflect actual hardware.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Teams can evaluate a single‑host, multi‑tenant Kubernetes environment that offers both control‑plane isolation and fine‑grained GPU sharing without purchasing additional hardware. The approach is reproducible: the exact Helm chart versions, container images, and YAML snippets are provided in the lab. Practitioners should verify Konnectivity tunnel connectivity, validate HAMi webhook responses, and confirm that tenant etcd data remains segregated before scaling the pattern to production clusters.
