A central Prometheus that collected GPU utilization for every namespace was augmented with a Kubernetes‑native proxy layer that authenticates each request, forces a namespace selector on every query, and can remote‑write a curated metric set into a per‑tenant Prometheus. This change gives each team a self‑service view of its own GPU usage while keeping other teams’ data hidden and protecting the shared store from noisy‑neighbor queries.
Why Direct Prometheus Access Fails
Giving every team read access to the monolithic Prometheus violates two core constraints. First, the Prometheus HTTP API is not aware of Kubernetes namespaces, so a single query can retrieve any series, exposing confidential request rates or capacity plans. Second, the central store already handles thousands of series; ad‑hoc range queries from many engineers can saturate its CPU and memory, causing latency spikes for all consumers.
Design of the Tenant‑Aware Proxy
The solution introduces three logical steps: identify, isolate, and deliver.
- Identify:
kube-rbac-proxyauthenticates the caller using the Kubernetes identity model and determines the tenant’s namespace. - Isolate:
prom-label-proxyrewrites every incoming PromQL expression to inject a{namespace=""}matcher, ensuring the backend Prometheus only sees data for that tenant. - Deliver: The proxy can periodically remote‑write the filtered series into a small, tenant‑owned Prometheus instance, allowing dashboards and alerts to run locally without hitting the central store.
The tenant’s policy is expressed via a custom resource called MetricAccess, where the team lists the metrics it needs. The platform operator creates the proxy deployment and the RBAC rules; the tenant only manages its MetricAccess object.
Implementation Details
Incoming traffic hits an Nginx load balancer that distributes requests across multiple proxy replicas. Each replica uses the Kubernetes API to discover the central Prometheus endpoints and any tenant‑specific Prometheus pods. For read requests, the flow is:
client → Nginx → kube‑rbac‑proxy (auth) → prom‑label‑proxy (namespace injection) → central Prometheus → filtered result
For write‑back, the proxy performs a remote‑write to each replica of the tenant’s Prometheus, using DNS‑based service discovery to locate them. The proxy runs as a non‑root user (UID 65534), with a read‑only root filesystem and all Linux capabilities dropped, following a least‑privilege model.
Operational and Security Implications
Isolation is enforced at two points: query rewriting prevents cross‑tenant reads, and optional remote‑write limits the amount of data each tenant stores, reducing storage costs. Because authentication and namespace enforcement happen before the query reaches Prometheus, the central store no longer needs to be exposed to end users, preserving the existing security posture. The architecture also mitigates noisy‑neighbor effects: tenant dashboards query their own lightweight Prometheus, leaving the central instance free for platform‑wide aggregation.
From an operations perspective, the proxy adds a new component to monitor for health and latency, but it reuses existing Kubernetes service discovery and Prometheus remote‑write mechanisms, keeping the operational footprint small. Teams gain immediate visibility into idle GPUs, enabling cost‑saving actions without waiting for platform engineers.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Adopt a tenant‑aware Prometheus proxy when you need per‑team metric visibility on shared hardware such as GPUs. Ensure you configure kube-rbac-proxy for authentication, prom-label-proxy for namespace injection, and define MetricAccess resources to let teams declare required metrics. Monitor the proxy’s health and remote‑write latency, and verify that each tenant’s Prometheus instance is sized appropriately for its filtered data set. This pattern delivers secure, self‑service observability while protecting the central metrics store from overload and preserving cost accountability.