Hardening AI Inference Workloads with GKE ClusterNetworkPolicy

What changed? GKE introduced ClusterNetworkPolicy (CNP), a policy type that operates across the entire cluster rather than being confined to a single namespace. Why does it matter? The broader scope enables AI engineers, cloud/platform engineers, SREs, and security practitioners to apply mandatory guardrails, baseline defaults, and fine‑grained traffic controls to high‑value workloads such as large language model (LLM) inference.

ClusterNetworkPolicy Overview

CNP follows a three‑tier evaluation hierarchy with explicit precedence:

1. Admin tier (top precedence): Enforced by cluster administrators; rules cannot be overridden. Example from the source is blocking access to the metadata service.
2. Network policy tier (middle precedence): Namespace‑scoped policies managed by application teams for typical ingress/egress control.
3. Baseline tier (bottom precedence): Fallback defaults set by administrators, such as a default‑deny posture across namespaces.

This hierarchy ensures that mandatory security rules always win, while teams retain flexibility for service‑specific policies.

Applying CNP to an LLM Inference Pattern

The referenced codelab demonstrates a concrete pattern: an LLM served in its own namespace must only accept traffic from authorized clients, and the rest of the cluster must be prevented from reaching the metadata API except for DNS (port 53). The architecture combines several GCP services:

• Virtual Private Cloud (VPC): Provides private networking, pod and service subnets, and outbound egress via Cloud NAT.
• TPU v5e: Accelerates model inference for the Gemma 4 model.
• GKE with Dataplane V2 (eBPF): Enforces the multi‑tier CNP rules at the kernel level.
• Gemma 4 model served by vLLM: Represents a sensitive AI workload that benefits from strict network isolation.

By defining a CNP that targets the LLM namespace, the experiment restricts inbound traffic to a whitelist of client pods and blocks all other inter‑namespace communication. A separate baseline rule denies metadata API access cluster‑wide, with an explicit allow for DNS traffic on port 53.

Operational and Security Implications

Adopting CNP changes both the operational workflow and the security posture:

• Policy lifecycle: Administrators must now maintain three distinct policy sets. Changes to the admin tier propagate instantly across all namespaces, reducing the risk of accidental policy drift.
• Observability: Because enforcement runs in eBPF, traditional network‑policy logs may need augmentation with eBPF tracing tools to verify rule hits.
• Testing: The hierarchy requires validation that lower‑tier policies do not unintentionally override higher‑tier mandates. A staged rollout—first baseline defaults, then namespace policies, finally admin mandates—helps catch regressions.
• Compliance: Mandatory admin rules (e.g., metadata blocking) provide a clear, auditable control that aligns with security standards that require “cannot be bypassed” network restrictions.

From an SRE perspective, the default‑deny baseline can simplify troubleshooting by narrowing the set of allowed paths, but it also demands that any new service explicitly declare its required egress, increasing the upfront configuration effort.

Related CloudNinjas coverage: Google Cloud.

What This Means For Practitioners

Practitioners should take the following steps:

1. Audit existing namespace‑scoped NetworkPolicy objects and identify gaps that could be covered by an admin‑tier CNP.
2. Define a baseline ClusterNetworkPolicy that enforces a default‑deny posture for all namespaces.
3. Implement mandatory admin rules for high‑risk resources, such as the metadata service, ensuring they are expressed as top‑precedence CNP objects.
4. Validate the three‑tier hierarchy in a staging cluster, using eBPF‑compatible observability tools to confirm rule evaluation order.
5. Monitor for unintended side effects, especially on services that rely on DNS, and adjust the baseline allow for port 53 as needed.

By integrating CNP into the network‑security stack, teams can scale governance across clusters while preserving the flexibility required for AI inference workloads.