In early 2026 a multi‑cluster GKE deployment in the europe‑west region stopped being able to reach its own Kubernetes API server. Pods across namespaces logged timeout errors when dialing the cluster’s internal VIP (1.2.3.4:443) that is exposed through Private Service Connect. The outage persisted despite removing custom operators, external‑secrets workloads, and security agents, indicating a deeper stack problem that ultimately required a Linux kernel patch.
Kubernetes API connectivity broke across layers
The affected environment used Private Service Connect to keep the control plane private and ran system‑level workloads such as Twistlock, External Secrets Operator, and Cilium. Initial logs showed intermittent i/o timeout errors from many pods, first in a user‑acceptance testing cluster and then in a second cluster in the same region. Because the symptom appeared in both customer workloads and core system pods, the investigation quickly moved beyond application code.
Root causes spanned three stack layers
Cross‑team debugging revealed three distinct defects that aligned in time:
- GKE networking layer: a regression in the control‑plane networking component that interfered with the VIP routing used by Private Service Connect.
- Private Service Connect routing: a bug that mishandled connection tracking for the API server’s private endpoint, causing packets to be dropped under certain traffic patterns.
- Linux kernel: an issue in the kernel’s conntrack handling that manifested as table exhaustion, ultimately requiring a kernel source patch rather than a configuration tweak.
Operational takeaways
While the specific bugs are unique to this incident, the pattern highlights several considerations for engineers managing GKE clusters:
- Monitor API‑server connectivity as a first‑order health indicator; timeouts can signal issues beyond pod‑level problems.
- Track conntrack table usage on nodes, especially when using Private Service Connect, to detect early signs of exhaustion.
- Include Private Service Connect health checks in your observability stack; failures may surface as seemingly unrelated pod errors.
- Maintain a rapid escalation path that spans platform, networking, and kernel teams when symptoms cross multiple layers.
- Stay current with kernel patches provided by the cloud provider, as critical fixes may arrive outside of typical GKE version upgrades.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
Practitioners should treat API‑server reachability as a cross‑layer reliability metric. Implement regular probes against the internal VIP, watch kernel conntrack metrics, and verify Private Service Connect endpoints with health checks. When timeouts appear, broaden the investigation beyond application code to include networking and host‑level diagnostics. Keeping the underlying kernel up to date and having clear escalation channels can reduce mean‑time‑to‑resolution for similar multi‑layer failures.
