Recent discussions at KubeCon + CloudNativeCon NA 2026 highlighted a concrete change: AI workloads are generating far more internal traffic than traditional services, with one human request translating into about twenty distinct agent invocations. At the same time, the community is delivering new Kubernetes‑native runtimes—such as the sandbox project kagent and the Agent Substrate runtime—to treat agents as first‑class resources. Practitioners need to understand how these patterns affect latency, scaling, and compliance responsibilities.
Request churn and sub‑200 ms latency expectations
AI engineers have long accepted that large language model (LLM) inference can be slow, but the emerging architecture treats the LLM as a backend service called by many lightweight agents. The CNCF interview cited in the event notes that agents expect responses in under 200 ms, yet a freshly scheduled pod typically takes about a second to become ready. This mismatch creates a pressure point for platform teams: either keep a warm pool of pods or redesign the control loop to avoid cold starts.
New runtimes for large‑scale agent deployments
Two projects are being positioned to close the latency gap. llm-d focuses on distributed inference, spreading model execution across multiple nodes. Agent Substrate provides a runtime that can orchestrate thousands of agents, handling their lifecycle and inter‑agent communication (A2A). Both are still early in the CNCF ecosystem, but they signal a move toward treating agents as a workload class rather than an ad‑hoc add‑on.
kagent: agents as Kubernetes custom resources
The sandbox project kagent demonstrates a practical way to declare an agent in YAML, letting a controller materialize it as a pod. A minimal example looks like:
apiVersion: ai.example.com/v1
kind: Agent
metadata:
name: builder-agent
spec:
systemPrompt: "You build other agents"
model: "ollama/phi"
tools:
- name: "kubectl"
- name: "curl"
When the resource is applied, the controller validates the tool allowlist, requires human approval, and creates a pod that speaks the A2A API. The project supports both local Ollama models and hosted providers such as OpenAI, Anthropic, Azure OpenAI, and Gemini. Smaller models may struggle to emit valid manifests, a limitation noted by the authors.
Compliance ownership and operational hand‑off
A separate CNCF blog post raised the question of who owns Kubernetes compliance under regulations like NIS2 and DORA. The argument is that security teams should write policy, but platform teams must implement and maintain the Helm charts and runtime configurations. The proposed solution is a “traceability chain” that maps legal requirements to sprint items and recurring operational tasks, allowing risk and security groups to hand off concrete artifacts to engineering owners.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
- Expect a higher internal request rate per user action; size your pod pools or adopt warm‑start strategies accordingly.
- Evaluate whether
llm-dorAgent Substratefit your latency budget before committing to a custom solution. - Consider adopting
kagentif you want declarative agent management, but test with production‑scale models to avoid manifest generation failures. - Establish clear compliance hand‑off processes: define policy in security, map it to platform artifacts, and track evidence through the traceability chain.
- Monitor the emerging CNCF sandbox projects for stability and community support before using them in critical pipelines.
