OpenAI’s recent announcement that ChatGPT now serves roughly 1.2 billion weekly active users exposed a new class of identity load: each user‑initiated AI request can spawn many downstream calls, each requiring its own token introspection. To keep latency acceptable, OpenAI replaced traditional monolithic identity SaaS with a self‑hosted, composable stack built around Ory’s Kratos, Hydra, and Keto components.
Agentic Workloads Multiply Token Checks
When an AI agent processes a user request, the platform often launches dozens of sub‑tasks—database queries, third‑party tool calls, and additional model invocations. Every step must verify the session and inspect the access token, turning a single front‑end request into a burst of token validations. The source notes that this ratio grows non‑linearly, meaning that token introspection becomes a primary performance factor.
Limitations of Centralized IdP Designs
Legacy identity providers typically perform token validation through centralized RPCs or database lookups. At billions of requests per minute, those round‑trips add network hops and create contention on the underlying datastore. The result is a latency bottleneck that can dominate the request path, especially when each downstream call repeats the same validation.
Composable Edge‑Native Stack with Ory
OpenAI’s solution separates identity concerns into distinct microservices: Kratos handles credential storage and session lifecycle; Hydra issues signed OAuth2/OIDC tokens; Keto provides fine‑grained, relationship‑based access checks. Validation is performed at the edge using cached JSON Web Key Sets, allowing stateless JWT verification without a remote call. Data residency is achieved by running the stack inside the organization’s VPC or multi‑cloud Kubernetes clusters, avoiding vendor‑managed user stores. Persistence relies on distributed SQL engines (e.g., CockroachDB, Spanner, Postgres) that replicate writes across regions, ensuring low‑latency reads for edge nodes. Cost scales with compute and memory rather than per‑MAU or per‑API request fees, and the Go‑native services expose clean APIs that can be extended without vendor‑imposed sandboxes.
Operational and Security Shifts
Running identity infrastructure in‑house introduces new operational responsibilities: managing distributed databases, monitoring edge cache freshness, and handling certificate rotation for JWKS. Security practice also evolves. Instead of enforcing strict policies from day one—an approach that can break autonomous agents—OpenAI adopts an “observe‑first” mode. Agents are fingerprinted, and their credential usage is logged before policies are hardened, providing visibility into real‑world behavior. This observability step is essential when agents act on behalf of users across many services.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
- Audit your token validation path: identify any centralized introspection calls that could become bottlenecks at scale.
- Consider edge‑native JWT verification with JWKS caching to eliminate per‑request network hops.
- Evaluate whether a composable, self‑hosted IAM stack (e.g., Ory) aligns with your data‑residency and cost‑model requirements.
- Plan for geo‑distributed persistence if you need sub‑millisecond token checks from multiple regions.
- Adopt an observability‑first policy rollout for machine‑to‑machine agents, logging credential usage before enforcing restrictive rules.
