Octave shifted from building single‑purpose AI add‑ons to a reusable, security‑first platform that runs on Amazon Bedrock. The new foundation isolates every tenant’s data, enforces per‑user execution permissions, and can embed billions of documents in a single workday, which directly impacts AI engineers, platform teams, and security operators who must balance compliance with rapid feature delivery.
Four Design Tensions That Shaped the Solution
During the initial workshops the team identified four non‑negotiable constraints:
- Tenant‑level data isolation. Complaint records, CAPAs, SOPs, and audit findings must never be visible to another customer. All vector stores, retrieval paths, and generation contexts are scoped to a single tenant.
- Per‑request agent isolation. The permissions governing an agent’s tools, data stores, and model endpoints are derived from the calling user rather than a shared service role, and are only active for the duration of the request.
- Untrusted content ingestion. Documents arrive as PDFs, scanned images, or email attachments that may contain hidden prompts. The architecture must mitigate LLM01 (prompt injection), LLM02 (sensitive data leakage), LLM05 (output handling), and LLM06 (excessive agency) as defined in the 2025 OWASP Top 10 for LLM applications.
- Scale of embedding. With over 2 billion documents across tenants, the pipeline must finish onboarding the largest customer within a single working day without sacrificing quality.
Three‑Layer Bedrock‑Based Architecture
Amazon Bedrock provides a single API surface for model access, regional residency, and IAM‑driven governance, which satisfies the isolation and control requirements. The resulting architecture consists of:
- Data preparation layer. Documents are normalized, OCR‑processed where needed, and passed to the Cohere Embed v4 model via Bedrock. The embedding step runs in parallel across shards to meet the one‑day onboarding goal.
- Vector store layer. Tenant‑specific vector indexes are created in a managed store that enforces IAM policies per tenant. Retrieval queries are automatically scoped to the caller’s tenant identifier.
- Agent execution layer. When a user invokes a feature (e.g., complaint triage), a short‑lived execution context is created with permissions limited to that user’s data stores and the Bedrock model endpoint. No static credentials are shared across requests.
Operational Highlights
Embedding 2 billion records required a batch pipeline that leverages Bedrock’s managed scaling; the team reports a 50 % reduction in time‑to‑deploy for subsequent AI use cases. Because each tenant’s vector index lives in isolation, the risk of cross‑tenant data leakage is mitigated at the storage level, not just at the application layer.
Security considerations include:
- Relying on IAM for both data‑store and model‑endpoint access, ensuring that only authorized users can invoke an agent.
- Applying Guardrails on Bedrock to enforce content policies during generation, reducing the chance of inadvertent disclosure.
- Validating and sanitizing all inbound documents before they reach the embedding model to limit prompt‑injection vectors.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Teams building SaaS AI features should consider a foundational approach that isolates data and execution per tenant from day one. Leveraging a managed service like Bedrock can simplify compliance with regional residency and IAM policies, while Guardrails provide a baseline defense against known LLM‑specific threats. When scaling embeddings, design a parallelized ingestion pipeline that can meet business‑critical onboarding windows without compromising quality. Finally, treat per‑request permission scoping as a core requirement rather than an after‑thought, especially in regulated domains where a single data leak can have severe consequences.
