Orca Security moved from a single‑prompt model that consumed millions of tokens per request to a modular Discovery AI pipeline built on Amazon Bedrock. The change reduces token usage, execution time, and cost while improving the reliability of generated Sonar queries, directly impacting anyone who builds or operates cloud security tooling.
Why Engineers Should Care
For cloud, platform, DevOps, and security engineers, query latency and cost are operational constraints. The original design required roughly 2 million input tokens per workload and 60 seconds of latency, making large‑scale adoption impractical. After the redesign, Orca reports a 90 % reduction in query‑related cost, a 93 % drop in input tokens, and a 75 % cut in response time, enabling thousands of queries per day with sub‑15‑second turnaround. Faster feedback loops improve incident response, compliance checks, and automated remediation pipelines.
Discovery AI Architecture at a Glance
The new system follows a nine‑stage flow, each matched to the most suitable Bedrock model:
- Request Entry and Context Setup – Combines the user’s natural language with tenant metadata (cloud providers, organization ID) to give the model situational awareness.
- Quick Deterministic Matcher – Uses regex and rule‑based logic to spot high‑confidence identifiers (IP, CVE, ARN, hash, etc.). When a match is found, a pre‑built Discovery filter is returned without invoking an LLM.
- LLM Guardrail Screening – Validates that the request is safe, relevant, and supported, rejecting prompt‑injection or off‑topic inputs before further processing.
- Retrieval Augmented Generation (RAG) Step – Retrieves relevant portions of the 900+ Sonar schema from a vector store, limiting the prompt size and focusing the model on needed definitions.
- Schema‑Aware Prompt Construction – Crafts a concise prompt that includes only the retrieved schema fragments and the user query.
- LLM Generation – Generates a candidate Sonar JSON filter using the selected foundation model.
- Syntax Validation – Checks the JSON against the Sonar schema to catch structural errors early.
- Semantic Verification – Runs a second LLM pass to ensure the filter matches the original intent, flagging mismatches for refinement.
- Final Dispatch – Sends the validated filter to Orca’s Sonar engine for execution.
Amazon Bedrock’s unified API lets Orca swap foundation models per stage without rewriting integration code, and the service inherits AWS IAM, VPC endpoints, and CloudTrail for auditability.
Operational and Security Implications
Adopting a multi‑stage pipeline introduces new operational considerations:
- Model Management – Teams must track which model version powers each stage and plan upgrades without breaking downstream validation.
- Vector Store Governance – The RAG component relies on a searchable embedding store; access controls and data residency policies must be enforced.
- Guardrail Effectiveness – The LLM screening step is the first line of defense against malicious prompts; its rules need regular review as threat patterns evolve.
- Observability – Bedrock integrates with CloudTrail, but additional logging around each pipeline stage (e.g., matcher hits, validation failures) is advisable for debugging and compliance.
Because the pipeline returns a pre‑built filter for deterministic matches, the system can bypass LLM inference entirely for common identifiers, further reducing exposure to model‑related risks.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Teams building cloud‑security query interfaces should consider a hybrid RAG architecture rather than a monolithic prompt. By separating deterministic pattern matching, guardrails, and schema‑aware generation, you can achieve measurable cost and latency improvements while retaining flexibility to evolve the underlying models. Evaluate your own query schema for opportunities to externalize large definitions into a retrieval layer, and implement early‑stage validation to keep LLM usage focused and auditable.
