Developers can now reach Bedrock foundation models via a single OpenAI‑compatible API that is backed by a self‑service gateway built on LiteLLM and AWS Fargate. The change removes the need for each team to integrate directly with Bedrock, while giving ops, finance, and security groups visibility into usage, costs, and performance.
LLM gateway architecture
The production topology consists of five core layers:
- LiteLLM on Fargate – A containerized proxy that translates OpenAI calls into Bedrock inference profile requests. The task runs with target‑tracking auto‑scaling based on CPU utilization.
- Self‑service portal – A lightweight web UI bundled with LiteLLM that can be tied to corporate SSO. Engineers request API keys and administrators set spend limits.
- State stores – Persistent metadata (API key details, spend tracking) lives in an Amazon RDS PostgreSQL instance, while short‑lived response caching and cross‑task rate‑limiting use Amazon ElastiCache for Valkey.
- Bedrock inference profiles – Each model (e.g., Claude Sonnet, Claude Haiku) is represented by an Application Inference Profile managed via the AWS CDK. Profiles enforce per‑model quotas and isolate configuration from the gateway.
- Observability stack – Amazon CloudWatch records requests‑per‑minute and tokens‑per‑minute metrics, and Langfuse provides prompt‑level telemetry for debugging.
Implementation outline
Practitioners can reproduce the pattern with the following high‑level steps:
- Provision an Amazon RDS PostgreSQL database and an ElastiCache Valkey cluster for state and cache.
- Define Bedrock inference profiles in CDK, assigning each model its own IAM task‑role permissions.
- Deploy a Fargate task running the LiteLLM AI Gateway image, configuring the task role to assume the Bedrock profile role.
- Expose the task behind an Application Load Balancer, fronted by Route 53 and CloudFront for DNS and edge caching.
- Enable CloudWatch alarms on RPM/TPM thresholds and configure Langfuse to ingest request logs.
Operational and security implications
From an ops perspective, the gateway centralizes latency and error monitoring, allowing SREs to set alerts on token‑rate spikes or downstream Bedrock failures. Finance teams can query the RDS spend table to attribute inference costs to individual API keys, supporting charge‑back models.
Security considerations include:
- API keys are stored as virtual keys in the RDS table; rotation and revocation are performed through the portal.
- Secrets for Bedrock access (e.g., IAM role ARNs) are kept in AWS Secrets Manager and injected into the Fargate task at launch.
- IAM task roles are scoped to the specific inference profile, limiting the gateway’s permissions to the intended model set.
- While the article does not cover Bedrock Guardrails, teams should evaluate adding content‑filtering or grounding controls before production use.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopting this pattern gives engineering teams a reusable, centrally managed entry point to Bedrock models, reducing per‑project integration effort. It also creates a clear audit trail for usage and cost, which aligns with DevOps, SRE, and security governance requirements. Teams should start with a pilot limited to a few users, validate the spend‑tracking logic, and then expand the portal’s SSO integration and guardrail policies as confidence grows.