We now have a reference deployment that places a customer‑operated LiteLLM gateway on Amazon ECS in front of an OpenAI Codex model hosted on Amazon Bedrock. The change introduces a controllable API layer that mediates every model turn, allowing teams to enforce budgets, rate limits, and model‑alias policies while still letting Codex run its local tool loop.
Solution overview
The architecture inserts LiteLLM between the developer workstation and Bedrock. A request from Codex hits the /v1/responses endpoint, passes through an Application Load Balancer (ALB) and optional AWS WAF, and is processed by a Fargate task running LiteLLM. LiteLLM authenticates the caller, checks the configured policy, and uses its ECS task role to invoke the approved Bedrock model. The response – plain text or a function call – is returned to Codex, which may execute a tool locally and continue the loop.
Supporting services include:
- Amazon RDS for PostgreSQL storing LiteLLM state, usage, and budget data.
- AWS Secrets Manager and AWS KMS for gateway and scoped‑key storage.
- Amazon CloudWatch logs, Container Insights, and alarms for observability.
- Amazon ECR holding an immutable gateway container image.
- Optional WAF rules for source‑IP rate limiting and managed protections.
Deploying the LiteLLM gateway on ECS
The quick‑start guide in the referenced repository walks through creating an ALB, attaching a WAF policy if desired, provisioning the Fargate service, and wiring the task role to call Bedrock. The gateway image is pulled from ECR, and the RDS instance is provisioned for persistent state. Secrets for Bedrock credentials are stored in Secrets Manager and encrypted with KMS. Once the service is healthy, Codex is configured to point at the gateway’s /v1/responses endpoint.
Operational and security implications
Running the gateway shifts several responsibilities to the customer team:
- Availability and scaling of the Fargate tasks and ALB become part of the service‑level objective.
- Database lifecycle – backups, patching, and retention – must be managed in RDS.
- Version upgrades of LiteLLM and the container image require a controlled rollout process.
- Incident response now includes the gateway path, not just Bedrock failures.
- Capacity planning must account for request‑per‑minute or token‑per‑minute limits enforced by LiteLLM.
Security controls are explicit: network‑level filtering is handled by the ALB and optional WAF, while secret handling relies on Secrets Manager and KMS. LiteLLM’s authentication layer enforces user‑ or team‑scoped keys, preventing unrestricted Bedrock access. However, the gateway does not replace Codex’s local sandbox approvals; it only governs the model turn.
When a managed gateway may be preferable
If an organization’s compliance posture can be satisfied by native IAM policies, CloudTrail logging, and direct Bedrock access, the added operational overhead of a self‑hosted gateway may not be justified. In such cases, using AWS IAM Identity Center or a managed solution like Portkey can provide similar controls with less operational burden.
Related CloudNinjas coverage: AWS.
What this means for practitioners
Adopting the LiteLLM gateway gives you fine‑grained control over model usage at the cost of owning the gateway stack. Evaluate whether your team can support the required operational tasks – monitoring, database management, secret rotation, and capacity planning – and whether the policy enforcement benefits (budget caps, scoped keys, model alias gating) outweigh that cost. If you need consistent controls across multiple developers or teams, the gateway pattern is a practical way to centralize governance while keeping Codex’s local execution model intact.
