Today the Ray LLM Deep Learning Container (DLC) enables a multi-node LLM inference deployment on Amazon EKS, where one GPU node handles the prefill phase and a second node handles the decode phase, all behind a single OpenAI‑compatible endpoint. Practitioners gain a pre‑built, GPU‑stack‑aligned image that removes the need to assemble CUDA, framework, and serving libraries manually, but they must also account for the split‑phase architecture and the inter‑node communication path.
Why the prefill/decode split matters
LLM inference consists of a compute‑heavy prefill that builds a key‑value cache and a latency‑sensitive decode that generates tokens one at a time. When both phases share a GPU, the bursty prefill work can stall the steady decode work. By assigning each phase to its own GPU node, you can select hardware and serving parameters that suit the distinct resource profiles, potentially increasing overall throughput for workloads such as retrieval‑augmented generation, document summarisation, and agentic applications.
Architecture delivered by the Ray LLM DLC
The container bundles several components that have been validated together: vLLM as the inference engine, Ray Serve for HTTP serving, NIXL for KV‑cache transfer, and the Elastic Fabric Adapter (EFA) stack for low‑latency interconnect. Because the image is built and patched by AWS, the CUDA runtime, framework libraries, and serving layer remain in sync, eliminating version drift. The deployment pattern consists of two EKS pod replicas—one labelled as prefill, the other as decode—connected via the EFA network so that the cache can be handed off quickly.
Operational implications
- Node provisioning: You must provision at least two GPU‑enabled worker nodes in the EKS cluster, each capable of running the Ray LLM DLC. Scaling the prefill or decode side independently is possible, but the KV‑cache hand‑off requires a matching pair of pods.
- Network requirements: The KV‑cache transfer relies on the EFA stack; without EFA the latency of cache movement could negate the performance benefit of disaggregation.
- Image lifecycle: Security patches are baked into each DLC release. Keeping the deployment up‑to‑date means pulling the latest container tag and redeploying the pods, rather than patching individual libraries.
- Observability: Ray Serve exposes metrics for request latency and throughput. Monitoring both prefill and decode pods separately helps identify bottlenecks introduced by uneven scaling or network contention.
Security considerations
Using the AWS‑provided DLC reduces the attack surface associated with custom‑built images because the stack is assembled, tested, and patched by AWS. However, the standard container security practices still apply: scan the image for known vulnerabilities, enforce least‑privilege IAM roles for the EKS service account, and ensure that the EFA network is confined to the VPC subnet used for the inference workload. The separation of prefill and decode does not introduce additional authentication boundaries; the OpenAI‑compatible endpoint remains the single ingress point.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopting the Ray LLM DLC lets you replace a hand‑crafted, single‑GPU inference stack with a tested multi‑node deployment that isolates prefill and decode workloads. Evaluate whether your workload benefits from phase disaggregation, verify EFA availability in your VPC, and plan for independent scaling of the two pods. Keep the container image current to inherit AWS security patches, and instrument both Ray Serve metrics and EFA health checks to maintain performance and reliability.