TensorFlow’s TorchServe inference server has been officially marked as no longer maintained, meaning no further bug fixes, feature updates, or security patches will be released. AWS now offers the Ray Serve Deep Learning Container (DLC) as a drop‑in replacement for inference workloads, providing a pre‑built, patched image that bundles PyTorch, the Ray Serve stack, and the GPU runtime.
Why the shift matters
When a serving platform stops receiving updates, teams must manually track compatibility across the entire GPU stack—CUDA, PyTorch, and any auxiliary libraries. That responsibility adds friction to model delivery and creates a hidden security liability. The Ray Serve DLC removes that friction by delivering a single, version‑aligned image that is rebuilt with security patches at each release. Engineers can therefore focus on model code rather than low‑level dependency management.
Ray Serve DLC architecture
The GPU variant of the Ray Serve DLC is built on the NVIDIA Amazon Linux 2023 base image, which already contains the CUDA runtime libraries. On top of that base, the image adds:
- PyTorch, matching the CUDA version in the base layer
- Ray Serve, exposing model endpoints via FastAPI and Uvicorn
- FFmpeg compiled with NVIDIA hardware acceleration for video preprocessing
All components are validated together, ensuring no version drift between the GPU stack, the framework, and the serving layer. Separate images exist for Amazon EKS, Amazon EC2, and Amazon SageMaker, each with an entrypoint tuned to the target environment, but they share the same underlying stack.
Implementing a model with the Ray Serve DLC
Deploying a model follows the same pattern as any Ray Serve application: define a Python class, decorate it with @serve.deployment, and bind it to the HTTP server. The example in the source uses the Qwen3‑VL‑2B vision‑language model, loading it onto the GPU and exposing a single endpoint that accepts an image URL and a text prompt. The serving code is injected into the container via a Kubernetes ConfigMap, allowing updates without rebuilding the image.
from ray import serve
from transformers import AutoModelForImageTextToText, AutoProcessor
import torch
@serve.deployment(ray_actor_options={"num_gpus": 1})
class QwenVLService:
def __init__(self):
model_name = "Qwen/Qwen3-VL-2B-Instruct"
self.processor = AutoProcessor.from_pretrained(model_name)
self.model = AutoModelForImageTextToText.from_pretrained(
model_name, torch_dtype=torch.float16
).to("cuda")
async def __call__(self, request):
body = await request.json()
image_url = body.get("image_url")
prompt = body.get("prompt")
# Model inference logic omitted for brevity
Operational and security implications
Because the DLC image is rebuilt with security patches, the surface area for known vulnerabilities is reduced compared with a self‑maintained TorchServe stack. However, practitioners still need to manage the container lifecycle, monitor for new DLC releases, and apply updates promptly. Using the DLC also simplifies GPU quota planning: the example targets a g5.xlarge instance, and the same image can be run on a single‑GPU node in EKS.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Switching to the Ray Serve DLC eliminates the need to manually align CUDA, PyTorch, and serving‑layer versions, and it provides a regularly patched runtime. Teams should evaluate existing TorchServe deployments, plan a migration to the Ray Serve DLC on their preferred AWS compute target, and integrate ConfigMap‑driven code updates to keep the container immutable. Ongoing vigilance is still required to track DLC version releases and apply them in a timely fashion.
