Running llama.cpp Decision Models on SageMaker: Architecture and Ops Guide

AWS has added a second deployment path for llama.cpp on SageMaker: the existing DLC continues to expose an OpenAI‑compatible chat endpoint, while a new child image built on top of the same container replaces the server with a SemIf adapter that returns probability scores for a predefined set of options. Practitioners can choose the path that matches their latency, cost, and integration requirements without leaving the SageMaker managed service.

Solution Overview

Both paths start from a single packaged model stored as model.tar.gz in Amazon S3. The standard path references the published AWS DLC for llama.cpp directly; the container scans /opt/ml/model for the first .gguf file, maps POST /invocations to llama-server at /v1/chat/completions, and forwards environment variables prefixed with SM_LLAMA_CPP_ to the underlying llama.cpp binary.

The structured‑decision path uses AWS CodeBuild to create a child image that retains the DLC’s compiled CPU or CUDA libraries but replaces the entry point with a SageMaker‑compatible SemIf adapter. The resulting image is pushed to Amazon ECR and can be deployed to a real‑time endpoint on either CPU or GPU, depending on the workload’s latency profile.

Input and Output Schemas

The chat endpoint follows the OpenAI Chat Completions schema. A typical request includes a messages array, generation controls such as max_tokens, and an optional stream flag:

{
  "messages": [
    {"role": "user", "content": "A customer cannot receive a password reset email. Reply with the best support queue."}
  ],
  "max_tokens": 64,
  "stream": false
}

The response contains a choices array; generated text is read from choices[0].message.content.

SemIf expects a decision‑oriented payload that enumerates allowed options. The request format includes a caller‑defined identifier, unstructured state, a question, and an ordered list of option objects:

{
  "decision": {
    "id": "support-route",
    "state": "The customer cannot receive a password reset email.",
    "question": "Which queue should handle this request?",
    "options": [
      {"id": "account_access", "description": "Account access support."},
      {"id": "billing", "description": "Billing support."}
    ]
  }
}

The response returns parallel arrays option_ids, probabilities, and raw option_logits, together with timing and model metadata. Stable option identifiers let downstream services apply thresholds, fallbacks, or policy checks without parsing free‑form text.

Building the SemIf Child Image

To create the decision‑model endpoint, start a CodeBuild project that uses the base DLC as its build stage. The Dockerfile copies the SemIf adapter source, replaces the default entry point, and preserves the existing libc and CUDA layers. After the build completes, the image is stored in Amazon ECR and referenced in the SageMaker model definition.

Key operational considerations include:

• Ensuring the child image version matches the base DLC’s CPU or GPU variant to avoid library mismatches.
• Validating that the model.tar.gz contains exactly one .gguf file, as the container does not perform additional model discovery.
• Monitoring endpoint latency, since the SemIf path bypasses the higher‑level chat server and may have different performance characteristics.

Related CloudNinjas coverage: AWS.

What This Means For Practitioners

Practitioners can now serve a single quantized llama.cpp model in two distinct modes without managing separate infrastructure:

• Use the default DLC for free‑form text generation when an application needs conversational output.
• Deploy the SemIf child image when the business logic requires structured probability scores over a known option set.
• Choose CPU or GPU instances per endpoint to balance cost against latency, leveraging the same underlying container image.
• Incorporate the decision endpoint into existing SageMaker monitoring pipelines, but treat the two APIs as separate contracts—one returns generated text, the other returns deterministic scores.

Future evaluation should focus on the trade‑offs between model quantization level, endpoint instance type, and the latency requirements of each API. Teams should also verify that any downstream policy enforcement relies on the stable option identifiers rather than on parsing generated prose, to keep the decision path robust against model output variations.