Applications often need to score choices rather than generate free-form text. Deploying decision models with llama.cpp on Amazon SageMaker AI provides a managed path for serving quantized models on CPU or GPU infrastructure. This approach uses the AWS Deep Learning Container (DLC) for llama.cpp and extends it with SemIf, a decision model adapter that returns probabilities over declared options without generating answer tokens.
Solution Overview
The solution offers two deployment paths from the same packaged model. The standard text-generation path uses the AWS DLC for llama.cpp, serving OpenAI-compatible chat completions. The structured-decision path builds a child image, replacing the default server with a SemIf adapter that returns option probabilities. Both paths can run on CPU or GPU, depending on latency requirements.
For the standard path, package the model as model.tar.gz in Amazon S3 and reference the published DLC image. The container serves the first .gguf file under /opt/ml/model, mapping POST /invocations to llama-server at /v1/chat/completions. Environment variables configure llama.cpp arguments.
The structured-decision path uses AWS CodeBuild to create a child image stored in Amazon ECR. This image retains the DLC's CPU or CUDA libraries but replaces the entry point with a SageMaker-compatible SemIf adapter.
Input and Output Schemas
The AWS DLC for llama.cpp accepts the OpenAI Chat Completions schema, with a messages array containing role and content, plus controls like max_tokens and stream. The response follows the OpenAI choices array format.
SemIf uses a decision schema with a caller-defined ID, unstructured state, question, and ordered options list. The response returns option IDs, probabilities, and raw logits in the same order, along with prompt and timing metadata.
Choosing Decision Workflows
Use this pattern when your application defines choices at request time and needs a model to interpret context. Examples include routing cases, extracting values, assigning priorities, or selecting actions. Keep business rules outside the model and validate probabilities before operational use.
Use the standard text-generation path for OpenAI-compatible chat completions. Build a child image only when needing behavior not provided by the standard interface, such as SemIf's direct logits access.
Performance Tuning
Model size, quantization, prompt length, runtime, and CPU threads affect performance. Benchmarks show GPU configurations are significantly faster than CPU for the same model. A 4B Qwen model on a g6e.xlarge instance achieved 0.352-second p50 latency compared to 1.715 seconds on a c6i.8xlarge CPU instance.
Smaller models reduce latency but sacrifice accuracy. The 4B model maintains quality while balancing speed. Use CPU for non-interactive workloads and GPU for low-latency requirements. Keep prompts concise and group related decisions into single requests to reuse prompt prefixes.
Prerequisites
You need an AWS account, an S3 bucket in your Region, Python 3.10+, Git, the AWS CLI, and a CodeBuild service role for the SemIf extension.