AWS publishes guide to deploying decision models on SageMaker AI with llama.cpp, using SemIf to return per-option probabilities

Verified (fact)

AWS published a technical guide to deploying decision models on Amazon SageMaker AI using the AWS Deep Learning Container for llama.cpp. The setup provides OpenAI-compatible chat completions. Extending the runtime with SemIf lets it return probabilities for each of a set of predefined options, and it runs on both CPU and GPU. On the same day, Ollama v0.40.4-rc0 also updated llama.cpp to support decision models.

Why it matters (interpretation)

As decision models that output option probabilities instead of text are increasingly used for routing and classification, major clouds and local runtimes are starting to support serving them.

Sources

• Official Deploy decision models with llama.cpp on Amazon SageMaker AI AWS
• Official v0.40.4 Ollama

More about AWS

• 2026-10-13 P2 Infrastructure Generally available Efficiency & infraMLOps & LLMOpsAI platforms & solutions AWS
  AWS shows how to split prefill and decode across GPU nodes with its multi-node Ray LLM Deep Learning Container

  AWS published a method for deploying a single large language model across two GPU nodes on Amazon EKS using its multi-node Ray LLM Deep...

• 2026-10-12 P1 Other Unconfirmed Generative mediaEfficiency & infra AWSDecartAnthropic
  Amazon in talks to buy AI startup Decart for $7 billion, WSJ reports

  AI Times, citing a Wall Street Journal (WSJ) report from Oct. 9 (local time), said Amazon is negotiating to acquire San Francisco AI...

• 2026-10-12 P2 Infrastructure Generally available AI platforms & solutionsLanguage models AWSOpenAI
  AWS shows how to use OpenAI GPT-6 Sol, Luna and Astra on Amazon Bedrock via global cross-Region inference from Southeast Asia and Taiwan

  In an official blog post, AWS explained how to call OpenAI's GPT-6 Sol, Luna, and Astra on Amazon Bedrock through global cross-Region...

• 2026-10-10 P2 Open source Generally available Robotics & physical AIMLOps & LLMOpsAI platforms & solutions AWSNVIDIA
  AWS open-sources 'Physical AI Toolchain on AWS' built on NVIDIA's stack, linking robot training to deployment and retraining (report)

  According to a report by AI Times (인공지능신문), AWS released 'Physical AI Toolchain on AWS' as open source on the 8th (local time). It...

• 2026-10-10 P2 Agents Generally available AgentsAI platforms & solutions AWSPostman
  AWS and Postman share architecture for running 'Agent Mode' for 40 million developers on Amazon Bedrock

  On the AWS blog, Postman and AWS shared the architecture patterns behind running Postman's 'Agent Mode' for 40 million developers. The...