← Analysis

Analysis · 2026-11-17 · By bombong

Will small decision models replace general-purpose LLM calls?

In the same week, Google, LangChain and Hugging Face released or began supporting decision models that 'answer over a set of options,' and a replacement case at Amazon was reported. This article lays out, with evidence, how these models differ from general-purpose LLMs, where the cost savings come from, and how far they can be used as replacements.

An illustration of a rail yard where a small signal box moves the switches to send most trains onto a short side track, while only a few head to a large, distant station.
AI-generated image · An illustration of a rail yard where a small signal box moves the switches to send most trains onto a short side track, while only a few head to a large, distant station.
Key points

What came out in the same week?

In the second week of November, models that 'answer over a set of options' appeared at several layers at once. A decision model does not write long answers. It returns only a probability and confidence for each given option (GeekNews, s6).

On November 11, Google released the Gemini Decisions API, which runs on a new model, Gemini Decision 1 (Google blog, s1). On the same day, LangChain added a decision model abstraction to langchain-core 1.8.0 source. It added a model router and auto-mode middleware to langchain 1.5.0 source. langchain-typesafe 0.1.0 implemented TypeSafeClassifier as a decision model source. Hugging Face also published a blog post on decision models on the Hub source. Before that, on November 10, a post appeared on GeekNews saying Amazon had switched its internal ticket classification model to Strands Decider E4B source. By this site's count, 'Strands Decider' was covered this week in 1 official announcement and 1 community post.

Model APIs, frameworks, a model hub and an operational case came together within a single week. Rather than a single new product launch, this reads as 'decision-making' splitting off from the components that make up an agent and becoming a standard component.

For practitioners, it is time to reconsider the default of 'have a general-purpose LLM do classification via a prompt.'

A week crowded with decision models

  1. November 10
    Amazon switching caseTicket classification moved to Strands Decider E4B
  2. November 11
    Gemini Decisions APIBased on Gemini Decision 1
  3. November 11
    New LangChain releaseDecision models and model router
A production case, a model API and a framework release followed one another within two days.

How do generative models and decision models differ?

When you classify with a general-purpose LLM, the model generates the answer one token at a time, and code parses that text to extract the label. Gemini Decision 1 works differently. According to Google, it does not generate an answer; it scores the candidate options in a single forward pass and returns probabilities and confidence source.

Google describes three question formats source: the probability of a yes/no condition, choosing among 2 to 255 options, and a zero-based ordinal rubric score. Text, images, audio, video and PDFs can be combined into a single input of up to 256K tokens. Up to 1,000 independent questions can be attached to it source. Every answer comes with a confidence between 0 and 1. Google says this makes it possible to hand ambiguous cases to a fallback path or a human reviewer source. The uses Google cites are tool selection, policy checks, model routing and context evaluation source.

LangChain also added a decision model abstraction and a model router on the same day sourcesource. Routing is a selection problem: choosing 'which model or tool to send a request to.' So this change can be seen as designed with decision models placed at the branch points of agent flows in mind. However, ModelRouterMiddleware is still marked as beta source.

When output comes as probabilities, it becomes easier to set a threshold that splits 'automatic handling' from 'human review.' It also reduces failures from parsing text.

General-purpose LLM calls vs. decision models

General-purpose LLMGenerates an answer

  • Generates the answer token by token
  • Code parses the text
  • Can explain reasoning, free-form output

Decision modelScores the options

  • Scores in a single forward pass
  • Returns per-option probabilities and confidence
  • Bundles several questions into one input

공통 Both can be used for routing, classification and policy checks

The differences when the same classification task is handled in two ways.

How much does cost change?

The most concrete figures come from the Amazon case. According to a GeekNews post, Amazon had been using Claude Sonnet 4.6 to assign one of 20 categories to a large volume of internal tickets, then switched to Strands Decider E4B source. In a separate ticket evaluation, E4B reportedly performed better than the model in production. Average call time fell 78%, tokens per ticket fell 70% and cost fell 98%, with expected savings of several million dollars a year source. All of these figures were reported as Amazon's own, and there is no independent verification.

Google put the price at $0.10 per million input tokens. Output tokens are not billed, and response time is said to be 150–250 ms source. Google claims that on a shared benchmark suite it is nearly 5 times cheaper than using Gemini 3.5 Flash-Lite source. It also said latency stays almost the same even when the number of questions in a single request rises from 1 to 100 source.

Taken together, the two sources suggest costs drop in two places: not generating output tokens, and bundling several questions into one input. The size of the savings therefore appears to depend on the shape of the task. The effect is larger for repetitive classification with fixed options. It does not carry over directly to tasks that require explaining the reasoning in text.

Before adopting '98%' as your own number, first check how many of your calls produce a fixed label as output.

Reported savings figures

98%Amazon cost reduction
78%Reduction in average call time
70%Reduction in tokens per ticket
$0.10Gemini Decision, per million input tokens
Amazon figures are community reports; pricing is from Google's announcement. There is no independent verification.

How much can you switch?

The hardest part of switching is evidence of quality. Before switching, Amazon reportedly ran both models together on real traffic and validated them by applying only the existing model's results source. In effect, it compared the new model's decisions against real data without disrupting operations.

This week's evidence also has limits. The claim that Gemini Decision 1 leads across benchmarks and has the lowest calibration error comes from Google's own announcement source. The Amazon figures were passed along through the community source. The specifics of the Hugging Face blog post could not be confirmed from the collected material source. This is why you should re-measure on your own data rather than take the announced figures at face value.

There are also practical conditions to check when adopting. New LangChain releases have dropped Python 3.10 support and raised the minimum version to 3.11. In a 3.10 environment, older versions get installed sourcesource. Teams running on 3.10 need to upgrade their runtime before using the new decision model features.

Overall, this trend reads less as wholesale replacement of general-purpose LLMs and more as a division of roles. Decision models handle routing, classification and checks at the front end, while general-purpose LLMs remain where generation and explanation are needed.

What practitioners can try
  1. From the past month of LLM call logs, select only the calls whose output is a fixed label or yes/no, and calculate their share of total cost and latency.
  2. Pick one classification task as a switching candidate, run the existing LLM and a decision model together on real traffic while applying only the existing results, and compare accuracy.
  3. Set a confidence threshold and design a branch that sends cases below it to a general-purpose LLM or human review; if you use LangChain, first confirm the runtime is Python 3.11 or later.

Related terms

Decision model SLM Token billing LLM-as-a-Judge Human-in-the-loop Structured output A/B testing LLM gateway

Sources

This piece was drafted with AI, checked against the cited public sources for links and figures before publication, and reviewed again by the operator afterwards. It includes the author's interpretation. How we write and check · Corrections