← Key papers

Paper review · November 13, 2026

Prompt-level cost-aware LLM ensemble routing (PISA, PISA-E)

Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

Efficiency & infraLanguage models Intermediate Upvotes 14 cs.AI

By estimating each LLM's success probability per prompt and choosing both which models to use and how many, it improved ensemble performance while cutting cost by up to about 36%.

Problem

Services that use multiple LLMs have to balance quality and cost for each request. Existing routing picks only one model per prompt and cannot combine the strengths of several models. Fixed ensembles cannot change their composition or size per request. Many ensemble methods fix the ensemble size at training time, which also makes it hard to change cost preferences at inference time.

Method

PISA represents a prompt embedding as soft membership (membership probabilities) over X-means cluster centers and estimates each model's probability of answering correctly with a per-model logistic regression. It then picks a single model using a utility function in which α sets the weighting between quality and cost. PISA-E reuses the same estimates. It ranks models by utility, takes the top odd-numbered sets (1, 3, 5…) as candidate ensembles, and, assuming the models are independent, uses 'the probability that a majority is correct' as a proxy for ensemble quality. It evaluates this together with total cost to pick the best candidate, and outputs as the final answer the response from the selected models with the highest BERTScore weighted by estimated ability. α can be changed at inference time without retraining.

Results

Limitations

The ensemble quality proxy assumes that the models' correctness is mutually independent. The candidate search, which grows the ensemble from the top down, does not guarantee the globally optimal combination. Model cost is estimated from the average cost in the training data rather than per prompt. Evaluation used only model responses provided in advance by the benchmarks, and actual serving latency was not measured. The input text was cut off, so the list of baselines and detailed numbers were only partly checked.

Why it matters

It offers a lightweight way for services that mix commercial and open LLMs to decide, per request, 'which models to call and how many' according to cost preferences. Because a single α shifts the operating point between quality and cost without retraining, it is easy to apply to real operating policies.

Key terms

LLM routing
A technique that decides which of several LLMs should handle an incoming prompt, weighing quality and cost.
Hypervolume
The area covered by a method's operating points on the cost-performance plane. The larger it is, the better the overall performance for the cost.
Soft membership
A representation that, instead of assigning an input to a single cluster, expresses how close it is to each of several clusters as probabilities.
MBR (Minimum Bayes Risk) decoding
A method that picks the candidate response most similar to the other responses, that is, the one closest to consensus.

Original

Authors: Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini · Posted 2026-11-06 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.