Paper review · November 13, 2026
Prompt-level cost-aware LLM ensemble routing (PISA, PISA-E)
Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference
By estimating each LLM's success probability per prompt and choosing both which models to use and how many, it improved ensemble performance while cutting cost by up to about 36%.
Problem
Services that use multiple LLMs have to balance quality and cost for each request. Existing routing picks only one model per prompt and cannot combine the strengths of several models. Fixed ensembles cannot change their composition or size per request. Many ensemble methods fix the ensemble size at training time, which also makes it hard to change cost preferences at inference time.
Method
PISA represents a prompt embedding as soft membership (membership probabilities) over X-means cluster centers and estimates each model's probability of answering correctly with a per-model logistic regression. It then picks a single model using a utility function in which α sets the weighting between quality and cost. PISA-E reuses the same estimates. It ranks models by utility, takes the top odd-numbered sets (1, 3, 5…) as candidate ensembles, and, assuming the models are independent, uses 'the probability that a majority is correct' as a proxy for ensemble quality. It evaluates this together with total cost to pick the best candidate, and outputs as the final answer the response from the selected models with the highest BERTScore weighted by estimated ability. α can be changed at inference time without retraining.
Results
- On four routing benchmarks (LLMRouterBench, RouterBench, FusionBench, SPROUT), PISA achieved the highest macro-averaged cost-performance hypervolume among single-model routers both in-distribution (ID) and out-of-distribution (OOD), ranking first in 6 of 8 benchmark-setting combinations.
- In quality-first settings, PISA-E's macro-averaged task performance was 2.09 percentage points higher in ID and 2.87 percentage points higher in OOD than the strongest competing ensemble method.
- In lower-cost settings as well, PISA-E outperformed the strongest competing method on task performance while cutting average inference cost by about 36% in ID and about 25% in OOD.
- The authors reported that, compared with variants using a fixed ensemble size, varying the size per prompt was advantageous.
Limitations
The ensemble quality proxy assumes that the models' correctness is mutually independent. The candidate search, which grows the ensemble from the top down, does not guarantee the globally optimal combination. Model cost is estimated from the average cost in the training data rather than per prompt. Evaluation used only model responses provided in advance by the benchmarks, and actual serving latency was not measured. The input text was cut off, so the list of baselines and detailed numbers were only partly checked.
Why it matters
It offers a lightweight way for services that mix commercial and open LLMs to decide, per request, 'which models to call and how many' according to cost preferences. Because a single α shifts the operating point between quality and cost without retraining, it is easy to apply to real operating policies.
Key terms
- LLM routing
- A technique that decides which of several LLMs should handle an incoming prompt, weighing quality and cost.
- Hypervolume
- The area covered by a method's operating points on the cost-performance plane. The larger it is, the better the overall performance for the cost.
- Soft membership
- A representation that, instead of assigning an input to a single cluster, expresses how close it is to each of several clusters as probabilities.
- MBR (Minimum Bayes Risk) decoding
- A method that picks the candidate response most similar to the other responses, that is, the one closest to consensus.
Original
Authors: Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini · Posted 2026-11-06 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.