Paper review · October 21, 2026
OR-License: Evaluating LLM agents as real-world operations research (OR) practitioners
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
The authors turned 31 real paid OR projects into a benchmark using the original materials as-is and evaluated 11 frontier models; none reached expert-level, deployment-ready quality.
Problem
Operations research (OR) is the field that optimizes decisions such as scheduling, dispatching and allocation in manufacturing, logistics, healthcare and retail. In practice, experts turn incomplete client materials into a working system by clarifying requirements, formulating the problem mathematically, choosing algorithms and implementing solvers. Most existing OR benchmarks start from an already well-organized problem description, so they cannot evaluate the process of interpreting raw business materials and asking the client to fill in missing information.
Method
From paid OR contract projects worth a total of $3.68 million, the authors took 31 instances across 10 industries and de-identified them while keeping the original requirements, input data, deliverable formats and information boundaries intact. The client simulator ORSim provides only business facts, not OR expertise, and a deterministic controller limits it to revealing at most 3 facts per query. Submitted answers are checked for feasibility with a hidden verifier, and their objective values are compared with the expert solution the client actually accepted. The metric DAQ (Deployment-Adjusted Quality) averages quality relative to the expert but gives a score of 0 to infeasible submissions. In addition, the authors manually annotated trajectories to analyze when information was acquired, hypothesis-experiment-revision chains (HCAEU), and 8 error types.
Results
- In the Claude Code harness, all 11 models fell short of the expert baseline DAQ of 1. The top model, Grok 4.6, had a strict feasibility rate of 88.39% and a DAQ of 86.55%, a gap of 13.45 points.
- Differences between models came down to 'whether they produce a feasible answer' rather than quality. Feasibility rates ranged from 45.16% to 88.39%, a spread of 43.23 points, while objective quality on feasible submissions alone ranged only from 96.92% to 101.01%, a 4.09-point range.
- Several models achieved objective values above 100% of the expert baseline on the problems they solved. However, the authors cautioned that these are different subsets for each model and therefore do not provide grounds for directly comparing optimization ability.
- Of Grok 4.6's 13.45-point gap, 11.61 points came from infeasible submissions and 1.84 points from insufficient quality.
- Client queries were not limited to initial requirement clarification but continued into the solution development and verification stages, and in some cases the share of issues resolved was higher for queries made in later stages. Research-chain completion rates on evaluable tasks ranged from 24% to 45% across models.
Limitations
With 31 instances and 5 runs per model-task pair, the scale is not large. The authors stated that the reported deviations are within-task variation, not the standard error of the benchmark mean. The main evaluation harness is Claude Code alone, and the comparison with the Codex harness is relegated to the appendix, so harness effects may be mixed into the results. The trajectory analysis relies on manual annotation. The provided text is cut off partway through RQ1, so detailed figures for RQ2 and RQ3 could not be checked.
Why it matters
The paper shows in numbers what the bottleneck is when agents are put to work on real contract projects. The finding that the reliability of producing 'feasible outputs that satisfy all constraints' is a much larger source of the gap than solution quality prompts a rethink of criteria for evaluating enterprise adoption. It also differs from existing benchmarks in including, as an evaluation item, the ability to probe for information when requirements are incomplete from the start.
Key terms
- Operations research (OR)
- A mathematical decision-making field that optimizes things like scheduling, dispatching and allocation under limited resources. Problems are usually modeled as equations and solved with dedicated solvers.
- DAQ(Deployment-Adjusted Quality)
- A metric that averages objective quality relative to the expert solution but gives 0 points to infeasible submissions that violate even one constraint. The expert baseline value is 1.
- ORSim
- A simulator that plays the client, revealing only pre-approved business facts, at most 3 per query, and providing no OR expertise.
- Strict feasibility
- A strict success rate that gives 1 point only when all official test cases for an instance pass the hidden verifier.
Original
Authors: Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, et al. · Posted 2026-10-16 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.