FDE-Gym: A Forward Deployed Engineer Benchmark for Evaluating the Last Mile of AI Adoption

Problem

Existing agent benchmarks give tasks that are well defined by issues, specifications or tickets. Engineers deployed at real customer sites (FDEs) must clarify vague requirements, work on systems in production and uncover unstated constraints. Sometimes the request itself targets the wrong problem. No benchmark evaluated these abilities.

Method

Industry experts adapted real delivery projects carried out in China, Singapore and the United States since 2024 into 20 scenarios across 18 industries. Including variants with key facts changed, they created 95 tasks in total. The agent starts in a workspace with a copy of the production system, documents and a staff directory. It modifies the system while asking questions of about 130 stakeholders whose responses are fixed by scripts. The score is the improvement in customer KPIs on held-out data, normalized so that doing nothing is 0 and the expert reference solution is 1. Stakeholder contact costs (messages, test runs, waiting) are subtracted from this, and an incident such as an unauthorized action sets any positive score to 0. The amount of instruction information is divided into four levels, from L0 (both the problem and constraints provided) to L3 (the sponsor's original request), to isolate bottlenecks. The authors verified that solutions implementing the request literally receive 0.2 or less, and shortcut solutions receive 0.8 or less.

Results

• In the L3 setting, Claude Opus 5.5 scored the highest with an average of 0.56 and solved 14 of 95 tasks. Five of the 13 configurations solved none. Even picking only the best-performing configuration for each task gave an average of 0.70 and 30 tasks solved.
• When told the constraints, the top 3 models scored 0.73–0.82, but when they had to find them on their own, scores fell sharply to 0.39–0.62.
• The top models obtained 72–85% of the key facts from stakeholders, but the improvement from the systems they delivered was only about 0.60–0.70 of the reference solution. Contact costs cut scores by a further 0.14–0.22.
• Claude Haiku 4.5 scored below the do-nothing baseline on 79 of 95 tasks. The authors reported that weak models can actually make the customer's situation worse.

Limitations

Stakeholder responses are fixed by scripts for reproducibility, and a small model maps questions to topics, so they are less flexible than real conversations. The execution environment allows only Python and SQLite and blocks general network access, making it more restrictive than real sites. The L0–L2 comparison was done only on the 20 original tasks. The benchmark has 95 tasks.

Why it matters

This shows in numbers that the truly hard part of adopting AI in companies is not implementation but figuring out 'what the real problem is' and finding 'hidden constraints.' It can serve as a checklist before putting agents to work at customer sites.

Key terms

FDE(Forward-Deployed Engineer)

An engineer whom a vendor places inside a customer organization to handle everything from understanding requirements to building and deployment.

Normalized business uplift

A value showing how much a delivered system improved customer KPIs, scaled so that doing nothing is 0 and the expert solution is 1. A value below 0 means things got worse than before.

Task instruction levels (L0–L3)

A design that keeps the environment and scoring unchanged and only reduces the information in the task description. It makes it possible to separate whether failure happens in implementation, constraint discovery or problem reframing.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, et al. · Posted 2026-10-15 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.