FrontierPhysics: A Benchmark for Agents Carrying Out the Full Physics Research Process

Problem

Existing physics benchmarks mostly use fixed questions, split-up subtasks, or tasks that reproduce an original paper given to the model. As a result, they cannot evaluate the full open-ended research process from literature review to paper writing. They also rarely assess research quality alongside correctness.

Method

Tasks were built from research actually carried out by 42 working physics researchers (from 26 institutions). They cover nine areas of theoretical, computational and experimental physics; of 182 submissions, 56 were accepted after general review, domain peer review and failure-case analysis. The agent receives a prompt at the level of a PI's instructions, performs literature review, research planning and experiments, and submits numerical outputs and a paper.pdf. Grading combines deterministic verifiers (test scripts) with rubric evaluation by a reviewer agent (GPT-5.6 Sol). A run passes only if it passes both the verifiers and all required (blocker) rubric items; non-required items form a weighted quality score. Each task was run three times across 22 models, 26 configurations and 4 harnesses.

Results

• No configuration passed more than 46 of 168 rollouts. The overall pass rate (Pass@1) was below 30% for every agent, with the best being Claude Opus 5.5 at 27.4%.
• In general, spending more raised the pass rate, but for some Claude models the pass rate fell as cost rose (in the order Opus 5.5 > Fable 5.1 > Opus 5). The authors report that GPT models were generally more cost-efficient.
• When regraded with Claude Opus 5.5 as the reviewer, pass/fail outcomes for GPT-6 Astra rollouts were nearly the same. However, it was more lenient toward rollouts from its own model family.
• A manual analysis of 168 GPT-6 Astra rollouts found that the main weaknesses were not a lack of physics knowledge or execution errors, but non-rigorous reasoning and a lack of expert intuition.

Limitations

With only 56 tasks and few tasks per area, differences between areas are uncertain (noted by the authors). Much of the grading relies on an LLM reviewer agent, and self-family bias was observed depending on the reviewer model. The manual failure-mode analysis was limited to one model (GPT-6 Astra). Each task had three rollouts within a 10-hour limit, which differs from real research conducted over long periods.

Why it matters

It offers a framework for testing claims that AI agents "do scientific research" through the full research process and paper quality, rather than problem solving. Its diagnosis that current top models are limited more by reasoning rigor and research sense than by knowledge is useful for setting the development direction of research-assistant agents and expectations for adopting them.

Key terms

End-to-end research benchmark

An evaluation approach in which an agent carries out and is assessed on the entire research process: literature review, planning, experiments, result verification and paper writing.

Agent-as-judge

An evaluation approach in which, instead of a single LLM call, a tool-using agent directly inspects trajectories, code and output files and grades them with a rubric.

Blocker rubric

Pass/fail criteria essential to scientific validity. If any one fails, the final score is 0 even if other scores are high.

Pass@1 / Pass@3

Pass@1 is the average pass rate over three rollouts, and Pass@3 is the share of tasks passed at least once in three tries.

Original

• arXiv abstract
• PDF
• HTML full text
• Project
• Hugging Face

Authors: Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma, et al. · Posted 2026-11-04 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.