Deep dives · 2026-11-21 · Advanced · By bombong
AI scientist agents: can they predict wet lab results?
AI scientists are agents that take on everything from forming hypotheses to designing and running experiments, but their ability to predict actual lab results is still weak. This article explains why that gap exists and how designs such as twins built from experimental records, critic agents and statistical tests try to narrow it.
Contents
- Why "wet lab prediction" is the test
- Core idea: a cook who has only read cookbooks
- How it works: the closed loop and experiment surrogate environments
- How it developed: from automation to verifiable discovery
- Where it is heading: scoring against real records
- Limitations and open issues: when to trust and when to doubt
- Further reading
- One diagnosis holds that the bottleneck for AI scientists lies less in ideas than in the ability to actually carry out verification (the implementation gap).
- LLM training data lacks the tacit procedures and failure experience of the lab, so it is hard to predict wet lab results by learning from text alone.
- As a result, a design approach is taking hold in which prediction is not a one-shot answer but one component of a closed loop: "predict → experiment → mismatch → revise."
- Even on TwinLab-Bench, which scores against real experimental records, frontier models struggled, and exploration that followed only the points where models disagreed missed unmeasured phenomena.
- In practice, AI predictions should not be used as grounds for skipping experiments. Use them as tools to narrow candidates and diagnose failures, with a human making the final judgment.
Why "wet lab prediction" is the test
An AI scientist (AI Scientist) is an agent system built around an LLM (large language model) that aims to handle the entire research workflow, from generating ideas to implementing experiments and writing papers. Expectations rose when AI-written research reports were accepted at an ICLR 2025 workshop and at ACL 2025. However, the implementation gap study paper systematically evaluated 28 papers produced by five AI scientist systems and concluded that the fundamental bottleneck is "the ability to actually carry out the necessary verification procedures." This gap is called the implementation gap.
A wet lab experiment is one that directly handles reagents, cells and samples. Unlike code experiments, each run costs time and money, and results fluctuate with equipment condition and environmental changes. So "can AI predict wet lab results in advance?" becomes the question that decides whether AI scientists are useful outside the computational world.
The position paper "Agentic AI scientists are not built for autonomous discovery" paper also raises this problem in terms of how it is measured. It points out that most science benchmarks measure only one-question, one-answer prediction accuracy and do not capture the feedback in which results from physical experiments flow back into computational models. In the end, properly measuring "is it good at prediction?" requires an evaluation environment connected to real experiments.
Core idea: a cook who has only read cookbooks
The same position paper paper points out that the text learned by LLMs, the foundation of AI scientists, lacks the tacit procedural knowledge and failure knowledge of the lab floor. It is like a cook who has read thousands of cookbooks but never actually worked over a flame. They can explain a recipe convincingly, but hands-on sense such as "this pan burns at the edges first" or "dough spreads on humid days" is rarely written down. The fact that failed experiments are almost never published is the same problem.
The paper also points out that preference optimization in the post-training stage narrows output diversity toward "consensus answers." This is risky for predicting experimental results, because new phenomena usually lie away from the existing consensus.
So the center of the design shifts. Instead of expecting the model to get the right answer from the start, prediction is placed as one component of a closed loop. Predict, check with an experiment (or an environment that stands in for one), and revise beliefs where things diverge. The price is experimental cost and time, plus a new problem of deciding when to stop the loop.
How it works: the closed loop and experiment surrogate environments
Taken together, recent systems generally run the loop in the following order. ① Hypothesis generation: create a hypothesis to test from the literature or earlier results. ② Experimental design: decide what to measure and under what conditions. ③ Execution: run it on real equipment, or in a surrogate environment built from simulations or experimental records. ④ Reading results: read the measurements and compare them with the hypothesis. ⑤ Critique and testing: another agent raises counterexamples or applies statistical tests. ⑥ Belief update: record what has been learned and what is still unknown, then move on to the next hypothesis.
What is used in ③ determines how "wet lab prediction" ability is measured. Real equipment is the most reliable but expensive and slow. The position paper paper recommends using scientific simulations as verifiers for training. TwinLab-Bench paper took another route. Each task has an environment agent called the "TwinLab agent," which reads the lab's raw data and calls calculators built from experimental records to run, in its place, the experiments designed by the agent under test. The idea is close to a digital twin of the lab.
The reason for choosing this design is consistency. The TwinLab-Bench researchers report that, compared with a language model that sees only the task description, the TwinLab agent nearly eliminates measurements that deviate from the real records and answers that change when the same request is repeated. They also ran closed-book checks, querying without the data, to tell whether a model was getting answers right by memorization.
⑤ and ⑥ are not simply tacked-on steps either. The position paper paper argues that a "persistent, revisable epistemic state" that carries beliefs and shifting goals throughout an investigation is needed. If a wrong prediction is not reflected in the next hypothesis, running the loop teaches nothing.
How it developed: from automation to verifiable discovery
Jr. AI Scientist paper mimics the research workflow of a junior graduate student. It analyzes the limitations of a baseline paper given by a human mentor, forms improvement hypotheses, repeats experiments until performance improves, and then writes a paper. Its novelty was handling complex implementations across multiple files with coding agents. However, the authors' evaluation and Agents4Science reviews revealed important limitations, and the researchers released a risk report alongside it.
After the 2025 implementation gap study paper identified "the ability to carry out verification" as the bottleneck, the 2026 position paper paper broadened the problem to more fundamental design choices. The paper names four challenges: the McNamara fallacy of choosing only problems that are easy to measure, training data lacking the lab's tacit knowledge, outputs narrowing toward consensus, and benchmarks without physical experiment feedback. The authors argue these challenges cannot be solved just by scaling models or adding harnesses.
AHOIS paper is an example of running the loop on a real physical system. On a multimode optical fiber setup, a physics critic agent presses hypotheses with "Socratic" questioning: asking about causation, checking constraints, constructing counterexamples and setting falsification criteria. The system formed and tested a random interference encoding hypothesis on its own. It diagnosed distinct failure causes, namely encoding instability, fluorescence contamination and detector noise, and translated a published imaging protocol into an executable procedure that runs on a device configuration different from the original. The discovered encoding achieved an effective rank of 56.9 on 16×16 measurements, 76.97% classification accuracy on MNIST and 83.17% on Fashion-MNIST. In ablation experiments, Socratic questioning improved physical consistency.
Research on verification and oversight appeared in the same period. DiscoPER paper requires every proposed discovery to pass a statistical test, and adds second-order reasoning that reanalyzes the accumulated discoveries as data to find confounders and gaps. AIMC paper visually analyzes papers produced by the autonomous AI scientist FARS to help humans find recurring methodological weaknesses. The "agent economy" study paper argues that because hypothesis testing and experiments consume large and limited physical and economic resources, systems for resource allocation, credit attribution and accountability tracking are needed alongside reasoning ability.
Where it is heading: scoring against real records
TwinLab-Bench paper divides the abilities AI scientists need into three increasingly open-ended levels: accurately carrying out a given computational procedure, choosing the next measurement within a given design space, and designing experiments on their own. Existing benchmarks mostly measured only the first two. TwinLab-Bench contains 52 tasks from physical science, life science and engineering labs and more than 100 types of experimental actions, and scores agent reports against hidden ground truth taken from real lab records. The researchers report that these tasks remain difficult even for frontier models, and that even the strongest, Claude Fable 5.2, had a low average score. This means that "predicting the results of experiments it designed itself" is still an open problem.
The hypothesis chain study paper reveals a structural blind spot in the prediction loop. The researchers deliberately removed a narrow peak from the initial data and had two frontier AI scientists (Claude Sonnet 5.5, GPT-6.1-Sol) run 32 closed-loop trials. Both models measured the peak region less than random exploration did. Because the candidate models drew similar simple curves in the unmeasured region, the strategy of "experiment where models disagree" gave no reason to look there. The agent that followed that strategy more faithfully missed the peak in all 32 trials. Conversely, cases that probed the unmeasured region early and followed anomalous signals to the end succeeded in the discovery.
Sage-Taiyi paper uses a metacognitive architecture (the ability to check and regulate its own understanding) to choose the next research target and method. On Taiyi-Bench, 265 tasks drawn from 50 papers across 10 fields, it scored 54.0 overall, ahead of the strongest baseline, AI Scientist-v2, at 47.9. More notable are the results of five expert-led collaborations in which its proposals were checked with real experiments. The yield of a cobalt-catalyzed carbonylation reaction rose from trace amounts to 84%, and the yield of a photocatalytic allylic amination reaction rose from 13% to 69%. However, these should be read as cases of optimizing reaction conditions together with human experts, not cases of AI predicting experimental results on its own.
Open-endedness of AI scientist capabilities
- Carrying out computational procedures accurately
- Choosing the next measurement from given candidates
- Designing its own experiments · reporting results
Limitations and open issues: when to trust and when to doubt
Based on the evidence so far, it is hard to say that "AI scientists accurately predict wet lab results on their own." Scores were low even on TwinLab-Bench paper, which scores against real records. The cases with lab results paperpaper came from loops involving real equipment or experts. Benchmark scores should not be read as "predictive performance in the lab." Another issue is memorization. If the results of an experiment have already been published, the model may be recalling rather than predicting. This is why TwinLab-Bench includes a separate closed-book check.
There are also methodological issues. The position paper paper proposes preregistering every AI-generated hypothesis in one place. Because agents generate hypotheses in bulk, reporting only the ones that happened to be right can look like discovery. The McNamara fallacy noted in the same paper, that is, the tendency to choose only problems that are easy to measure, also calls for caution. Resources are a problem too. As the agent economy study paper notes, experimental resources are limited, so how many times to run the loop and who bears the cost and responsibility should be decided first.
The practical criteria are as follows. These systems are useful for narrowing experiment candidates, diagnosing causes of failure, and organizing and tracking experimental records. On the other hand, skipping confirmation experiments on the strength of AI predictions alone should be avoided. There are three common mistakes: experimenting only where models disagree paper, not checking whether answers change when the same request is repeated, and accepting discoveries without statistical tests. It is safer to make a human-in-the-loop structure, in which humans handle hypothesis selection and final interpretation, the default.
Checklist for using AI scientist predictions
Evaluation
- Was it scored against real experiment records?
- Was it checked closed-book to rule out memorized answers?
- Does the same request give the same answer when repeated?
Exploration
- Is it only experimenting in regions where the model disagrees?
- Does it probe unmeasured regions early?
- Did each finding go through a statistical test?
Operations
- Were hypotheses preregistered?
- Are experiment resources and the accountable party defined?
- Does a human handle the final interpretation?
Further reading
To grasp the core problem, read the implementation gap study paper and then the position paper paper. The first gives the diagnosis that "execution is the bottleneck"; the second gives a causal analysis and prescriptions that go down to training data, post-training and benchmark design.
For a concrete look at loop design, read AHOIS paper and DiscoPER paper together. You can compare using critic agents to maintain physical consistency with using statistical tests to filter discoveries. If you are interested in human oversight and resource questions, read AIMC paper and the agent economy study paper.
For the latest evaluation trends, start with TwinLab-Bench paper. Then look at the hypothesis chain study paper for the pitfalls of exploration strategies, and Sage-Taiyi paper for a case that extends to lab validation, to get a fuller picture of the current answer to the question "can they predict?"
Related terms
AI agent Agentic workflow Digital twin Benchmark Human-in-the-loop Multi-agent Post-training AI drug discovery
References
Foundational papers
- AI Scientists Fail Without Strong Implementation Capability (2025)
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery (2026)
- Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper (2025)
- Socratic agents for autonomous scientific discovery in high-dimensional physical systems (2026)
- Autonomous Scientific Discovery via Iterative Meta-Reflection (2026)
- AI Scientist Mission Control (AIMC): Visual Analytics for Human Oversight of Autonomous Scientific Discovery (2026)
- Agentic Economies for Autonomous Scientific Discovery (2026)
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.