Paper review · October 24, 2026
Deep search on the real web: the OpenDS-World benchmark
Deep Search in Real-World Web Environments
The authors built 200 tasks that start from a video and require going through multiple web sources, formats and interactions to find the answer, and measured the agentic deep search ability of 12 recent models.
Problem
Deep search agents have advanced quickly, but their ability to carry out end-to-end research on the real open web has not been adequately evaluated. Real web information is scattered across multiple modalities (text, image, audio, video) and formats (HTML, PDF, XML, Parquet), and often can only be reached through interactions such as operating a virtual tour or entering parameters. Existing benchmarks do not cover these together.
Method
Each task starts from a public video. Expert annotators choose an anchor entity (mapped to a Wikidata item) that can be identified only from the video's audiovisual content. They then extend an evidence path by selecting sources through Google Search, cross-checking each node against multiple sources. Finally, the path is reversed to build a question in the form of a combination of constraints that describe the intermediate entities indirectly. Ten reviewers verified the evidence and the uniqueness of the answers, finalizing 200 tasks across 8 domains. Fifty people attempted the tasks independently, and the 12 models were evaluated with the same tool environment and execution protocol.
Results
- The best-performing model, GPT-6 Astra, reached 39.0% accuracy on the 200 tasks.
- Human participants, without LLM tools, reached 21.1% accuracy over 1,000 independent attempts, and 42.5% of tasks were not solved by any of the people assigned to them.
- The authors report that more tool calls or longer run times do not lead to higher accuracy. They interpret this as a limitation in source selection, tool coordination and long-horizon task management.
Limitations
The benchmark is small at 200 tasks, and every task follows the fixed structure of starting from a video. Because it uses the real open web, reproducibility may decline as pages change or disappear over time (the videos are archived and distributed). The human baseline prohibited LLM tools, which differs from the conditions of real-world users.
Why it matters
When evaluating in-house research automation or deep research products, it offers a yardstick for multimodal, heterogeneous-format and interactive access abilities that static search benchmarks do not reveal. It also shows that even the best model does not exceed 40%, leaving substantial room for improvement.
Key terms
- Deep search agent
- An AI agent that answers complex questions by using search, browsing and file-processing tools over multiple steps to gather and connect evidence.
- Evidence Path
- The chain of entities and sources leading from the starting clue to the final answer. Each link is backed by verifiable web evidence.
- Question Inversion
- A method that reverses a completed evidence path to build a question from a combination of indirect descriptions that do not directly reveal the intermediate steps.
- Interactive access requirement
- Information that cannot be obtained through simple search and only appears after actions such as navigating pages, entering parameters, operating an interface or downloading files.
Original
Authors: Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, et al. · Posted 2026-10-19 · License CC BY-NC-ND 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.