Deep dives · 2026-10-24 · Advanced

GUI agents: how AI that looks at the screen and clicks works, and its open problems

GUI agents operate computers and browsers the way people do, repeating a loop of looking at a screenshot and acting with the mouse and keyboard. The focus of recent research is shifting from "getting a single click right" to holding up over long tasks. The topics are memory, division of labor among multiple agents, error recovery, completion verification and security.

Contents

Why it was needed: work in a world without APIs
Core idea: a person operating someone else's computer over screen sharing
How it works: the observation-action loop
How the field developed: data, training, memory, division of labor
Where it is heading now: recovery, verification, training environments
Limitations and open issues: risks that come from operating through the screen
Further reading

Key points

A GUI agent repeats "observe → decide → locate the target on screen (grounding) → act → observe again." When the number of steps grows to dozens, small mistakes accumulate and lead to failure.
Performance is not determined by model size alone. The execution environment (harness), training data, memory structure and task decomposition approach all play a role.
Recent research treats the ability to notice and undo errors, and the separate verification of an agent's "I succeeded" reports against the logs, as core problems.
Some security risks arise only from operating through the screen, such as attacks that change the screen during the few seconds between observation and action, and personal data leaks by browser agents.
Designs are needed in which a person gives final approval for hard-to-reverse actions and the agent's self-reports are not taken at face value.

Why it was needed: work in a world without APIs

Much of a company's work happens in programs that have no API (an official channel through which programs exchange data). Old internal systems, desktop apps and websites that require a login are examples. People use these programs by looking at the screen and clicking. So if AI could work with them the same way, they could be automated without building separate integrations. This idea gave rise to GUI (graphical user interface) agents, which the industry also calls computer use agents or browser agents.

The problem is that screens are made for people. Button positions change with resolution and theme, and pop-ups and notifications can appear at any time. A desktop GUI security study paper defines these agents as systems that control a computer through a "take a screenshot and click" loop. The loop is simple, but if a single step goes wrong, every following step goes off track.

Agents that work well on short tasks break down on long ones. Wuying-Browser-Agent paper points out that, unlike clean short demos, real deployments require making dozens of decisions on live websites and recovering from mistakes. The execution memory study EAM paper likewise argues that reinterpreting the UI and re-deciding the action on every screen is fragile for long tasks.

Core idea: a person operating someone else's computer over screen sharing

A GUI agent can be compared to a remote support agent. Watching a customer's shared screen, the support agent decides "click the settings button in the top left" and moves the mouse themselves. They cannot see the code inside and judge only by what is visible on the screen. After clicking, they check again how the screen has changed.

In this analogy, the AI has three jobs. First, the eyes that understand the screen. This is mostly done by a VLM (vision-language model, a model that handles images and text together). Second, planning what to do next. Third, the ability to pin down exactly which coordinates on the screen the "settings button" is at, which research calls grounding. If any one of the three goes wrong, the agent clicks in the wrong place.

As a support agent works longer, they build up know-how such as "in this company's system, this menu comes after that one." Large jobs may also be split among several people. Recent GUI agent research is expanding in these two directions as well: remembering and reusing experience, and dividing work among multiple agents.

How it works: the observation-action loop

The flow of one step is as follows. ① The user gives a goal (e.g., "organize this month's receipts in Excel"). ② The agent observes the current screen. Screenshots are often used, and in browsers the agent may also read the web page structure (DOM). ③ The model looks at the goal and the history so far and decides the next action. ④ It locates the screen position to act on. ⑤ It actually executes an action such as clicking, typing, scrolling or switching windows. ⑥ It observes the changed screen again and returns to ③. When it judges that the goal has been reached, it stops and reports the result.

The execution environment that wraps the model and runs this loop is called a harness. In Wuying-Browser-Agent paper, a structured browser harness handles stable execution primitives and management of the context needed for decisions. ShowUI-Aloha paper, which learns from human demonstrations, designed its executor to perform clicks, drags, typing and window operations at the operating system level while also running safety checks and providing real-time feedback.

For long tasks, agent memory, which holds "what has been done so far and what has been learned," matters. If the agent decides based only on the screen at each step, it forgets information it saw earlier. So structures are added that bring past experience into the context. The final completion check is a hard problem in its own right and is discussed again later.

The observation-action loop of a GUI agent Given a goal, the model decides on an action and executes it. The loop repeats, with the changed screen and past experience feeding back into the model, until it judges the task complete and stops.

How the field developed: data, training, memory, division of labor

The studies below all appeared between late 2025 and 2026. Rather than building on one another, they are more accurately seen as tackling the same problem at different layers.

Starting with data and training, ShowUI-Aloha paper targeted the shortage of good training data. It built a pipeline of a recorder that captures screen recordings and clicks and keystrokes of people working on desktops, a learner that turns these into natural-language descriptions, a planner that plans the next action, and an executor that actually carries it out. Continual GUI Agents paper showed that the grounding of existing methods becomes unstable in environments where new domains or resolutions keep arriving. It therefore proposed GUI-AiF, a reinforcement learning fine-tuning method that gives rewards based on interaction points and regions so the model does not overfit to fixed coordinates or element sizes. Wuying-Browser-Agent paper argues that scale alone is not enough and that execution, supervision, optimization and evaluation must all be aligned. It used curriculum SFT that learns from error-recovery trajectories and complex UIs, and online reinforcement learning with refined credit assignment for long tasks. It also built BrowserBench, made up of 350 tasks averaging 37.9 steps. A 27B model trained this way scored 80.6% on WebVoyager, 66.7% on Online-Mind2Web and 65.1% on BrowserBench, which the authors report as the best open-source results on browser use benchmarks.

On the planning and execution side, EAM paper changed GUI planning from "generating anew every time" to "retrieving and executing." It compresses multi-step routines into a knowledge graph, and a lightweight value model guides search over the graph. It reportedly outperformed UI-TARS-7B by up to 19.6% on AndroidWorld, cut token cost to one-sixth that of GPT-4o, and had an average latency of 2.8 seconds. Multi-Agent Computer Use paper addressed the limits of a single agent working sequentially. A manager model splits the task into a dependency graph (DAG), assigns ready subtasks to subagents in parallel, and keeps revising the graph based on newly learned facts. It scored 3.4–25.5% higher than a single agent on desktop and web benchmarks.

Where it is heading now: recovery, verification, training environments

Studies released in October 2026 are shifting attention from "how often it succeeds" to "how it handles failure." CUA-Recovery paper focused on how, in long tasks spanning several desktop apps, errors in earlier steps silently corrupt the state and cause cascading failures. It built a test set of 2,250 human-verified error states (100 tasks) and a training set of 8.6K trajectories (377 tasks). Evaluating six agents, the rate of noticing inherited errors (EAR) reached at most 44.0%, and Pass@3 at most 25.5%.

Completion checking is also being studied on its own. A study on self-report leakage paper points out that when an agent confidently reports "success," the next step may proceed as is even if the execution log provides no evidence. When the same judge model was given the same task and log with only the agent's self-report removed, the rate of catching dangerous false reports rose by 43.4 percentage points for Qwen3.5-35B-A3B and 13.3 percentage points for DeepSeek-284B-A13B. The researchers therefore built an auditor that separates claim interpretation from log analysis, achieving an F1 of 0.75–0.78 on 465 audited cases.

Research on building training environments is also active, because running reinforcement learning at scale on real apps is difficult. Trace2World paper reconstructed 50 executable app environments from 1,570 real interaction traces. It reproduced 91.1% of 200 workflows across ten apps end to end, higher than the strongest baseline (59.8%). In environments verified against provenance traces, 94.0% of the gains from simulated training held up on real devices, compared with 54.7% with self-verification only. On 100 real-device tasks, scores rose by 16.3 points. This means that how closely the environment matches the real app is key to carrying simulated gains over to the real world.

Trace2World and baselines

Trace2WorldBaseline

Workflow recall−31.3%p

91.1%

59.8%

On-device learning retention−39.3%p

94%

54.7%

The left bar shows Trace2World and the right bar shows the baseline. For recall, the baseline is the strongest task-level method; for on-device retention, the baseline is self-verification.

Limitations and open issues: risks that come from operating through the screen

A design that looks at the screen and acts has its own attack surface. The desktop GUI security study paper measured an average gap of 6.51 seconds between observation and action in real OSWorld tasks. In this window, an unprivileged attacker can swap the screen, a TOCTOU (time-of-check to time-of-use) attack. An attack that manipulated window focus redirected actions elsewhere 100% of the time, leaving no visual trace at observation time. The researchers' proposed PUSV is a three-layer defense that rechecks the screen state just before acting. In 180 attack trials it blocked 100%, with zero false positives and under 0.1 seconds of added time. However, it blocked almost 0% of web DOM injection attacks, which leave no trace on the screen, exposing a structural blind spot. The researchers conclude that no single layer can stop every attack.

Privacy is another major issue. A browser agent privacy study paper evaluated eight popular browser agents on 5 factors and 15 measurement items and found 30 vulnerabilities. These included turning off the browser's privacy protection features and "autofilling" sensitive personal information into input fields. Because agents act with the user's login sessions and personal information delegated to them, the damage from an incident can be large. Indirect prompt injection, which steers an agent through instructions hidden in web pages, also calls for caution in the same context.

Benchmark scores should not be read as real-world job performance. The figures above come from the environments and tasks each study defined. In practice, the baseline is to combine human-in-the-loop final approval before hard-to-reverse actions such as payments, deletions and sending, isolated environments such as sandboxes, and the principle of least privilege. A procedure is also needed that separates the fact that an agent said "done" from the fact that the task was actually done, and confirms the latter from the logs.

GUI agent security and privacy figures

6.51 sAverage gap between observation and action

100%PUSV block rate (180 trials)

About 0%DOM injection attack block rate

30Browser agent vulnerabilities

The observation-action gap and the PUSV block rate come from a desktop security study; the vulnerability count comes from a privacy study that evaluated eight browser agents.

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.