FocusGUI: An omni-modal GUI agent that chooses what to focus on at each step

Problem

A GUI agent receives the screen, audio, video, action history and device state all at once at every step. In the authors' analysis, the modality most important for the decision changed between adjacent steps 45.3% of the time, and 90.1% of steps had at least one distracting modality. Because each executed action feeds back into the next input, per-step misreadings accumulate and the Trajectory drifts off course; the authors call this Sequential Distraction. Existing methods that crop part of the screen cannot handle audio or video signals, and they also discard the surrounding context needed to determine coordinates.

Method

FocusGUI does not reduce the input; it only changes how the decoder reads it. It projects each modality and the screen spatial channels into a shared space (128 dimensions) and computes softmax weights by comparing them with a per-step reference vector built from the full observation. The weighted sum is compressed into a small 64-dimensional focus latent, which is added as a zero-initialized residual to the embeddings at the 4 prompt positions just before action generation. The backbone stays frozen, and about 1.15 to 1.45 million parameters are trained using only the likelihood of the next action tokens. No labels indicating which modality matters are needed.

Results

• On the OmniGUI benchmark (29 smartphone apps, 708 episodes), the average Type Match across four omni-modal backbones rose from 31.0% for the frozen backbones to 64.2%, and Exact Match rose from 11.2% to 33.5%.
• With Qwen2.5-Omni, overall TM was 71.65 and EM 40.16, higher than the strongest baseline, GUI-Actor (50.20, 13.78). TM and EM were highest in all 24 cells combining four backbones and five task types.
• Screen-refining baselines (DRS-GUI, OmniParser, RegionFocus) changed TM by only +0.9 points on average, and in relative terms lowered EM by 28% and Goal Progress by 35%.
• The authors report that from the first step to the fourth, TM for frozen Qwen2.5-Omni fell by half, while FocusGUI retained 86.8% of its first-step TM.

Limitations

Evaluation used only per-step comparison against ground truth on a single smartphone GUI benchmark (OmniGUI, 2,572 action steps). Closed-loop evaluation, in which the agent continues acting on its own in a real environment, was not confirmed. Goal Progress remains low at 6.8–17.5 even after the improvement. The authors also state that the modality weights do not fully explain the internal decisions of the frozen backbone.

Why it matters

For GUI agents that also handle off-screen signals such as voice notifications and video changes, this shows that telling the model where to focus can be more effective than adding or cropping information. Because it is a small adapter that leaves the backbone untouched, it is easy to attach to existing omni-modal models.

Key terms

Sequential Distraction

A phenomenon in which small errors caused by being distracted by irrelevant signals at each step feed into the next step's input through executed actions and gradually accumulate.

Omni-modal model

A model that processes multiple kinds of input, such as text, images, audio and video, together in a single network.

Zero-initialized residual

An additional signal designed to start at zero, leaving the original model's output unchanged, and to gradually add corrections as training proceeds.

Type Match / Exact Match

Type Match checks whether the action type, such as click or text input, is correct. Exact Match counts an answer as correct only if the coordinates and input text are also exact.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, et al. · Posted 2026-10-19 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.