AI Trend Research

← AI glossary

AI glossary · Basic · Agents

Computer use

Computer use is the ability of an AI to look at a screen and operate computer programs directly, like a person, by clicking a mouse or typing on a keyboard. A key feature is that it can handle programs that have no dedicated connection interface (API).

About this term
Level
Basic · Technical terms you often meet in AI news
Fields
Agents
Added
2026-10-08
In plain words It is like a remote support agent who looks at your computer screen and moves the mouse for you to change settings. However, the AI can be slower than a person or sometimes misread the screen.

How it works

When a user describes a task, the AI agent takes a screenshot of the current screen and shows it to a multimodal model. The model recognizes buttons and input fields on the screen and decides 'where to click and what to type'. After carrying out that action, it takes another screenshot of the changed screen to check, and repeats this process until the task is done.

The loop of a computer-use agentUser instruction → Screen capture; Screen capture → Model decides; Model decides → Perform action; Perform action → Screen capture(Check again); Model decides → Task complete(When done)User instructionScreen captureScreenshotModel decidesChoose where to clickor typePerform actionClick · keystrokeTask completeCheck againWhen done
The loop of a computer-use agent Capture the screen, let the model decide the next action, carry it out, then check the screen again, repeating the loop until the task is done.

Why it matters

Conventional tool use required an API built in advance by developers, but computer use can handle most programs as long as there is a screen that people use. It can be used to automate legacy internal systems, website form filling and repetitive work that moves between several programs. That is why it is drawing attention as an important ability for AI agents taking on real work.

Things to know

Because it looks at the screen and moves one step at a time, it is slower than using an API, and mistakes such as clicking the wrong thing can happen along the way. There is also a risk of prompt injection, where malicious text hidden in a web page tricks the AI. For this reason, actions that are hard to undo, such as payments or deletions, are often left for a person to confirm, and the agent is often run in an isolated environment.

Example

News covered on this site includes OpenAI training and evaluating computer-use agents using contract workflows together with Ironclad. Real work that moves across many screens, such as contract review and document processing, is cited as a typical use for computer-use agents. Several AI companies are also releasing agent features that operate a browser or desktop on the user's behalf.

Where it came up on this site

Related terms

AI agent Tool use Multimodal VLM Prompt injection Coding agent