What is a computer-use agent?

The key distinction is where the agent makes contact with software. Function-calling agents invoke structured API endpoints and receive machine-readable responses. The software must expose that interface first. A computer-use agent operates one level up: it reads the screen and drives the same mouse and keyboard any person would use. No API needed, no custom integration.

Each step starts with a screenshot

The agent captures a screenshot of the current screen. A vision-capable model interprets what is visible: buttons, text fields, menus, error messages. It then decides which action to take next: move the cursor to a coordinate, click, type text, press a key combination. The result is a new screenshot, and the loop continues until the task is complete or a decision point requires human input.

Nothing about the target application changes. A computer-use agent can operate a 20-year-old desktop program, a password-protected web form, or a PDF editor that has never exposed a programmatic interface.

No API means no contract

When a function-calling agent calls an endpoint, it knows the parameter shape and the response format. The interaction is defined. Computer-use agents have no such contract: every action is inferred from a visual snapshot.

The consequence shows in reliability. A function-calling agent that successfully books a meeting today can be expected to succeed on an identical request tomorrow. A computer-use agent performing the same task through a browser might encounter a changed layout, a slow-loading element, or a CAPTCHA. Any of these can cause a misclick with no error signal back to the caller. Silent failure is a genuine risk.

Best for software that resists programmatic access

Computer-use agents suit two scenarios that API-based agents cannot easily handle: operating legacy systems that expose no endpoints, and navigating portals that accept browser interactions but reject programmatic clients.

A common pattern is data migration: moving records from an old system to a new one when neither provides the right endpoints. Another is operating within procurement or compliance portals built for people, where form submissions are tied to browser sessions and cannot be replicated via API.

Still not reliable enough for unsupervised use

A 2025 survey of 87 computer-using agent systems identified persistent gaps: insufficient generalisation across tasks and environments, and limited planning capability for complex multi-step sequences. The authors characterise agents of this class as not yet mature for everyday use (Sager et al., “AI Agents for Computer Use”, arXiv:2501.16150, January 2025).

Two factors explain most of the practical shortfall. Latency accumulates because each action requires at least one model inference over a full screenshot. A 30-step task runs 30 inference passes. Cost follows the same pattern: vision inference on high-resolution images is more expensive per action than a structured API call.

Providers including Anthropic have made computer use available as a beta capability (see Anthropic’s computer use documentation), reflecting that it works for constrained, monitored tasks while remaining too fragile for high-stakes unattended operation. The binding constraint is not whether the model can identify UI elements, but whether it can do so correctly every time rather than most of the time.

Universality at the cost of predictability

A computer-use agent can operate any software a person can. That breadth is the appeal. The tradeoff is that the structured guarantees of an API disappear: an API call either succeeds or returns a predictable error, while a screen interaction can silently misfire. For tasks where occasional errors are acceptable and the alternative is months of custom integration, computer-use agents are a practical option. For operations where every action must be auditable, AI agents that drive programmatic interfaces remain the more dependable choice.