What can AI agents actually do (and what they can't)?

Hand a data pipeline spec to an agent with a code interpreter and file access. It writes the script, runs it against a sample, diagnoses the error, revises, and returns a working implementation. No one directs each step.

The loop (perceive, decide, act, and check) is what distinguishes agents from systems that generate a response to a single prompt and stop. For the underlying mechanism, see what is an AI agent.

Agents chain tools that existed separately for years

Web search, code execution, file access, and external API calls have each existed for years as standalone capabilities. Agents sequence them: the output of one step becomes the input for the next.

A modern agent can:

• Fetch and extract content from any public web page
• Write and execute code in a sandboxed interpreter
• Call external APIs and services via function calling
• Read and write files in permitted directories
• Navigate a browser: click elements and follow links

A task like “research this topic, write a script to process the results, run it, and return a structured summary” now crosses four tools in a single uninterrupted run. The task categories that perform reliably in production today: web research and document summarization, code generation against a spec, data extraction from structured sources, and calendar or email management where the user has already handled authentication.

Failure concentrates at two well-defined points

Authentication. CAPTCHAs, multi-factor authentication, OAuth flows, and payment pages require human intervention. Agents reach the login page; completing it autonomously is outside what current agents do. Most frameworks work around this by providing a pre-authenticated browser session, which shifts the burden to setup rather than eliminating it.

Task length. Reliability decays exponentially with task duration. METR’s evaluation of frontier models (Kwa et al.) found that current agents achieve nearly 100% success on tasks a human finishes in under four minutes, drop to 50% on tasks of roughly 50 minutes, and fall below 10% on tasks taking over four hours (arXiv:2503.14499, 2025). Each additional step in a chain compounds the probability of an earlier error propagating forward.

There is also a security dimension. Agents that browse public web pages are exposed to hidden instructions designed to redirect their behavior, a class of attack called prompt injection. Sandboxing and permission scoping reduce the risk; no major framework has eliminated it.

Benchmark accuracy and production reliability diverge

Agent accuracy on standard benchmarks has improved substantially since 2023. Reliability, measured as task-completion consistency across many consecutive runs, has not kept pace. Princeton researchers evaluating 14 agentic models found that “recent capability gains have only yielded small improvements in reliability” and that single success metrics obscure critical operational failures (arXiv:2602.16666, 2025).

The gap matters because a production deployment is not a benchmark. A benchmark measures one run on one task. A deployed agent runs repeatedly, encounters authentication edge cases, meets web pages it was not tested on, and multiplies non-deterministic steps across a chain.

The task categories that hold across that kind of use (web research, code generation, document processing, structured data extraction) are genuinely within reach of current agents. The categories that do not yet hold (sequential workflows over authenticated systems, tasks requiring cross-session memory, anything where financial consequences attach to each decision, and operations demanding deterministic output) remain limited regardless of what single-shot evaluations report.