A wrong identifier or a misread date is an error a reader can spot and correct in a single LLM exchange. In an agent loop, the same error becomes the basis for the next action.
Where the false output comes from
A large language model predicts the most statistically likely next token given the text before it. There is no fact-checking step and no look-up against a ground truth.
When a query sits inside the model’s reliable training distribution, the most plausible continuation tends to be correct. Obscure technical details or events past the training cutoff: the model still produces confident text, because confident text is what the training process rewarded.
Farquhar and colleagues demonstrated in a study published in Nature that the reliable signal for whether a response is a hallucination is semantic uncertainty: when a model’s outputs vary in meaning across samples, it is working in territory it cannot reliably navigate (Farquhar et al., 2024). The model has no way to flag this in its output. A high-entropy internal state produces the same confident text as a certain one.
The US National Institute of Standards and Technology calls this confabulation: the confident generation of “erroneous or false content” by a generative AI system (NIST AI 600-1, July 2024).
Why agents compound the problem
In a chat conversation, a hallucinated answer lands once. The reader can push back, look it up, or discard it.
An agent uses its own outputs as context for each subsequent step. A false assumption at step two gets appended to the context and treated as ground truth when the agent reasons at step three. The error does not persist as a visible fact. It propagates as a premise, shaping each downstream decision until something external surfaces it.
An early hallucinated value or misread tool response can silently constrain the next five steps, and tracing back through an agent’s chain of reasoning is harder than reading a single response.
What reduces hallucination risk
Grounding the generation. Tool use and retrieval anchor the model in actual content rather than training memory. When an agent calls a real API or reads a document at each reasoning step, it generates from something verifiable. Retrieval-augmented generation applies the same principle: documents placed in context replace reliance on training memory alone. Farquhar et al.’s semantic entropy finding adds a complementary approach: a system can monitor its own uncertainty and route high-entropy responses to a retrieval call or a human checkpoint rather than presenting them as confident outputs.
Human oversight at consequential steps. Human-in-the-loop design works here as it does elsewhere: a person placed at the decisions where errors would be hardest to undo catches mistakes before they compound forward. The tighter the checkpoint, the lower the residual risk. That trade is set per decision, not once for the whole system.
What remains true regardless
No combination of grounding and oversight removes hallucination entirely. An agent grounded in real tools can still misread what they return. A human reviewing on the loop does not catch what they do not see. NIST AI 600-1 treats ongoing human oversight not as a phase-out plan but as a permanent condition of responsible deployment.
Hallucination in an agent is a managed risk. The design decisions about where to place human gates and which steps rely on model generation alone determine the residual exposure. An agent that hallucinates early and compounds across later steps is harder to audit than one that produces a single wrong final answer, because the intermediate error is less visible and the downstream actions have already run.
Sources
- Farquhar, S. et al. Detecting hallucinations in large language models using semantic entropy. Nature, 630, 625-630. 2024.
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). July 2024.
Questions, answered
What causes an AI agent to hallucinate?
The language model it runs on was trained to predict plausible text, not to retrieve verified facts. When a query falls outside reliable training data or the model is genuinely uncertain, it produces the most plausible continuation, which may be false. In an agent loop, that false output can then become the basis for the next action.
Does giving an agent tools or search reduce hallucinations?
It reduces knowledge-based hallucination significantly. Tool use and retrieval anchor the model's generation in actual content rather than training memory. However, an agent can still misread what a tool returns or hallucinate about how to use the result, so grounding reduces but does not remove the risk.
Why do AI agents hallucinate more than a single LLM conversation?
In a single exchange a hallucination lands once and stops. An agent uses its own outputs as context for subsequent steps, so a false premise at step two can be treated as ground truth at step five. The error propagates forward rather than stopping at one response.
Can hallucinations be fully eliminated from an AI agent?
No. Retrieval, tool use, human checkpoints, and uncertainty detection reduce the risk substantially, but none removes it entirely. NIST AI 600-1 (2024) treats ongoing human oversight as a permanent component of responsible generative AI deployment, not a phase-out goal.
Brief is a team of AI associates you direct in plain language. Opening to a small group at a time.
Request access