How does an AI agent remember?

At any step during a task, an agent reasons from one source: the text currently in its context window. When that window closes, those contents are gone unless something was written out first. That gap between what the model sees right now and what should persist over time is what agent memory design exists to close.

The context window is not storage

The context window is a live working space measured in tokens: the model reads it on every step and reasons from whatever is there. It is temporary by nature. Modern models have extended these windows considerably, but a hard upper bound remains, and a long-running agent can exhaust it mid-task.

When that happens, older content must be dropped or compressed to make room for new input. That is not forgetting as metaphor. It is a literal loss of accessible state, unless the dropped content was written somewhere external before it left the window.

Long-term memory lives outside the model

For information to survive beyond a single context window, an agent writes to an external store. Retrieval is how records re-enter the context window when they become relevant. The most common mechanism is semantic search over dense vector embeddings: the agent issues a query, the store returns closely matching records, and those records enter the context for the current step. That retrieval layer is the same one behind RAG; agent memory uses it as one component of a larger write-manage-read cycle.

An AI agent decides when to write and when to retrieve. Those decisions happen at runtime, not at design time. The agent manages its own memory rather than reading from a fixed knowledge base a developer pre-populated.

What gets written, and how

Writing to external memory is itself a reasoning step. Which records to keep and how to label them are design choices encoded in the agent’s instructions or delegated to a separate memory-management pass.

Du et al.’s 2026 survey of LLM agent memory formalises this as a write-manage-read loop tightly coupled with the agent’s perception and action cycle. The management layer, deciding what to retain and what to discard, is where most of the design complexity sits, and where current systems show the largest gaps when evaluated on multi-session tasks.

A third kind most explanations skip

Context and external storage dominate discussion, but there is a third kind of memory already present in every language model: parametric memory. It is the knowledge encoded in the model’s weights during training. The model carries it before any session starts, without retrieving anything.

Parametric memory is always available at zero retrieval cost. It is also completely static. A fact the model was never trained on, or one it learned incorrectly, cannot be corrected by writing to a database during a session. There is no runtime write path into the weights. External storage compensates for this: facts absent from parametric memory must be made available through retrieval.

Sources

• Du, P. Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. arXiv:2603.07670, 8 March 2026.
• Packer, C. et al. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 12 October 2023.