Deep dives · 2026-10-22 · Advanced
Agent memory: how agents remember beyond a single conversation
Agent memory lets an LLM agent store experiences and facts outside its context window and retrieve them when needed. Recent research has moved beyond what to store. It now also covers how to divide and organize memory, when to recall it, when to forget it, and the privacy and security risks that come with it.
Contents
- Why it was needed: memory that vanishes when the conversation ends
- Core idea: a diary, an encyclopedia and a recipe notebook
- How it works: record → organize → recall → update
- How the field developed: from what to remember to how to organize it
- Where things are heading: from retrieval to lifecycle management
- Limitations and open issues: risks that grow the longer an agent remembers
- Further study
Watch & listen
- An LLM sees only what is inside its context window, so it forgets earlier content once a session ends. Agent memory stores that content outside the model and feeds it back in when needed.
- If memory is kept in a single store and searched only by semantic similarity, temporal, causal and entity information get tangled together. More designs therefore divide memory into episodic, semantic and procedural memory, or into multiple relation graphs.
- When successful Trajectories are organized into procedural memory, performance improves even when that memory is transferred to other models, especially smaller ones.
- REM (2610.27377) showed that the bottleneck in reusing experience is not retrieval but management: deciding when to store, recall and forget.
- The longer an agent remembers, the greater the risk of exposing personal information and of contamination by untrustworthy information. Mechanisms are needed to record sources and permissions and to control how the agent responds.
Why it was needed: memory that vanishes when the conversation ends
Each time an LLM is called, it sees only what has entered its context window (the amount of text a model can read at once). When a conversation grows past the window or a new session starts, the earlier content no longer exists for the model. As a result, users have to re-explain today the preferences they shared yesterday, and the model struggles from scratch with problems it solved last time.
This limit becomes more pronounced when several agents work together. Intrinsic Memory Agents paper points out that, because of context window limits, multi-agent systems face fundamental difficulties with memory consistency, keeping to their assigned roles, and following procedures through to the end.
A common solution is to split past conversations into chunks, embed them (convert text into numeric vectors), and then find similar chunks and feed them back in. MAGMA paper argues that searching a single large store only by semantic similarity tangles temporal, causal and entity information, and that the retrieved evidence drifts away from the intent of the question. GAM paper highlights a tension on another side. Memory that accumulates conversations as a stream is easy to update but is swayed by transient noise. Structured memory preserves knowledge stably but struggles to follow a story as it changes.
Core idea: a diary, an encyclopedia and a recipe notebook
It is easy to understand if you think about how people work. They write down what happened and when in a diary, facts that rarely change in an organized encyclopedia, and how to get things done in a recipe notebook, each kept separately. The broad direction of agent memory research is similar: organize memories by type and retrieve from the part that fits the question.
Agent Zero Memory paper argues that relying on a single structure also means inheriting that structure's blind spots, so it organizes the same history in three ways at once: an episodic timeline that captures what changed and when, an entity-event knowledge graph that links people and projects across sessions, and a semantic memory of long-lasting facts refined together with their sources.
The counterpart of the recipe notebook is procedural memory. Memp paper observes that an agent's procedural memory is fragile because it is either hand-crafted by people or locked inside the parameters. It therefore Distills past Trajectories (records of the actions an agent actually took) into step-by-step instructions and higher-level scripts and stores them. In the end, the key is not storing a lot. What matters is how memory is divided and organized, and when it is retrieved.
Three kinds of memory
Episodic memoryWhat happened when
- Records events in time order
- Tracks what changed
Semantic memoryLong-lasting facts
- Graph linking people and projects
- Facts refined with sources
Procedural memoryHow to do the work
- Trajectories into step-by-step instructions
- Summarized into higher-level scripts
공통 All are kept outside the context window and retrieved when needed
How it works: record → organize → recall → update
The first step is recording. GAM paper clearly separates recording from organizing. Ongoing conversations are accumulated separately in an event progression graph and merged into a long-term topic network only when the conversation topic changes. The design aims to preserve long-term consistency while keeping moment-to-moment noise from disturbing knowledge that should last. Agent Zero Memory paper attaches a source, a timestamp and an evidence location to every item it records.
Next comes recall. Agent Zero Memory first goes through an intent check. If the question can be answered on its own, the system does not search memory, so no latency is added. If memory is needed, a source router decides where to look, and search loops run in parallel over each of the three memories, using hybrid search (combining embedding search and keyword search) as a tool. MAGMA paper treats recall as a policy-guided traversal across multiple relation graphs. Because it chooses paths to fit the question, it is easy to trace why a given piece of evidence was retrieved.
The last step is use and update. Agent Zero Memory uses a "citation lock" that allows citing only evidence that was actually opened. This structurally prevents fabricated content from getting in, and when there is no evidence, the system withholds an answer instead of guessing. Memp paper has procedures for continuously updating memory contents whenever new experience arrives, correcting what is wrong and discarding what is outdated, so the store evolves along with experience.
How the field developed: from what to remember to how to organize it
Research in 2025 focused on defining what kinds of memory are needed. Memp paper showed on TravelPlanner and ALFWorld that as the procedural memory store is refined, success rates and efficiency on similar tasks rise steadily. It also confirmed that procedural memory built with a stronger model substantially improves performance when transferred to a weaker model. Intrinsic Memory Agents paper gave each agent a memory tailored to its role and proposed a general-purpose memory template that does not require writing new prompts for each problem. It achieved state-of-the-art or near state-of-the-art performance on PDDL, FEVER and ALFWorld, with the highest consistency.
In 2026, memory structures became more sophisticated. MAGMA paper represents each memory across four independent graphs (semantic, temporal, causal and entity) and outperformed existing agent memory systems on LoCoMo and LongMemEval. GAM paper used a hierarchical graph that separates recording from organizing, along with retrieval that considers multiple factors together, to improve both accuracy and efficiency on LoCoMo and LongDialQA. Agent Zero Memory paper added source tracking and citation lock on top of this, putting "trustworthy memory" front and center.
Research on transferring memory between models has also appeared. Agent Memory Distillation paper, without any additional training, builds three kinds of memory from the successful Trajectories of a large Teacher agent (GPT-5-mini): Workflow memory, which holds task strategies; Subtask memory, which holds mid-level action examples; and Function memory, which holds per-function calling rules and common mistakes. Workflow and Subtask memory are inserted up front when a task starts, while Function memory is retrieved only when a tool-call error occurs. On Student models of 4B to 8B parameters, it showed average accuracy gains of 27.2 percentage points on AppWorld, 11.2 percentage points on BFCL V3 and 3.4 percentage points on ToolSandbox. The largest contribution came from Subtask memory, and the 4B model benefited the most.
Where things are heading: from retrieval to lifecycle management
The focus of recent research is shifting from "finding the most similar past well" to "when to put memories in and when to take them out." REM paper points out that existing memory systems have focused only on retrieval. Retrieval cannot see the state of a running agent, so it cannot judge whether the agent needs experience right now, at what point, or for how long.
REM makes three decisions explicitly inside the agent's think-act loop. What to store: it compresses finished Trajectories into documents with a step-by-step structure, which are two orders of magnitude smaller than the original Trajectories. When to recall: these documents are inserted when a task starts, while the more expensive implementation records are inserted only when a gate inside the loop detects that the agent is struggling. What to forget: those implementation records are removed after being shown once.
On a repository-level bug-fixing benchmark (SWE-ContextBench Lite), REM retrieved relevant Trajectories more accurately than the best existing memory system (match@1 68.7% vs. 55.6%), and its final resolution rate also rose across all repeated runs (27.6% vs. 24.9%). Cost was 17% lower. When the same experience was inserted as a summary or as the raw Trajectory, these improvements did not appear, even when the correct experience had been retrieved. Removing any one of the three decisions lowered the resolution rate. Seen together with Agent Memory Distillation paper, which divided memory into "what to insert up front" and "what to retrieve when an error occurs," this suggests that injection timing is becoming a key variable in memory design.
REM bug-fixing results
Best existing memoryREM
Limitations and open issues: risks that grow the longer an agent remembers
The biggest issue is privacy. Disclosure Begets Disclosure paper simulated multi-session conversations between virtual users with 24 personal attributes and support agents. Across all four support models, using memory improved response quality, but the rate at which attributes could be correctly inferred from the accumulated records (leakage) rose as well. The gap widened as sessions went on. The cause is not an attack but a loop in the ordinary process of helping. When the agent phrases a remembered value as a premise, the user affirms it or does not bother to correct it, and as a result the agent's guess becomes a recorded value. Users have a hard time breaking this loop on their own. Correcting a value is itself a disclosure, avoiding the question pushes later predictions toward the sensitive value, and leaving a wrong value in place lets it harden. The study finds that breaking the loop at the response stage is most effective. When the agent was instructed not to state or presuppose attributes, leakage fell to a level similar to not using memory at all.
There are also security problems. SAFE-Mem paper warns that untrustworthy claims coming from external sources can later be treated as if they were authorized preferences, policies or permissions. The risk resembles indirect prompt injection. The study attaches three labels to each claim (type, authority and scope) and allows a claim to be retrieved only after matching the conditions a query requires against the claim's authority. In experiments with four LLMs on MPBench and LoCoMo, it had the lowest attack success rate among defenses that preserve legitimate memory: at least 77% lower than the strongest of those defenses. Its answer accuracy was the highest among the defenses evaluated (up to a 15-point gain).
Quality limitations also remain. As GAM paper noted, memory can be swayed by transient noise. Approaches that borrow experience from other models, such as Memp paper and Agent Memory Distillation paper, depend on the quality of the source model. Agent Memory Distillation reported that the effect depends not only on the Teacher's capability but also on how well the Teacher and Student fit each other.
Further study
To get the structure down first, it is best to start with MAGMA paper and GAM paper. They clearly show why searching a single store only by similarity falls short and what changes when memory is divided and organized into graphs. Reading Agent Zero Memory paper next shows a real system design that runs episodic, relational and semantic memory at the same time and ties answers to their sources.
If you are interested in how agents remember "how to do the work," start with Memp paper and move on to Agent Memory Distillation paper. For multi-agent settings, Intrinsic Memory Agents paper shows a role-specific memory design.
If you deal with long tasks, as with coding agents, REM paper is the closest to practice. Before adding memory to a service, we recommend reading Disclosure Begets Disclosure paper and SAFE-Mem paper together and building response behavior and memory permission management into the design.
Related terms
Agent memory Context window Context engineering RAG Knowledge graph Hybrid search Multi-agent Indirect prompt injection
References
Foundational papers
- MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026)
- GAM: Hierarchical Graph-based Agentic Memory for LLM Agents (2026)
- Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents (2026)
- Memp: Exploring Agent Procedural Memory (2025)
- Intrinsic Memory Agents: Heterogeneous Multi-Agent LLM Systems through Structured Contextual Memory (2025)
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory (2026)
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.