Agent Memory: Writable, Findable, and Correctable External State

Agent depends on context in every conversation or every reasoning step, but no matter how long the context window is extended, it cannot serve as a replacement for "attaching all history forever". The reason lies in four mutually independent limitations.

First is budget. Stuffing raw records of every interaction into the context will quickly exhaust the available token budget, leaving insufficient space for the instructions and tool results that the current task actually needs. The window length raises the upper limit, but does not change the fundamental contradiction between "infinite history" and "finite budget".

Second is expiration. A large amount of content in history becomes invalid over time: the user changed their preferences, the project switched dependency versions, a decision was revoked. Copying old records verbatim treats already invalidated information as still valid evidence.

Third is conflict. The same fact may appear in mutually contradictory versions at different points in time, for example, "the default language is Chinese" followed by "from now on always reply in English". Including everything means passing the contradiction to the model at the same time without providing any basis for adjudication.

Fourth is slowing down calls. Once irrelevant history occupies the context, the model has to process more content, latency increases, attention is diluted, and the truly critical information becomes harder to use reliably.

In addition, there is the issue of persistence across sessions. When a single conversation ends, the context disappears with it, but user preferences, completed work, and ongoing project status need to remain available in the next session, otherwise the background must be reintroduced every time.

External memory is precisely the answer to these problems: it lets the system retrieve relevant state only when needed, rather than always keeping all history in the context. It relieves capacity constraints while supporting long-term personalization and task continuity. But this capability is not free—once retrieval is introduced, it is possible to retrieve incorrect records, and the system thereby takes on the responsibility of data governance, including what to write, what to read, when to modify, and when to delete.

Therefore, external memory can be defined as: a state system that an Agent can persistently write to, retrieve on demand, and correct. Its inputs are events, facts, and preferences that exceed the current window but may still be useful in the future; its outputs are the memory evidence that the current task is permitted to use. The system maintains cross-session continuity through writes, retrieval, and updates. It must be repeatedly emphasized that retrieval "hit" only means that a record was found, and absolutely does not mean that the record is the latest, true, or that the current caller has the right to use it. Hit and correctness are two different things, and this distinction runs through the entire design of the memory system.

"Memory" is not a single store that can be treated the same, but is composed of several layers with different lifespans, sources, and purposes. Distinguishing them is meant to answer a specific question: Should current tool results and user long-term preferences live in the same place and be handled in the same way? The answer is no—their validity periods and reliability levels are completely different.

Based on what is stored, memory can be divided into five layers.

In-context history stores the messages and tool results visible in the current turn. It has the shortest scope, serving only the immediate call, and dissipates along with the context once the conversation ends. It barely needs retrieval because the content is already in the window, but it is also the least stable.

Working state stores plans, to-dos, constraints, and verified facts. It lasts longer than in-context history, spanning multiple turns or multi-step tasks, and is an intermediate product that needs continuous reference and updating during execution.

Episodic memory stores specific events with timestamps and sources. It records "when, where, and who did what," providing traceable raw material for later judgment. The reliability of episodic memory lies in that it is a record rather than an inference, but it does not by itself answer "whether it still holds now".

Semantic memory is stable facts or preferences integrated from multiple events. For example, a record that states "reply in Chinese by default" twice is promoted to "this user prefers Chinese". Semantic memory is the highest-level sedimentation, but it must be built on multiple pieces of evidence; a single event is not sufficient to support it.

Procedural rules store reusable processes, policies, and operating specifications, for example "tests must be run before deployment". It describes how to do something, not what happened or what the user likes.

The input to this layering is a candidate piece of information, along with its purpose and validity period; the output is to assign this information to one of the categories: in-context history, working state, episodic memory, semantic memory, or procedural memory. The direct benefit of classification is to help select a write policy and an eviction policy: current tool results serve the immediate task, specific events retain time and source, only mutually corroborated events can be integrated into stable preferences, and rules require explicit scope and trigger conditions.

But classification is not an automatic conclusion. In reality, information often spans layers; a record may be both an episodic event and contain clues to preferences. You cannot automatically upgrade it to a long-term rule or permanent fact merely because it has been labeled with the name of a certain layer. Promotion between layers must have evidence and governance basis; this is exactly the issue to be solved in the subsequent write and integration phases.

If storage is cheap enough, saving every sentence verbatim actually makes memory worse. The reason is not capacity, but that unselective writing systematically pollutes memory: duplicate records cause the same fact to exist in multiple copies, noise drowns out the truly important information, sensitive information is persisted in places where it should not remain, and mutually contradictory records make it impossible for the system to determine which one represents the current state during retrieval.

Therefore the core of the writing stage is not "storing it", but "deciding whether it should be stored and in what form". The write gate receives candidate content, along with its source, confidence, future utility, sensitivity, scope, and retention period, and outputs three possibilities: reject, pending confirmation, or a versioned record.

The basis for judgment is first future utility—will this information be used again in later tasks? One-off temporary instructions do not deserve to occupy Long-term Memory. Second is confidence and source: information from explicit user statements has high credibility, while information from model inference or second-hand retelling has low credibility. Third is sensitivity: content involving privacy or credentials should either not be written or be sanitized and strictly restricted in read permissions. Finally, scope and retention period determine to whom this record is visible, under what conditions it takes effect, and after how long it should expire.

A typical pitfall is directly elevating "the user requested brevity this time" into a permanent preference. A one-time request may only concern the style of this immediate reply; writing it as a long-term preference is an erroneous generalization. Important facts should be confirmed by the user, or be supported by multiple independent pieces of evidence before being elevated; this is precisely the threshold required for upgrading from Episodic Memory to Semantic Memory.

The write gate addresses three types of problems—noise, conflict, and privacy: it reduces noise by rejecting low-utility content, handles contradictions through versioning, and protects privacy through sensitivity and scope constraints. It is necessary to clarify its boundary—successful writing only indicates that this content conforms to the retention policy; it does not prove that the content itself is true, nor does it guarantee that it will be useful in the future. Writing and verification are two different things: the former manages "whether it should be kept", and the latter manages "whether what is kept is correct".

Retrieval is the intersection of the memory system and Retrieval-Augmented Generation (RAG), but it adds two factors that ordinary similarity search does not consider: time and permissions. If you only look for memories by vector similarity, you will miss the two key questions: "Does this record still hold now?" and "Can the current caller see this record?"

The input to memory retrieval includes the current task, subject permissions, time, as well as candidate memories and five types of scoring signals; the output is filtered and reranked records with sources.

The ranking can be expressed as a weighted score. For a candidate memory m, its ranking score is:

score(m) = α × Rel(m) + β × Rec(m) + γ × Imp(m) + δ × Src(m) − Penalty(m)

Here Rel is relevance, measuring how well the record matches the current task; Rec is recency, measuring how recent the record is relative to the current moment; Imp is importance, measuring the weight of this information in decisions; Src is source confidence, measuring the reliability of the information's origin. α, β, γ, δ are the corresponding weights, determining the contribution of these four signals to the ranking; Penalty is the conflict penalty, a deduction applied to relevant records when multiple records contradict each other.

The key point of this formula is that it produces a ranking score, not a truth probability. A high score only means "it is more worth being retrieved for the model's reference", and does not mean that the record is true or should be adopted unconditionally. The four weights need to be calibrated by task: some tasks favor relevance, while others must prioritize recency.

What really determines privacy and security is the position of permissions. Permissions must be hard-filtered before scoring; they must never be turned into a "negative-score penalty item" mixed into the above formula. If permissions are only deducted, a memory that is "highly relevant but belongs to another user" may still rank near the top because of extremely high relevance, and finally be read by the model, causing a privacy leak. The correct order is: first apply hard filtering by user and project scope, keeping only records that the current subject has permission to access, then score and rerank the remaining records, and finally hand them to the model.

Real systems usually do not rely solely on a single vector retrieval; instead they combine multiple methods such as semantic retrieval, keyword matching, time filtering, entity relationships, and reranking to compensate for the shortcomings of pure vector similarity in exact matching and recency expression. No matter how complex the combination is, that red line remains unchanged: scope filtering must precede all scoring, and permissions can never be replaced by soft penalties instead of hard filtering.

When new facts conflict with old facts, the system faces three choices: overwrite the old, let both coexist, or ask the user. The answer depends on the strength of the evidence, and the correct approach is to preserve the source and time, storing the "current value" separately from the "historical events" rather than physically overwriting the old records.

Integration management takes new and old records, times, sources, and user corrections as input, and outputs the current view, historical trace, conflict status, or deletion result. Its core principle is version invalidation rather than physical overwriting: when high-confidence new evidence appears, the old record can be invalidated and no longer participate in retrieval as the current value, but its past existence is never erased. The significance of this audit trail is that if the new evidence itself is later found to be erroneous, the system can still go back to the invalidated version, rather than facing a blank.

Summaries are lossy compression, which is the essential difference between them and the original records. When multiple events are compressed into a single sentence "user prefers X", details are inevitably lost. Therefore summaries must retain key constraints and link back to the original references, so that any details that have been compressed away remain traceable when needed. A summary without evidence links is merely a second-hand retelling of memory by the model, and its reliability is no higher than a hallucination.

This leads to a clear boundary: model reflection is not fact verification. The "summary memory" written by the model can still hallucinate, even if it reads fluently and plausibly. Therefore the system must retain evidence and confidence scores, so that the integration result can always be traced back to its source, rather than allowing a piece of model self-summary to automatically be upgraded to fact.

The lifecycle of memory should not be one of only growth. The system needs to support expiration, deletion, user correction, and regulatory retention. Expiration applies to records that have a defined retention period; deletion responds to explicit user requests; user correction is the highest-priority update source; and regulatory retention means that certain records must be retained in accordance with compliance requirements even if deletion is requested.

The final "current value" is only the record that is most applicable under clear rules, not the "absolutely correct fact". When conflicts cannot be adjudicated by rules, the correct approach is to let them coexist and mark the conflict status, or ask the user directly, rather than the system arbitrarily picking a version. Model summaries can never serve as fact verification; they can only propose candidates, not act as adjudicators.

Once memory is writable, it also becomes an attack surface. The most dangerous path is a malicious webpage content polluting future tasks through memory: when the agent browses a webpage or calls a tool, it writes prompt injection or error content returned by the tool into long-term memory, so this single point of pollution is persistent, and every subsequent retrieval related to that may bring malicious instructions back into context, affecting a chain of subsequent tasks.

Memory security receives external candidates, sources, sensitivity, and target scope as input, and outputs either data records that are allowed to be written or rejection decisions. Its first line of defense is source classification: external content is by default just data, not rules. Data can be retrieved and cited, but never automatically gains the status of "instructing the system how to act". Only explicitly authorized content may become rules, and this promotion cannot be triggered by external content itself.

The second line of defense is content filtering and scope restriction before writing. Candidate content must be checked before entering long-term memory to identify instructional language and suspicious patterns; at the same time, writes should be limited to the smallest scope so that any polluted content can only affect the smallest range.

The third line of defense is indicating provenance at read time. When a piece of memory is retrieved and passed to the model, it must carry a source label, letting the model know that this is "data from some webpage", not "the system's own policy". Provenance annotation gives the model an opportunity to be vigilant about suspicious content, rather than treating external text as trustworthy instructions to execute directly.

There is also an untouchable red line: permissions, system policies, and secrets must not be overwritten by ordinary memory. No matter how authoritative external content appears or how well it matches the current task, it cannot rewrite system-level rules or credentials. Such content either never enters the layer that can be affected by ordinary memory, or is forcibly isolated at retrieval time.

The effectiveness of filtering mechanisms has clear boundaries: filtering is only effective against attack patterns that have already been tested. More covert attacks such as cross-user leakage and indirect injection cannot be handled by content filtering alone and must be continuously tested through dedicated evaluation. Write filtering is a necessary line of defense, but not a sufficient security guarantee.

Stringing together the previous steps, for a memory to safely produce value, it must pass through a complete chain process, where each step has clear inputs, outputs, and governance decisions.

The first step is extraction: identify candidate facts and events from the task execution process. Of the raw content generated by the model during conversations or tool calls, only a portion is worth entering memory; this extraction step is precisely about distinguishing "possibly useful" material from "fleeting" noise.

The second step is the write decision: decide whether to write based on utility, confidence, and sensitivity. This step answers "is it worth storing, is it trustworthy enough, will it leak secrets?" Low-utility, low-confidence, or high-sensitivity content is rejected here or turned into pending confirmation.

The third step is integration and association: deduplicate content that passed the write decision, associate it with existing records, and set scope and retention period. Deduplication prevents multiple copies of the same fact; association connects new records to related entities and existing evidence; scope and retention period stipulate in advance who it will be visible to and when it will expire.

The fourth step is retrieval and reranking: retrieve relevant records according to task, permissions, and time, and sort them. Scope filtering is enforced strictly before scoring; only then are candidates sorted by combining relevance, timeliness, importance, and source confidence, and finally only the sourced records most worth consulting are handed to the model.

The fifth step is injection and use: inject sourced memory into the context to support completion of the current task. At this point, what the model sees is "a record from a certain time, place, and provider," not a piece of bare text with no provenance, which allows the model to weigh it appropriately.

The sixth step is updating and forgetting: update or delete records according to task results, user corrections, and expiration rules. When an old record is found no longer applicable during a task, users can explicitly correct it, and expiration triggers invalidation; these all return to the integration step, allowing memory to continuously converge in a closed loop.

These six steps form a repeatable cycle, and running through it all is the same set of constraints: writes are gated, retrieval has permissions, conflicts have versions, and external content is always data rather than rules. Memory is not a one-time storage-and-retrieval action, but lifecycle management in which every step on this chain carries governance decisions.

Use a concrete case to walk through the entire memory chain. The user first says "report defaults to Chinese", then later says "this project must be in English". Which sentence should the system remember? The answer is that both sentences are remembered, but in different scopes, and ultimately the more specific constraint wins in the specific scenario.

During the write phase, the first sentence, "report defaults to Chinese", forms a user-level candidate preference, sourced from message m17 with confidence 0.8. The second sentence, "this project must be in English", forms an explicit constraint at the project P level, sourced from message m42. The key difference between the two lies in scope: m17 applies to the entire user, m42 applies only to project P, and m42 is an explicit constraint with higher priority. They do not overwrite each other because their scopes differ—one is a global default, the other a local override.

In the retrieval phase, when generating a report for project P, the system first applies hard filtering by user and project permissions, then re-ranks candidates by project scope, recency, and explicitness. m42 wins because its scope is closer to the current task and it is an explicit constraint; m17, as a global default, remains valid but is overridden by the more specific rule. If the report belongs to a different project, m42 is out of scope, and m17's Chinese preference still applies.

In the usage phase, when injecting context, the output is a statement with scope and source such as "project P: English (m42)" rather than a vague "user language preference". The sourced statement lets the model clearly know that this conclusion applies only to project P and which message it is based on, so that a local constraint is not misused as a global preference.

In the correction phase, if the user then says "P should also revert to Chinese", the system writes a new event m58 and marks m42 as invalid rather than deleting history. m42 remains in the audit trail but no longer participates in retrieval as the current value. At this point the status of the three records is: m17 default Chinese (user-level, valid, overridden by a more specific rule); m42 project P must be English (project P level, valid→replaced by m58, used before replacement); m58 project P reverts to Chinese (project P level, current, used with source).

In the forgetting phase, when the user asks to delete the language preference, the system deletes the current record, derived summaries, and retrieval index, and leaves a compliance deletion credential that does not contain the original value. The original value itself is removed, but the deletion action leaves an auditable record to meet compliance requirements.

The core conclusion of this case is that m42 winning only means it is more specific to project P and better suited to the current task; it absolutely does not mean the user's global Chinese preference has been deleted or negated. Scope determines which record takes effect in which scenario, while version and source make every choice traceable and correctable.

If all you do is "vectorize and write to a database," the memory system actually completes only the easiest small step. The reason is that writing to storage itself brings no governance capability: it does not automatically have permissions, versioning, and deletion capabilities, and it does not automatically distinguish between "current value" and "historical event." Treating this step as the entirety of memory is precisely the starting point of problems in many systems.

This original figure describes the complete lifecycle of memory: candidate events first pass through a write gate, and only content that passes utility, confidence, and sensitivity judgments can enter a memory store with versions and sources; then, after retrieval constrained by permissions, time, and task, the retrieved content is used for action; feedback from the action then triggers correction or deletion in turn, forming a closed loop. The entire path can be summarized as the four elements shown in Figure 1: write control, versioned storage, read isolation, and reversible update, which together constitute the necessary conditions for trustworthy memory.

The inputs to the lifecycle diagram are candidate events and task requests, and the output is memory that has passed through the write gate, versioned storage, permission-based retrieval, and feedback correction. Each arrow in the diagram is a recordable state change, which allows it to serve as a fault-location tool: when memory goes wrong, you can follow the arrows to determine whether the error occurred at the write, store, retrieve, use, or update stage, rather than vaguely attributing the problem to "the model misremembered."

It is important to note the positioning of this diagram: it describes a governance process, not a specific database implementation. A vector database, graph database, or ordinary relational store can all host this process, but the process itself—especially permission filtering, version invalidation, and deletion—must be explicitly implemented. Merely completing vectorized writes to a database does not automatically obtain any of the gates shown in the diagram.

"lives in Shanghai" and "in Beijing next week" — why can't they overwrite each other? Because they are not the same kind of fact at all: the former is a stable current state, and the latter is a future plan. Treating a plan as the current situation, or overwriting a plan with the current state, are both wrong. Conflict detection is not about "picking the one with the highest similarity"; it is about first conducting temporal and scope reasoning.

The first step is to distinguish fact types. Stable attributes, current state, plans, one-time events, and inferences each have different validity periods and credibility. Stable attributes (such as "the user lives in Shanghai") hold over the long term; current state may change at any time; plans point to the future and have not yet occurred; a one-time event is a record of the past; inferences are candidates derived by the model from evidence with lower confidence. Because the types differ, the way conflicts are adjudicated differs.

The second step is to delineate the scope of the conflict. Only two records with the same entity, same attribute, and overlapping temporal scope constitute a direct conflict. Records with different scopes should coexist and be selected by scope at read time, rather than being forcibly merged into a single global preference. Plans and events that have already occurred belong to different fact types and should each retain their own type, rather than treating a plan as the current state.

The third step is to adjudicate the current view according to rules. Source authority, user confirmation, event time, and write time together determine which record serves as the current value: explicit user confirmation outranks inference, authoritative sources outrank hearsay, and newer events outrank older events. When a new value for the same attribute appears, the old value becomes invalid but its history is retained; audit records are never physically overwritten.

The fourth step handles cases that cannot be adjudicated. When sources are on the same level and contradict each other, the correct approach is to mark the conflict and seek clarification, rather than using vector scores to determine truth. A language model merging two contradictory records into a fluent passage based on fluency is precisely the most dangerous path, because fluency carries no truth information.

The inputs to conflict reasoning are entity, attribute, fact type, scope, validity time, and source; the output is one of four results: coexist, invalidate, select current, or request clarification. Its core constraint is that the current view is not eternal truth, but only the most applicable record under explicit rules at the current time; plans and current state, and preferences for different projects, cannot be forcibly overwritten by similarity scores. True conflict adjudication depends on structured reasoning over time and scope, not on the magnitude of vector distance.

Retrieval Recall@10 is high, so why can an Agent still be misled by old memories? Because recall is only a measure of "findable"; it does not answer "whether what is retrieved is the value that should be used now." An old memory that has already become invalid may be retrieved with high recall and then be adopted by the model as the current state. Evaluation therefore cannot only look at retrieval, but must cover the five stages of the memory lifecycle: write, find, use, correct, and forget.

Evaluation should construct cross-turn tasks offline: embed stable preferences, temporary constraints, mutually conflicting updates, sensitive information, and malicious external text in a long conversation, and then separately test whether these five things are all correct—correct writing (store what should be stored, reject what should not be stored), retrieving in appropriate tasks (relevant memory appears in needed scenarios, irrelevant memory does not appear), citing the latest effective value (after conflicting updates the current value is used rather than the old value), executing correction and deletion (user corrections and deletion requests actually take effect). Each stage should record error attribution, separating generator failures from retrieval failures, to avoid miscounting a problem of "retrieved but the model did not use it correctly" as a retrieval failure.

But correctness rates at each stage alone are not enough; the value of memory must be measured by net contribution. The net contribution V of memory relative to a no-memory baseline is defined as:

V = Success(memory) − Success(noMemory) − λ × Cost(privacyStale)

Here Success(memory) is the task success rate when memory is used, Success(noMemory) is the task success rate when memory is not used, and the difference between the two is the task gain brought by memory. Cost(privacyStale) is the cost from privacy boundary violations and stale references, and λ is the weight that converts this risk cost to a scale comparable with success rate. A positive net contribution indicates that memory is still worthwhile after deducting the cost.

Evaluation reports need to simultaneously provide end-to-end task success, incorrect memory usage rate, cross-user leakage rate, correction latency to take effect, deletion residual rate, as well as token consumption and P95 latency, and compare these results with several baselines such as "recent history only", "rolling summary", and "retrieval memory" in ablation. Security testing must also include persistent prompt injection: even if a piece of malicious web content is semantically highly relevant to the current task, it must never be escalated into a procedural rule, and this must be explicitly verified in evaluation.

The input of the five-stage evaluation is cross-turn tasks containing stable, temporary, conflicting, sensitive, and malicious records; the output is metrics for the write, retrieval, use, correction, and forgetting stages as well as end-to-end metrics. Two boundaries must be kept in mind: net contribution depends on the specific metric scale and coefficient λ and cannot be directly compared across projects; high recall also cannot be used to offset cross-user leakage—a system with perfect recall but that leaks others' memory still has a negative net contribution.