AI agent memory: how long-term memory is built

How an AI agent's long-term memory is built: a file tree of notes instead of a database, reached through read, grep and list, with a ranked manifest on top and a nightly distillation of finished sessions. Three non-obvious decisions: a note keyed by topic rather than by session; every fact checked for entailment against its transcript and dropped when in doubt; stale knowledge faded by half-life instead of deleted. Plus a section almost nobody writes: why an agent's own memory is a delivery channel for injections.

In short. Our agent's long-term memory is built as an environment rather than a database: a file tree of notes the agent walks with the same actions it uses on any other files — read, read_many, grep, list. Once a day a background job distils finished sessions into that tree. Three decisions in this design are worth naming separately, because none of them is obvious: a note is keyed by topic rather than by session; every fact is checked for entailment against its transcript and is not written when in doubt; nothing is deleted — stale knowledge fades in the ranking by half-life. Plus one section that rarely comes up in discussions of agent memory at all: why writing into its own memory turns a one-turn quotation into a standing instruction.

The problem: the tenth session should be smarter than the first

A person works with an agent for months and hundreds of sessions. By the tenth session the system must be noticeably more useful than it was in the first — it should know what the project is, who is involved, which formats the person prefers, how the last analysis ended. If every conversation starts from zero, that is not a colleague; it is a search box with good manners.

The problem has two halves. Knowing a fact is not enough — the agent has to know that the fact exists at all and how to reach it. Having reached it, the agent has to know whether it is current and where it came from. A lone fact with no source and no neighbours is worth little: it can be neither verified nor updated.

The two obvious answers are both bad. Dumping every memory file into the context trades precision for recall: the window fills with steadily less relevant material, and trimming it degrades the answer. Putting everything in an external database is flexible, but pushes navigation onto the agent: a vector store returns disconnected fragments, and a graph has to be queried competently to give anything back.

Memory as an environment

We hand memory to the agent in the shape it already knows how to work with: a tree of files. There is no bespoke interface to memory, because any such interface is one more language the model has to learn and will make mistakes in.

There are four actions and all of them are file actions: read for a whole note, read_many to pull several in one turn, grep to search by content, list to see what exists at all. On top of the tree sits a manifest — a compact, ranked list of what is known. It answers the first of the two questions, that the fact exists, and without it grep is useless: you cannot search for something you do not know is there.

Edges between notes are computed deterministically rather than written by the model: an edge by shared skill, an edge by shared topic, an edge by proximity. A model placing links in prose fails exactly where models fail in general — on the rare and important cases; arithmetic over skills and topics never fails and never guesses either. We chose the second deliberately: the memory graph is navigation, not content, and in navigation predictability is worth more than inventiveness.

Nightly distillation

Synthesis is not the same thing as conversation, and it has no place inside a user's turn. Once a day, at 04:00 UTC, the job fans out per organisation and works through the sessions that have accumulated: which of what was said is a durable fact, which topic it belongs to, what it supersedes in the existing notes.

The result is written as an atomic batch: either the whole set of edits lands or none of it does. Half-applied memory is worse than empty memory — its notes refer to things that are not there.

None of this deletes notes. Distillation rewrites and adds; pruning is a separate conversation, and we will come back to it.

Three decisions we made deliberately

	How it works here	What it buys
Write key	the topic, not the session: two conversations about one thing merge into one note	the store holds a picture of what is known, not a log of when it was learned
Check before writing	an entailment gate against the transcript; when in doubt, do not write	an error does not survive into next month as the agent's "own knowledge"
Stale knowledge	never deleted: it fades in the ranking by half-life	a note is not judged on a signal it could not have earned

A note is keyed by topic, not by session. This is a property of the key itself, not the result of later tidying: two conversations about the same thing land in one note and merge. The difference shows over distance — a log grows linearly with the number of conversations, a picture grows with the number of topics. After six months of work that is the difference between memory you can search and an archive you can drown in.

A fact does not enter memory unless it follows from its source. A separate lightweight judge checks the claim against the captured transcript, never refetching live data, because what must be verified is exactly the material the claim came out of. If it is not supported, or the judge's reply cannot be parsed, the claim is dropped: the gate is closed by default. The price is honest — some true facts are lost. We think the trade is right, because an error in long-term memory lives for months and is rewritten into new answers as if it were the agent's own, at which point nobody can tell it apart from what the person said themselves.

We delete nothing. A note that stops being read is not pruned after a month; it sinks in the manifest ranking by half-life. The reason is mechanical rather than sentimental: deletion judges a note on a signal it could not possibly have earned. A note is only read if the manifest offered it, and every note we used to prune had zero reads. That was a sentence for not having been shown.

Memory as a laundering channel for injections

Plenty gets written about memory architecture. Almost nothing gets written about the fact that writing into an agent's own memory turns a one-turn quotation into a standing instruction — and that is the most unpleasant property any agent memory has.

The mechanism is this. Everything an agent reads from a web page, someone else's MCP server or someone else's CRM arrives for one turn and arrives fenced: it is data, not commands. But the moment the agent writes what it learned into its own memory file, the same text comes back unfenced, in the stable prefix, under a directive that calls it the agent's own memory. No attacker is required for that; an honest summary of a hostile page is enough.

So the project memory file passes the same screen the project rules file passes, and a filtered verdict is deliberately distinguishable from an empty file: if the body was withheld, the render has to say so. Otherwise "keep this file current" plus "the file is empty" adds up to an invitation to overwrite real memory with nothing.

What comes next

Four items, in descending order of effect per unit of work:

The memory index in the first message. The manifest already exists; the question is whether it arrives in the starting context instead of on request. As long as the agent has to think to go and fetch the list, part of memory does not exist for it.
Citations as edges. Notes are verified against their source on the way in, but the link to that source does not live in the note's body as something to navigate. A guarantee at write time and the ability to re-check a month later are different things.
A staged tree for the nightly distillation. We already write in an atomic batch, but the set of edits is assembled one note at a time. Updating the graph as a whole is more correct, especially when edits are interdependent: a new note and its rewritten neighbour should land as one change.
A paired evaluation. The most inconvenient item and the most necessary one: we have no number that says what our memory is worth — enabled versus disabled, on the same questions. Until that number exists, memory work is done by taste.

Honestly about the limits

We quote no numbers of our own here, and that is not modesty: a paired "memory on / memory off" evaluation does not exist yet, and every other number we have is about load and volume rather than usefulness. Everything above is a description of an architecture and the reasoning behind its decisions, not a promise of percentages. That is exactly why item 4 is last by effort and first by importance.

FAQ

How is this different from RAG? Retrieval returns chunks that resemble the query. A note is an artifact compiled offline: written in advance, bound to a topic and to a source, linked to its neighbours, and outliving many sessions. Retrieval answers "what resembles the question", memory answers "what is known about this topic". Neither replaces the other — the raw material behind the notes still has to be found.

Why files rather than a database? Because the agent already knows files. Any bespoke interface to memory is one more language the model has to learn and will make mistakes in. It already knows grep.

Why update memory at night rather than in the conversation? Because a good note requires going across several sessions, comparing them, discarding contradictions and rewriting what is already there. A user's turn has neither the time nor the context for that, and the result has to outlive the turn.

What happens when a fact changes — say, the person changes jobs? The note is rewritten, not appended to. Otherwise both versions sit in memory as alternatives, and the agent picks between them at random.

Can an agent see someone else's memory? No. Distillation fans out per organisation, and the note tree belongs to the organisation rather than to a shared pile.