Agent Memory and Context Compaction

Three tiers, and the two people always conflate

The LLM Agents lesson gave memory two sentences: a scratchpad for the current run, a store across runs. That split is right and it is not fine-grained enough to design against, because it hides the tier that actually breaks.

Table

The usual design error is treating the middle row as the bottom row, so the transcript becomes the memory. A transcript is a log. A memory is a claim someone chose to keep.

Tier	Lifetime	Typical size	Who writes it	What it holds
Working context	One model call	Everything the model will see this turn	The orchestrator, assembled fresh each turn	The preamble, the goal, the constraints, and whatever slice of the run you chose to include
Run scratchpad	This task, then gone	Grows with every tool result	The loop, by appending	Every tool call and every result, in order, including the ones that turned out to be dead ends
Durable store	Until corrected or expired	Small on purpose	A deliberate, gated write	Decisions, preferences, and stable facts about the user or the codebase

Notice that only the middle tier grows without anybody deciding it should. The working context is assembled, so its size is a choice. The durable store is written to deliberately, so its size is a choice. The scratchpad grows because the loop ran, and that is where the trouble starts.

Context rot: it degrades before it fills

The intuition to unlearn is that a context window is a container. Full is an error you can catch; nearly full feels fine. What the measurements show is that quality falls continuously as the input grows, long before any limit is reached, and the fall produces no error at all.

Accuracy on the SAME question, varying only where the answer sits in the input

 high |  *                                                   *
      |     *                                            *
      |         *                                    *
      |             *                          *
      |                  *              *
  low |                       *  *  *
      +--------------------------------------------------------
        start                  middle                     end
                    position of the one document that answers it

Same model. Same question. Same documents. Only the POSITION moved.
Lost in the Middle reports this U shape, and reports that in the middle
of a long input a model can score below what it scores with NO documents
supplied at all.

Chroma's context-rot work extends that from position to length: across a wide set of current models, accuracy falls as input length grows even on tasks that are trivial at short lengths, the fall is not uniform, and it gets worse when the irrelevant material is semantically close to the answer. That last clause is the operationally nasty one, because the irrelevant material in an agent's context is exactly the near-miss search results it just retrieved.

So memory stops being a storage problem and becomes a curation problem. Every turn, something decides what goes into the working context. Left undesigned, that something is "everything so far", which is the one policy the measurements say is worst.

Compaction: summarize, then reinitialize

turn 41   context = [ preamble | goal + constraints | t1 t2 ... t40 ]   184k tokens
                                                                        threshold 180k crossed
                                             |
                    summarize(t1 ... t34) with a prompt YOU wrote
                                             |
                                             v
turn 42   context = [ preamble | goal + constraints | S | t35 ... t40 ]  30.6k tokens
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^   ^
                       re-attached verbatim, never     the summary,
                       part of what got summarized     about 3k tokens

  check the second number rather than trusting it: 184k over 40 turns is
  ~4.6k a turn, so six surviving turns plus a 3k summary is 3 + 6 x 4.6
  = 30.6k. compaction buys about 6x here, not the 10x the round number
  suggests, and the tail length is what decides which

Three parameters make that a design rather than a trick. The threshold decides how often you pay for a summarization call and how much rot you tolerate between them. The tail (how many recent turns survive uncompacted) decides whether the model can still see the thing it was in the middle of doing. And the summarization prompt decides what survives, which is the parameter people forget they own.

Prompt caching changes the price of that re-processing substantially and does not change its shape, and it interacts with compaction in a way worth knowing before you tune the threshold: compacting rewrites the prefix, so the cached prefix stops matching and the next turn pays full price to warm a new one. A threshold set too aggressively can spend more on cache misses than it saves on tokens.

What compaction loses, and how to bound it

A compaction step is a lossy summarization performed by the same fallible model, on a prompt that is competing with several thousand tokens of tool output for attention. Constraints go missing in summaries that read perfectly well. The operational signature is unmistakable once you have seen it: a run behaves correctly for forty turns and then, shortly after a compaction, does something it was explicitly told not to do.

So pin the invariants outside the summarizable region. The goal, the constraints, the approval limits, the tool allow-list, and anything else whose loss is unacceptable are re-attached verbatim on every turn and are never inputs to the summarizer. Everything else is fair game. That is a two-line change to how the context is assembled and it converts an unbounded failure into a bounded one: the worst a bad summary can now do is lose detail about the work, not lose the rules of the work.

Context editing: cheaper than summarizing

Compaction is not the only lever, and it is not the first one to reach for. Most of an agent's context is not reasoning; it is tool results, and a tool result has a short useful life. The agent searched a codebase, read 30,000 tokens of matches, extracted one file path, and has no further use for the other 29,900 tokens, which will nonetheless be re-sent on every remaining turn.

Context editing clears the results of old tool calls in place while keeping the record that the call happened. The model still knows it already searched for retryPolicy and what it concluded, so it does not repeat the search, and the bulk is gone. Reach for editing first because it is lossless about decisions and cheap (no extra model call), and reach for compaction when editing is no longer enough.

The durable store is a claim, not a log

What to write: decisions and their reasons, stable preferences, and facts about the world that will still be true next week. What not to write: transcripts. A conversation is not a memory, and storing one guarantees that retrieving it costs more than it returns.

When to write: at a decision, at an explicit statement by the user, and at the end of a run when the agent knows what it learned. Writing on every turn produces a store full of intermediate guesses.

How to read: this is retrieval, so everything from the retrieval material applies, including the fact that a top-k over a growing store gets less precise as the store grows. Memory retrieval has one failure of its own though. A user who gets a bad search result rephrases and searches again. An agent that fails to retrieve a memory does not know a memory existed, so the failure is silent and looks like the agent simply not knowing. That is an argument for keying what you can (memories scoped to a user, a project, a topic) rather than relying on similarity for everything.

Poisoning, staleness, and the way out

Two failures come with durability, and they are the ones a design review should ask about.

A false memory written once is read forever. If untrusted content can influence what gets written, an injection stops being a single bad turn and becomes a persistent one that reloads itself on every future run. That is why writes are a gated action, not a side effect: the same authority rules that govern a tool that spends money should govern a tool that changes what the agent believes.

A stale memory is a true fact past its expiry. "Prefers the staging database" was correct in March and is wrong in August, and nothing about it looks wrong.

The design that answers both: provenance on every memory (which run, which turn, which source, and whether a human confirmed it), a review date or TTL on anything that decays, and a correction path a user can actually reach, which means memories have to be inspectable and individually deletable rather than living as an opaque blob. When a memory is corrected, the correction wins over the original and the original is kept for audit rather than silently overwritten.

Interview nuance: the strong answer names who is allowed to write to memory and how a bad write is undone. "The agent decides" is not an answer, and neither is a retention policy. Say what a user does on the day the assistant believes something false about them.

Recap: memory is three tiers with different lifetimes and different writers; quality degrades with context length before any limit binds, so the working context is curated rather than accumulated; compaction summarizes and reinitializes at a threshold you set with a prompt you own, and it loses whatever you did not pin outside it; context editing clears stale tool results more cheaply; and the durable store holds gated, provenanced, expiring claims with a correction path, because a false memory is permanent and a poisoned one reloads every run.

Sources: Effective context engineering for AI agents · Context Rot · Lost in the Middle · Context editing