Every agent demo works. You give it a task, it plans, it calls three tools, it answers. Then you point the same agent at real work, the kind that takes sixty steps and touches forty files, and somewhere around step thirty it starts behaving like a different, worse model. It re-runs a search it already ran. It applies a constraint you mentioned once at the beginning that stopped being relevant twenty steps ago. It confidently contradicts a fact it retrieved correctly ten minutes earlier.
The reflex is to blame the model. Usually it is not the model. The model is doing exactly what you asked with exactly what you gave it, and what you gave it is a 180,000 token pile of stale tool output, abandoned plans, and half-finished reasoning.
1. A bigger window is not more memory
When a model ships with a million token window, the instinctive read is that context management just got solved. It did not. Two things stay true regardless of window size.
First, attention has to work harder as the haystack grows. Retrieving one fact from a short context is close to free. Retrieving the same fact from a context stuffed with forty similar-looking blocks means the model has to discriminate between plausible candidates, and that discrimination is where accuracy leaks. The classic needle in a haystack benchmarks look great precisely because the needle is unlike everything around it. Real agent context is the opposite: it is fifty tool outputs that all look roughly the same, and the right one differs from the wrong one by a timestamp.
Second, stale content does not sit quietly. A superseded plan from step 8 reads to the model exactly like a current plan. Nothing in the token stream is marked expired. If you tell an agent in turn one that it is working on the billing service and then pivot it to the auth service, both instructions are now in context with equal weight, and the only thing distinguishing them is position.
| What you think is happening | What is actually happening |
|---|---|
| The model forgot the instruction | The instruction is present but competing with three later ones that partially contradict it |
| The model hallucinated a file path | The path was real forty steps ago, before a tool call renamed it, and both versions are in context |
| The model is looping | The result of the earlier call is buried under 20k tokens of unrelated output, so re-calling is cheaper than finding it |
| We need a bigger window | We need to delete things |
2. Where the tokens actually go
Before optimizing anything, instrument the context. On most agent runs I have looked at, the breakdown is lopsided in a way nobody predicts. The system prompt and tool definitions are a fixed tax, often 5k to 15k tokens, and they are usually fine. The user's actual task is a few hundred tokens. Everything else, which is to say almost all of it, is tool output.
And tool output is where the waste concentrates. A file read returns 900 lines when the agent needed one function. A search returns twenty results when the agent used the first. An API call returns a JSON blob with 60 fields where three matter. Each of those is individually defensible and collectively fatal.
This is why tool design is context engineering. A tool that returns a 40 line summary with a handle for fetching the full payload on demand will beat an identical tool that returns everything, not because it is cheaper, though it is, but because the agent's fortieth decision is made against a cleaner context.
3. The memory hierarchy
Operating systems solved a version of this problem decades ago and the shape of the solution transfers. You do not keep everything in registers. You keep a tiny hot set close, put the rest somewhere addressable, and pay a small cost to fetch on demand. An agent wants the same layering.
Tier 0 is the contract. Who the agent is, what tools exist, what it must never do. This is small, stable, and never evicted. Rewriting it mid-run is how you get an agent that changes personality halfway through a task.
Tier 1 is the working set: the subtask in flight and the last handful of turns. This is the tier people over-fill. It should feel uncomfortably small. If your agent needs twelve prior turns to make the current decision, the subtask is probably too big and should be split.
Tier 2 is external memory the agent reads and writes on purpose. A notes file, a task list, a store it queries. The important property is that writing to it is an explicit action with a cost, which forces the agent to decide what is worth keeping. Automatic capture of everything just moves the pile.
Tier 3 is the one that buys the most and gets used the least. When the agent needs to search wide, spawn a sub-agent with a fresh context, let it burn 50k tokens exploring, and have it return three sentences. The parent never sees the transcript. You get the benefit of the search without paying for it in the main context forever.
4. Compaction should be lossy on purpose
Most compaction implementations try to be faithful. They summarize the conversation so far, preserve every thread, and hand back something 40% shorter. That buys one or two more rounds before you are back where you started, because the summary inherits the original's flaw: it treats everything as worth keeping.
Useful compaction is opinionated. It asks what the agent needs to finish the task, not what happened. Those are very different documents. A good compaction keeps the goal, the constraints discovered so far, decisions that are now settled, the files or entities in play, and open questions. It throws away the reasoning that led to settled decisions, tool outputs already acted on, and every dead end, except for a one line note saying the dead end was tried so nobody walks back into it.
| Keep | Drop |
|---|---|
| The goal, restated in current terms | The original phrasing if the goal has since narrowed |
| Constraints discovered during the run | How each constraint was discovered |
| Settled decisions, stated as facts | The deliberation that produced them |
| Identifiers currently in play | Full contents already read and acted on |
| One line per failed approach | The full transcript of each failure |
The tell that compaction is working: after it runs, the agent's next few steps do not repeat anything from before the compaction. The tell that it is not: the agent immediately re-reads a file it read four steps ago, which means the summary dropped a fact but kept the impression that the fact mattered.
5. Ordering matters more than it should
Position in the context is a signal the model reads whether you intend it or not. Content at the very start and the very end gets attended to more reliably than content in the middle. That has an uncomfortable implication for the standard append-only agent loop: your most important instructions drift into the weakest position as the conversation grows.
A cheap fix that consistently helps is restating the current objective and the active constraints immediately before the model's turn, not just at the top of the conversation. It costs a couple hundred tokens and it puts the thing that matters where the model will actually see it. It feels redundant. It is redundant. It also works.
The related habit is to stop treating the message history as an immutable log. It is a data structure you own. You are allowed to rewrite an old tool result into a one line summary once the agent has used it. You are allowed to delete a failed attempt entirely. Nothing about the API requires you to hand back verbatim what happened, and treating history as sacred is the single most common source of rot I run into.
6. How to tell if you have this problem
Run your agent on a long task and log the context size at every step alongside whether that step was productive. If the productivity curve bends down while the token curve goes up linearly, that is context rot and no amount of prompt tweaking will fix it.
A sharper test: take a failing run, truncate the context at the point things went wrong to just the system prompt plus the last three turns plus a hand-written summary, and re-run from there. If the agent immediately does the right thing, the model was never the bottleneck. That test has resolved more agent bugs for me than any amount of staring at prompts, and it usually takes ten minutes.
7. What I would do first
If you are shipping an agent and only have time for one change, cap tool output. Truncate anything over a few hundred lines, return a handle, let the agent fetch more if it asks. It is a small change and it removes the largest single source of junk.
If you have time for two, add sub-agent isolation for search. Wide exploration is exactly the workload whose byproducts you never want to keep, and shunting it into a disposable context is close to free.
Everything else, the tiering, the opinionated compaction, the re-stated objective, is worth doing, but those two account for most of the recovered ground. The general principle underneath all of it is unglamorous: the context window is a budget you spend, not a bucket you fill, and the agent that finishes the sixty step task is usually the one carrying the least.