kangwijen
Back to posts

Your AI Doesn't Need 1 Million Context. It Needs to Forget Better.

kangwijen

10 min read

For the past few weeks I've been building a tool-calling agent meant to stay on a task for a long time. The model behind it advertises a million tokens of context, and for a while I treated that number like a promise. Long runs filled the prompt with internal reasoning, tool results, and failed attempts. Past a certain point, the agent stopped acting like itself. Its answers got less precise, it repeated completed steps, and it lost track of the task. I eventually gave the model a much smaller working budget on purpose.

A marketed context window is a capacity claim. It is not a quality claim. More transcript in the prompt is not free intelligence, and in practice it can make the agent worse. This sits next to AI Can Write the Code. You Still Have to Guide the App.. That post was about drawing the product box. This one is about drawing the memory box while the loop is already moving.

Bigger windows still forget the middle

Research papers have looked beyond "can a model accept a huge prompt?" A popular demo called needle in a haystack hides a made-up fact inside a long document and asks the model to repeat it. For example, bury "the passcode is orange-kiwi" in hundreds of pages and then ask for the passcode. Models can score well on that demo, especially when the question repeats the same words as the hidden sentence.

That is not how a real agent run works. It may need to connect an old constraint, a tool error, a file it read earlier, and the user's latest instruction. Those details can use different words and sit far apart. Different papers test different parts of that failure:

  • Lost in the Middle (opens in a new tab) shows a U-shaped pattern: models often use facts at the start or end of a long prompt better than the same facts placed in the middle.
  • RULER (opens in a new tab) shows that near-perfect needle-in-a-haystack scores do not mean a model can handle harder long-context work. Almost every model they tested claimed at least 32k context, but only about half stayed above their quality bar at 32k.
  • NoLiMa (opens in a new tab) removes the shared wording between the question and the hidden fact. At under 1k tokens the scores look fine. At 32k, 11 of 13 models claiming 128k+ context fall below half of their short-context baseline. Even GPT-4o drops from about 99.3% to 69.7% in that setup.
  • LongMemEval (opens in a new tab) tests chat assistants on multi-session memory: recalling facts buried in realistic user-AI dialogues, not one long document. On histories of about 115k tokens, strong long-context models still drop 30% to 60% compared with reading only the evidence sessions. Dedicated memory products show similar gaps when tested the same way.
  • Hidden in the Haystack (opens in a new tab) shows another wrinkle: when the relevant passage itself is short, models miss it more often, even after controlling for position and distractor count. That matters for agents that must stitch together small tool outputs, error lines, and constraints scattered across a long run.

Coding work shows the same pressure. On LongCodeBench (opens in a new tab), Claude 3.5 Sonnet's success rate on LongSWE-Bench falls from 29% at 32k tokens to 3% at 256k. On LongCodeQA, Qwen2.5 peaks at 70.2% at 512k and falls to 40.0% at 1M. Another paper, Context Length Alone Hurts LLM Performance Despite Perfect Retrieval (opens in a new tab), finds drops of 13.9% to 85% even when the model can already fetch every relevant token. The length of the prompt itself becomes part of the problem.

Chroma's Context Rot report (opens in a new tab) and Anthropic's context engineering notes (opens in a new tab) describe the same effect in plain language: as the prompt grows, the model has a harder time using the right detail at the right time. Old tool output, abandoned attempts, and outdated instructions start to hide the part of the task that matters now. A larger context window lets the agent receive more text, but it does not make old information easier to find or more useful. Pasting the whole transcript is a bad default, even when the API accepts it.

You see the same answer across the industry. Model vendors, coding agents, and open-source frameworks have converged on a small set of patterns, even though the names differ.

How the industry handles it

Model APIs. Anthropic treats compaction as the first lever in context engineering (opens in a new tab). When input tokens cross a threshold, the API summarizes older content into a compaction block and continues from there. Its server-side compaction docs (opens in a new tab) support custom summarization instructions and pause_after_compaction, which lets you re-pin standing rules after a summary. A related primitive, tool-result clearing (opens in a new tab), drops bulky old tool payloads while keeping the record that the call happened.

OpenAI built a similar path into the Responses API (opens in a new tab). Set a compact_threshold and the server compacts during the response stream, or call /responses/compact explicitly when you want compaction at a phase boundary instead of mid-loop.

Product tools. Claude Code (opens in a new tab) exposes this to users as /compact, /clear, and subagents with fresh context. Its guidance treats context rot as normal even with a 1M window. Cursor (opens in a new tab) offers /summarize to compact on demand and uses subagents (opens in a new tab) so search and shell work stay in separate context windows. Community reports suggest background compaction kicks in before the hard limit, though the exact threshold is server-side rather than user-configurable.

Open-source frameworks. LangGraph's memory docs (opens in a new tab) recommend two layers: keep the full history for the UI, and send the model a trimmed or summarized working set. LangMem's SummarizationNode (opens in a new tab) replaces older messages with a summary once a token threshold is hit; trim_messages can keep only the last N tokens or messages. Microsoft AutoGen's BufferedChatCompletionContext (opens in a new tab) is simpler still: retain only the most recent buffer_size messages to avoid overflow. The Vercel AI SDK compaction cookbook (opens in a new tab) applies the same idea inside tool loops by returning a shorter messages array from prepareStep before each model call.

Google ADK. Google's Agent Development Kit (opens in a new tab) ships first-party compaction through EventsCompactionConfig. You can trigger on token volume (token_threshold plus event_retention_size for a raw recent tail) or on turn count (compaction_interval plus overlap_size so each summary includes overlap from the prior window). Same shape again: summarize the middle, keep recent events intact.

There is no formal standard yet. What you get is convergent design.

What good forgetting looks like

The pattern above is simple enough to copy without tying yourself to one vendor:

  1. Count the tokens that will actually be sent to the model.
  2. Before reaching the hard limit, replace older history with a summary. Leave enough space for the next reply and for writing that summary.
  3. Let the application do this automatically. Do not rely on the agent remembering to summarize itself.
  4. Keep the visible chat intact, but send the model a shorter version of the old history.
  5. Let the agent continue its work after the summary is made.
  6. Stop only when this keeps failing because a large new result fills the prompt again and again.

A useful prompt usually contains:

text
original task
+ short summary of the middle
+ recent turns (including the latest tools)

Keep the first user request when you can; it often holds the real limits of the task. Summarize the noisy middle and leave the newest messages unchanged so the model can follow the current work.

Show the full transcript in the chat UI even when the model receives something shorter. Someone may need to inspect a tool error, check an earlier finding, or understand why the agent made a decision. If the UI does not explain that split, a user can see an old result, assume the agent still has it, and wonder why it forgot.

Clear before you summarize

Compaction only helps if the summary is worth keeping. Summaries leave details out by design, so do not feed the summarizer data you can load again later.

Save large details outside the prompt when you can load them again: old tool results, base64 data, large HTML pages, scraped text, and requests that have already changed a file. Leave a short note and a file path or ID. Keep a fresh result only while the model needs it. Most context problems I have seen came from tool output, not from carefully written plans.

Then summarize what remains. A useful summary lets the agent pick up the work later:

If your summary only looks like the first example, do not use it for important work yet. Ask it to keep the goal, limits, completed work, next steps, IDs, and errors. Leave out the step-by-step story.

Practical rules that held up

These rules held up after watching long runs fail in different ways.

  1. Choose a realistic working budget: The number in a model advertisement is not automatically a useful limit. Leave space for the next reply and the summary.
  2. Watch token count, not only user messages: One user request followed by fifty tool calls is still a long-context problem.
  3. Summarize while the agent is working: Waiting for the next user message gives a tool loop too much time to fill the prompt.
  4. Keep the original request and the newest messages: Do not leave the only copy of an important rule inside a summary.
  5. Save summary progress on the server: If the browser sends the full chat history every time, the application may summarize the same messages again and again.
  6. Keep tool requests with their results: If you remove old history, do not leave a tool request with no answer.
  7. Load standing instructions again after a summary: Store them in a project file, because summaries can lose early instructions.
  8. Stop after repeated failures: If one large result keeps filling the prompt after several summaries, end the turn cleanly instead of looping forever.

Governance Decay (opens in a new tab) shows why rules 4 and 7 are not optional polish: summaries tuned for task continuity can drop standing constraints unless the application re-injects them.

Do this before the model reaches its hard limit. Quality can fall long before the provider returns an error.

References

Categories

Tags