Aanchal Mittal

← Architectures

Context engineering · the context pile

Here's the thing most people don't know about AI agents: the model has no memory between steps. Every single turn, the agent sends it everything again, its instructions, every file it opened, every error it hit, and the model reads the whole pile from the top just to decide one small next move. Picture it as a pile of blocks. The bigger the pile, the more each turn costs and the worse the model gets at finding what matters. So the real skill in building agents isn't the model, it's deciding what goes in the pile. Try it below.

Why those habits work

Three short ideas explain the whole race. The explorer below has one tab for each, with the same pile and the same bug. Click any block to see what it is.

1 · It re-reads everything

Step through the first turn slowly. The agent sends everything, the model reads it all, decides on one small instruction, and the agent's software (not the model) actually runs it. The result becomes a new block, and next turn the whole pile goes again. Providers soften the blow by reusing the reading they already did last turn, those are the frosty blocks, but only from the front and only on an exact match. Claude Code's docs on this put it plainly: "a change anywhere in the prefix recomputes everything after it."

Worth being precise about where things live, because "memory" gets used for three different things. The pile itself, the whole conversation, lives with the agent, on your machine. Kimi's API docs say it flat out: "The Kimi API is stateless and does not retain conversation history," so the agent has to send everything back each time, and the open-source Kimi Code keeps every session's message history in a folder in your home directory. Instruction files (AGENTS.md, CLAUDE.md) and the agent's own memory files also live with the agent. The only thing on the model's side is the cache, a processed copy of the start of the pile that makes the re-reading cheap. It's a shortcut, not memory. Lose it and nothing is forgotten, it just costs more. Even OpenAI, which can store the conversation on its side for you, says in its docs that when you chain requests that way, all the previous input tokens are still billed as input. Which is a fancy way of saying the model still re-reads the whole pile.

Let it run and by turn 60 the pile is past 400K tokens, roughly a thousand pages. It still fits, current Claude models can take 1M tokens per Anthropic's docs, but the same page admits models get worse at recalling things as the pile grows. They call it context rot. Fitting and working are different things.

2 · Cleaning up forgets things

The obvious fix is to shrink the pile once it gets long, either by throwing out old tool outputs or by having the agent write a summary of everything so far and swapping the old blocks for it. Both work, and both have a cost. The summary is the big one. On turn 9 you tell the agent never to touch tests/, on turn 52 it decides the easiest fix is changing the test, and if that one sentence didn't make it into the summary, nothing stops it.

That's not a made-up failure. The Governance Decay paper measured rule violations going from 0% with the rule in full view to 30% after summarizing, and 59% for some models. Agents also go back and re-fetch what got dropped: another paper found GPT-5.5 made about three times as many lookups (21.0 to 63.9) at 5× compression on a 24-turn planning task, while its success rate didn't change significantly. So the agent looks fine and is quietly doing triple the work. And summarizing after every single turn, which this paper tested, kept the pile tiny but made most models worse at finishing.

To be fair to summaries, they're not all bad. OpenHands, an open-source agent, keeps the first and most recent messages word for word and summarizes the middle, and its docs report up to 2× lower cost per turn with equal or better results on coding tasks. The problem is less summarizing and more trusting the summary with things that must not get lost.

3 · Keep the pile small to begin with

Which is why well-built agents mostly don't let the pile grow. Anthropic's guide to context engineering describes the three habits from the race. ✂️ Read only what's needed: search first, open the relevant lines, keep file paths instead of whole files. Claude Code builds this into its tools, a file too big to read in one go comes back as a first page, and huge command output gets saved to a file with just a preview in the pile.

🔀 Send a helper. A subagent explores on its own fresh pile and hands back a short summary, which the guide puts at often 1,000 to 2,000 tokens. Claude Code's subagents and Qwen Code's subagents both work this way, and so does the open-source Kimi Code, which runs its helpers "in isolated contexts while keeping the main conversation clean." It isn't free, the helper's reading still costs money, and the race counts it.

And 📒 keep a memory file. This is the same idea as the "memory" features everyone's shipping, just used inside one task: the agent writes what it learned to a file on disk and reads it back when the pile gets cut. Anthropic's memory tool docs pair it with summarizing for exactly this reason, and Claude Code's auto memory does the same thing across sessions with a MEMORY.md of notes from previous ones. Nothing about it is model-specific, it's files plus a harness, so it works the same with GLM, Kimi or Qwen. The real payoff shows up the next day, which is what 🌙 Day 2 in the race is for: a fresh session on a new bug in the same code. The memory tool docs describe the pattern as each new session reading the memory files first, which "restores the project state without re-exploring the code base or retracing earlier decisions." On day 1 the memory file barely changes the bill. On day 2 it's the difference between reading a few pages of notes and re-reading the codebase. (Older advice said Claude Code keeps a to-do list for this. That's historical now, its task tools are off by default on newer models.)

In this run, all three together cost about 45% less than the typical agent, never needed a summary, and never forgot the rule. Each habit does a different job though, which is the fun part of the race: the memory file alone saves the rule but doesn't touch the bill, reading less cuts the bill the most, the helper stops the re-reading. Token sizes in the scenario are illustrative, the mechanisms aren't.

For engineers · where the shortcut lives

The pile and the memory files stay with the agent either way. What moves is the cache. On a hosted API it's the provider's prompt cache and you pay per token. It's automatic on every API we checked: Kimi's API caches repeated prefixes on its own with a 5-minute default (1 hour optional) and reports cached_tokens in the response, and Z.ai's GLM API does implicit caching reported the same way. Anthropic's prompt caching bills cache reads at 0.1× the base input price and 5-minute cache writes at 1.25× (2× for the 1-hour option), with a 5-minute default lifetime refreshed on every hit. OpenAI's prompt caching kicks in automatically past 1,024 tokens, with a 30-minute default on GPT-5.6 and later. Switching models starts a fresh cache, since each model has its own.

On open weights the cache is GPU memory. Sebastian Raschka's explainer puts Qwen3 8B at about 144 KiB per token, so 32,768 tokens is roughly 4.5 GiB. vLLM's prefix caching and SGLang's RadixAttention keep your prefix around, but under pressure vLLM evicts the least recently used blocks, and an agent waiting on a tool call is exactly that. The CacheScout paper makes the same complaint about recency-based eviction. Newer open models attack the size problem directly: the GLM-5.3-Flash card (MIT, 320B parameters with 18B active, 300K context) describes a hybrid of sparse and linear attention for "sharply reducing long-context serving costs." Summaries and helpers run on your GPU too, and Qwen Code ships /compress and /compress-fast as commands.

Anyway, the model reads the whole pile every turn no matter what. Might as well care what's on it.

Sources