Context Engineering Explained: What Goes Into an AI's Memory

Quick answer: Context engineering is the work of deciding what an AI model sees each time it runs: its instructions, tools, examples, history and fetched data. Anthropic's guidance is to treat that space as a limited budget and keep only the smallest set of useful, high-signal information in it, especially for long-running agents.

If you have read about prompt engineering, context engineering is the bigger picture. This post walks through Anthropic's engineering article Effective context engineering for AI agents in plain English. It is part of our AI software development lifecycle (SDLC) series.

A quick definition first. A model's context window is the text it can "see" in one go, measured in tokens (small chunks of words). Everything an agent knows in the moment has to fit inside it.

What is context engineering, compared with prompt engineering?

Anthropic frames context engineering as the natural next step after prompt engineering.

• Prompt engineering is about writing and organising instructions so a model does a task well. It suited early, one-shot uses: ask once, get an answer.
• Context engineering is about the whole configuration of tokens the model sees during a run. That includes the system prompt, tool definitions, examples, message history and any outside data pulled in.

The article shifts the core question from "how should I word this?" to, roughly, which context setup is most likely to produce the behaviour you want.

The difference matters most for agents. Anthropic defines an agent simply as a language model using tools on its own, in a loop. Each loop adds tool results and messages. So the context is not written once; it has to be curated again on every turn.

Why does more context make an AI worse?

This is the counter-intuitive part. Anthropic describes context rot: as the number of tokens in the window grows, the model's ability to recall information from it goes down. The article says this shows up across models, though some degrade more gently than others.

Anthropic explains it with the idea of an attention budget. Every token you add uses up some of the model's limited attention. The reason is partly architectural: in a transformer (the design behind today's large language models), every token relates to every other token, so the number of relationships grows very quickly as context gets longer. Models have also seen fewer very long sequences in training.

The decline is gradual, not a cliff. But the practical lesson is clear: context is a finite resource with diminishing returns. Anthropic's guiding goal is to find the smallest set of high-signal tokens that gives the best chance of the outcome you want.

What goes into good context?

The article breaks context into a few parts. Here is the gist of each.

Part of context	What Anthropic recommends	Common mistake
System prompt	Pitch it at the "right altitude": specific enough to guide, flexible enough to let the model reason	Brittle if-else rules, or vague advice that assumes shared knowledge
Tools	Self-contained, clear in purpose, returning compact output	A bloated set of overlapping tools
Examples	A small, varied set of typical examples	Stuffing every edge case into the prompt
Retrieved data	Load what is needed, when it is needed	Dumping whole documents up front
History	Keep decisions and open issues, drop stale tool output	Letting old results pile up

System prompt "altitude"

Anthropic describes two failure modes. At one extreme, people hard-code complex logic into the prompt, which breaks easily and is hard to maintain. At the other, they give high-level guidance that is too vague to act on. The right altitude sits between them.

It also suggests organising prompts into clear sections, for example with Markdown headings or XML tags. And it makes a useful point: minimal does not mean short. Start with a minimal prompt on a capable model, see where it fails, then add instructions and examples that fix those specific failures.

Tools

Tools define what an agent can do and what information it can reach. Anthropic advises tools that are self-contained, unambiguous and token-efficient in what they return. Its test is a good one: if a human engineer could not say for sure which tool to use in a situation, the agent will not do better.

Examples

Few-shot prompting (showing the model a handful of worked examples) is still recommended. The advice is to pick a small number of diverse, canonical examples rather than trying to list every rule and edge case.

How should an agent fetch information?

Traditionally, many systems retrieved relevant documents before the model ran, often using embeddings (number-based representations used to find similar text). Anthropic says practice is moving towards agentic search, where the agent fetches information itself.

Just-in-time retrieval

The agent keeps lightweight identifiers, such as file paths, saved queries or links, and loads the real content with tools only when needed. Anthropic gives Claude Code as an example: for large data, it can use commands such as head and tail to look at part of a file instead of loading the whole thing.

Metadata helps too. Folder structure, file names and timestamps give the agent clues about what matters. Anthropic calls this progressive disclosure: the agent discovers context step by step, and each step informs the next.

The trade-off is speed. Exploring at runtime is slower, and without good guidance an agent can waste context on dead ends.

The hybrid approach

Anthropic describes a middle path: load some important information up front, and let the agent explore for the rest. Claude Code does this by loading CLAUDE.md files at the start and using search tools such as glob and grep for everything else. The article's advice is to "do the simplest thing that works".

We wrote more about this pattern in how to build huge agent knowledge bases without huge context windows.

How do agents handle tasks longer than their context window?

Some tasks run for tens of minutes or hours, such as a large code migration or a research project. Anthropic argues that bigger windows alone will not fix this, because irrelevant material still competes for attention. It offers three techniques.

1. Compaction. When a conversation nears the limit, summarise it and start a fresh window with the summary. Anthropic says Claude Code keeps things like architectural decisions, unresolved bugs and implementation details, drops redundant tool output, and keeps the most recently accessed files. Its tip for tuning a compaction prompt: first make sure nothing important is lost, then trim what is unnecessary. The lightest form is simply clearing old tool calls and results.
2. Structured note-taking. The agent writes notes to storage outside the window, such as a to-do list or a NOTES.md file, and reads them back later. Anthropic's example is Claude playing Pokémon, keeping progress tallies, maps and strategy notes over thousands of steps and picking up again after context resets.
3. Sub-agents. A lead agent holds the plan while specialised sub-agents do focused work in their own clean context windows. Each sub-agent may explore a lot, but it returns only a short, condensed summary. This keeps detailed searching separate from the lead agent's thinking.

Which technique fits which job?

Anthropic's rough guide:

• Compaction suits long back-and-forth conversations.
• Note-taking suits iterative development with clear milestones.
• Sub-agents suit research and analysis where parallel exploration pays off.

A simple context engineering checklist

This is our own condensed checklist, built from Anthropic's points.

1. Write the system prompt in clear sections, at the right altitude.
2. Start minimal; add instructions only to fix failures you have seen.
3. Keep tools few, distinct and compact in their output.
4. Use a handful of varied examples, not a rulebook.
5. Load pointers, not whole documents; fetch content just in time.
6. Put the few always-needed facts up front; let the agent search for the rest.
7. For long tasks, plan for compaction, notes or sub-agents before you hit the limit.

What we do at Pocket Grove

In our studio, Cong Le maintains around 53 iOS and Android apps with Claude Code and Codex, so we feel context limits every day. Our approach lines up with Anthropic's hybrid pattern, though we did not design it from the article.

• One shared AGENTS.md/CLAUDE.md (a real file plus a symlink) holds the rules that always apply, and it links out to detailed docs rather than containing them. We explain why in stop putting everything in CLAUDE.md.
• Each app has a TASKS.md, which works as the structured notes file: the next session reads it instead of relying on old chat history.
• Skills load detailed workflows only when a task needs them.
• Subagents handle wide searches and return summaries, so the main session stays focused.

We cannot claim this is optimal. It is simply the setup that has kept our sessions usable across many apps.

Sources

• Effective context engineering for AI agents — Anthropic — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents