Your CLAUDE.md Needs Tests Too

Quick answer: Test agent instructions the way you test code. Keep a small held-out set of real repository tasks with acceptance criteria, run them in clean sessions against the old and new instruction versions, and add a few semantics-preserving code variants to expose brittle behavior. Measure task success, regressions, cost, and time — not just whether the agent invoked a Skill.

Changing CLAUDE.md can feel like editing documentation. In an agent workflow it is closer to changing a shared library: the file changes what the system notices, what it does first, which tools it chooses, and how it decides that work is complete.

If that change is reviewed only by reading the diff, the team is relying on intuition about a stochastic system. A short instruction can fix one task and quietly damage another. A new Skill can trigger too often, never trigger, or produce an answer that sounds right but misses the repository's actual contract.

Start with real tasks, not toy prompts

Build a private benchmark from work the repository genuinely needs:

• a historical bug with a known regression test;
• a small feature with explicit acceptance criteria;
• a refactor where architectural boundaries matter;
• a release or migration task with a checkable artifact;
• a task that should be refused or escalated because it lacks authorization.

Store the task prompt, the relevant commit, the expected evidence, and the result. The commit matters because an agent can appear to improve while the repository, dependencies, or test fixtures changed underneath it.

Run each case in a fresh session with the instruction change and again with the baseline. A warm conversation can hide a missing rule because the model remembers the rule from authoring or from a previous attempt.

Test the instruction contract

There are at least two separate questions:

1. Does the agent invoke the Skill or instruction on prompts where it should?
2. When it does, does the resulting work meet the repository's acceptance criteria?

The Claude Code documentation's evaluation guidance makes the same distinction. Triggering is not proof of usefulness. Compare realistic prompts in fresh sessions with the Skill available and unavailable, then inspect the output and the token/time overhead.

Add negative cases too. A deployment Skill should not activate for a spelling correction. A read-only reviewer should not silently gain permission to edit. An instruction that passes only when the user phrases the request in one exact way is not a reliable contract.

Add a metamorphic layer

Most task suites keep the repository representation fixed. That leaves a blind spot: an agent may succeed because it memorised a filename, identifier, or code shape rather than because it understood the behavior.

For important cases, create semantics-preserving variants. Rename an identifier. Reorder equivalent control flow. Add harmless dead code. Change formatting without changing behavior. Then run the same task and compare the result.

A 2026 study, A Jagged Frontier, found that code agents usually showed small degradation under these transformations, but the most affected configurations dropped by up to 6.7 percentage points and statistically significant degradation appeared in 6 of 16 model/scaffold/dataset configurations. The result is not a universal failure rate. It is evidence that robustness can depend on the harness as well as the model.

Turn recurring failures into knowledge carefully

When a benchmark finds the same repository-specific mistake repeatedly, distil the fix into a Skill or reference. Then send it back through the benchmark. Do not promote every successful patch into permanent instructions; first check that the rule generalises and does not create a new failure elsewhere.

SkillForge describes a related research direction: proactively synthesize project-specific issues from test-covered functionality, resolve them, and distil reusable, entity-grounded knowledge for future issue resolution. That supports a useful engineering loop, even if your implementation is much smaller: failure, root cause, candidate instruction, held-out test, promotion.

A small CI gate is enough to start

You do not need a massive evaluation platform. A first version can run on every instruction change and report:

• baseline and candidate pass rates;
• regressions by task and by repository area;
• negative-trigger violations;
• metamorphic consistency;
• wall-clock time and input-token cost;
• the exact instruction commit and repository commit used.

Set a promotion rule before the next change arrives. For example: no held-out regression, no permission-boundary violation, and no unexplained cost increase above the team's limit. If a change fails, keep the failure as a fixture rather than polishing the prompt until the benchmark disappears.

That is the difference between an agent memory system and a pile of confident prose. The instructions become observable, versioned engineering artifacts — and your context stays small enough for the agent to use them.

Read the companion pieces on keeping CLAUDE.md small and building a large knowledge base without a large context window.