AI Evals for Beginners: How to Test an AI Feature

Quick answer: AI evals for beginners come down to one idea: an eval is a test that gives an AI an input and scores its output. Define what success means, collect a small set of realistic test cases, grade them with code where possible and a model or human where needed, then rerun the set on every change.

If you are adding an AI feature to an app, you need a way to know whether it works. Ordinary unit tests expect the same output every time. AI output varies. Evals are how you test anyway.

This post reshares official guidance from Anthropic and OpenAI in plain English. It is part of our AI software development lifecycle (SDLC) series.

What is an AI eval?

Anthropic's engineering post Demystifying evals for AI agents defines an eval as a test for an AI system: give it an input, then apply grading logic to the output to measure success.

OpenAI's Evaluation best practices guide describes evals as structured tests that measure performance despite the variability of generative AI.

A few terms from Anthropic's post are worth learning early:

• Task: one test, with defined inputs and success criteria.
• Trial: one attempt at a task. You often run several, because results vary.
• Grader: the logic that scores some part of the result.
• Transcript: the full record of a trial, including tool calls and intermediate steps.
• Outcome: the real end state. Anthropic's example: for a booking agent, what matters is whether the reservation exists in the database, not what the agent says it did.

Testing agents is harder than testing a single reply. Agents make many tool calls and change things as they go, so mistakes can build on each other.

How do you define success for an AI feature?

Both companies start here. OpenAI's guide lists "define the objective and success criteria" as step one. Anthropic's docs page Define success criteria and build evaluations says good criteria are:

Criterion	Meaning	Example
Specific	Say exactly what "good" is	"Accurate sentiment labels", not "good performance"
Measurable	Use a number or a clear scale	Accuracy, response time, a 1–5 tone score
Achievable	Base targets on evidence	Prior experiments or expert knowledge
Relevant	Match the product's purpose	Citation accuracy matters more in medical apps than casual chat

Anthropic also notes that most features need several criteria at once, such as accuracy, tone, consistency, privacy, speed and cost.

A bonus from Anthropic's agent post: writing eval tasks tests your requirements too. If you cannot write a task two experts would grade the same way, the requirement is probably too vague.

How do you build a test set?

A test set (or dataset) is your list of inputs, plus what a good result looks like.

Start small and real. Anthropic suggests around 20–50 simple tasks drawn from real failures is enough to begin with. Early on, changes tend to have big effects, so a small sample shows them. If you already check things by hand, turn those checks into tasks. If you are in production, turn bug reports and support issues into test cases, starting with the ones that hurt users most.

Mirror real use. Anthropic's docs say to match the real mix of tasks and include edge cases: irrelevant input, overly long input, harmful requests, and ambiguous cases where even people might disagree. OpenAI's guide warns against datasets that do not reflect real production traffic, and recommends logging during development so you can mine logs for realistic cases.

Test both sides. Anthropic stresses balanced sets. Check when a behaviour should happen and when it should not. Its example: if you only test whether a search agent searches, you may end up with an agent that searches for everything.

Make tasks solvable. Anthropic recommends a reference solution for each task, to prove it can be passed and that the grader works. If a strong model fails a task across many trials, suspect the task first.

Keep trials separate. Leftover files or shared state from one trial can skew the next.

Prefer volume you can grade automatically. Anthropic's docs say many automatically graded cases with slightly less signal tend to beat a few hand-graded ones.

What kinds of graders are there?

Anthropic describes three types. OpenAI's guide covers the same ground with slightly different names.

Grader	How it works	Strengths	Weaknesses
Code	Exact match, string checks, running tests, checking the final state	Fast, cheap, objective, repeatable	Brittle with valid variations; poor at nuance
Model ("LLM-as-a-judge")	A model scores against a rubric or compares two answers	Flexible, scales, handles open-ended output	Not deterministic; costs more; needs checking against humans
Human	Experts or reviewers score samples	Highest quality; used to calibrate the others	Slow and expensive

Anthropic's advice: use code graders where you can, model graders where you must, and humans for occasional checks.

Tips for model graders from the two sources:

• Anthropic's docs examples use a different model to grade than the one that produced the output, and ask for tightly limited answers such as "yes" or "no" or a single number, then check the grader's reply is valid.
• OpenAI's guide says pass/fail and pairwise comparisons are more reliable than open-ended scoring, and warns of position bias (favouring the first answer) and verbosity bias (favouring longer answers). It recommends clear rubrics.
• Both say to calibrate automated graders against human judgement.

Grade outcomes, not paths. Anthropic warns against checking for a fixed sequence of steps, because agents often find valid routes you did not expect. Give partial credit for multi-part tasks, and design graders so they cannot be passed by a shortcut.

Capability evals vs regression evals

Anthropic separates two jobs:

• Capability evals ask "what can it do well?" They should start with a low pass rate, giving you something to climb.
• Regression evals ask "does it still do what it used to?" They should pass almost every time. A drop means something broke.

When a capability eval passes reliably, it can "graduate" into the regression suite. OpenAI's guide makes the matching point about continuous evaluation: run evals on every change and keep growing the set.

Anthropic also explains two ways to count runs. pass@k is the chance of at least one success in k tries, which fits cases where one working answer is enough. pass^k is the chance that all k tries succeed, which fits customer-facing features where consistency matters.

Read the transcripts

Scores alone can mislead. Anthropic recommends reading transcripts from many trials to see whether a failure is a real mistake or a grader rejecting a valid answer. It also notes that when an eval is nearly always passed ("saturated"), it still catches regressions but no longer shows progress, so you need harder tasks.

What mistakes should beginners avoid?

OpenAI's guide lists anti-patterns. In plain terms:

• "Vibe-based" testing, where you try a few prompts and decide it feels fine.
• Leaving evals until after you ship.
• Relying only on generic academic metrics.
• Using test data that does not look like real traffic.
• Ignoring human feedback when designing automated scores.

Anthropic adds that evals are one layer among several, alongside production monitoring, A/B tests, user feedback and human review. It compares this to the Swiss cheese model: each layer catches some of what slips through the others.

A starter eval checklist

Our own condensed version of both companies' advice:

1. Write 2–4 success criteria that are specific and measurable.
2. Collect 20–50 real cases: past failures, bug reports, manual checks.
3. Add edge cases and "should not happen" cases.
4. Write a reference answer or expected outcome for each.
5. Grade with code first; add a model grader with a tight rubric only where code cannot judge.
6. Hand-check a sample of model grades against your own judgement.
7. Run each case more than once if output varies.
8. Read failing transcripts before changing prompts.
9. Rerun the whole set on every prompt, model or tool change.
10. Move reliably passing cases into a regression suite and keep adding new failures.

What we do at Pocket Grove

In our studio, Cong Le maintains around 53 iOS and Android apps using Claude Code and Codex. Our evals are less formal than Anthropic's, and we would not present them as a model to copy. What we do have is headless automated tests that agents run themselves, and a human approval gate before any store release. Agent reports of "done" are treated as claims to check, not proof.

We have also started applying the same thinking to agent instructions: if a rule matters, it should have a check. We wrote about that in your CLAUDE.md needs tests too.

Sources

• Demystifying evals for AI agents — Anthropic — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• Define success criteria and build evaluations — Anthropic (Claude Developer Platform docs) — https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
• Evaluation best practices — OpenAI — https://developers.openai.com/api/docs/guides/evaluation-best-practices