These are my notes from studying and implementing evals across my own agents. I've heard the term thrown around in ML circles for a while, but it wasn't until I started building agents that handle real, autonomous work that I understood why they matter. This isn't a finished framework - it's what I'm learning works in practice as I build.
Your AI agent is probably broken. You just don't know it yet.
We build agents that write emails, handle customer questions, manage inventory, generate reports. They produce outputs that look fine. So we assume they work. We run a few examples, check if the outputs "feel right," and ship. It's vibe-based development - and it works until it doesn't.
Then you start measuring. And you find the bugs you never would have caught by reading outputs.
This is about evals, graders, and why measuring your agents is the most important thing most people skip.
Why this matters now
Over the past few weeks I've been fine-tuning my memory systems and agents. I have multiple agents running, and the goal is for them to live autonomously - handling work without me checking every output.
But autonomy without measurement is just hope.
Every time I update an agent - to use context better, to handle more cases, to be more efficient with resources - I need to know: did this change actually make it better? Or did I break something that was working?
That's the role of evals. And the more autonomous your agents become, the more critical evals get.
What are evals?
Evals are tests for AI agent behavior. Not "does the code run" - but "does the agent actually do what it should?"
A developer writes unit tests for code. An agent builder writes evals for behavior.
The difference: code is deterministic - same input, same output, every time. Agents aren't. The same prompt can produce different responses, different tool calls, different quality. That's why traditional testing doesn't work here, and why evals exist.
Think of it this way: if you hired a new employee to answer customer emails, you wouldn't just check if they replied. You'd check if they replied correctly, in the right tone, to the right person, within a reasonable time. Evals do that for your agents.
Every eval has three parts: what you're testing (your test cases and expected results), how you run it (manually or automated), and how you grade it (the rules that decide pass or fail).
Two categories that matter:
Building your test case library
A test case is one example of what the agent will encounter in the real world. You don't need hundreds. You need the right ones.
A good library covers four types:
Start with 20-50 test cases sourced from real failures and real usage. Not hypothetical scenarios - actual inputs that caused problems. Your bug tracker and conversation logs are the best source material.
What are graders?
So you have your test cases. You run them. But how does the eval actually know if the agent got it right?
That's the grader. It's the answer key - the rule that decides pass or fail.
There are three types:
| Type | Strengths | Weaknesses |
|---|---|---|
| Code-based | Fast, cheap, objective, same result every time | Brittle to valid variations, can't judge nuance |
| Model-based (LLM judge) | Flexible, captures nuance, handles subjective quality | Non-deterministic, more expensive, needs calibration |
| Human | Gold standard for quality, expert judgment | Slow, expensive, needs domain expertise |
Code-based
Fast, cheap, objective, same result every time
Brittle to valid variations, can't judge nuance
Model-based (LLM judge)
Flexible, captures nuance, handles subjective quality
Non-deterministic, more expensive, needs calibration
Human
Gold standard for quality, expert judgment
Slow, expensive, needs domain expertise
Most teams start with code-based and model-based. Human graders calibrate the other two.
Here are the grader patterns I've been working with:
| Grader | What it checks | Example |
|---|---|---|
| String Check (exact_match) | Is the output exactly what you expected? | "Route this to sales" - did it say "sales"? Pass. |
| Set Match | Are ALL expected items present? | Extract 5 entities from a contract. Got all 5? |
| Action Taken | Did the agent DO it, not just SAY it? | It said it sent the email. Did the API actually fire? |
| LLM Judge | A second AI scores the output using a rubric | Does this customer reply sound empathetic, not robotic? |
| Text Similarity | How close is the output to a reference? | Compare a generated summary to a human-written one. |
| Composite | Multiple graders combined into one check | Email was sent (action) AND tone was right (judge). |
| Efficiency | Was it fast enough? Cheap enough? | Did it answer in under 2,000 tokens, not 15,000? |
String Check (exact_match)
Is the output exactly what you expected?
"Route this to sales" - did it say "sales"? Pass.
Set Match
Are ALL expected items present?
Extract 5 entities from a contract. Got all 5?
Action Taken
Did the agent DO it, not just SAY it?
It said it sent the email. Did the API actually fire?
LLM Judge
A second AI scores the output using a rubric
Does this customer reply sound empathetic, not robotic?
Text Similarity
How close is the output to a reference?
Compare a generated summary to a human-written one.
Composite
Multiple graders combined into one check
Email was sent (action) AND tone was right (judge).
Efficiency
Was it fast enough? Cheap enough?
Did it answer in under 2,000 tokens, not 15,000?
One distinction that changed how I think about grading:
Output vs Outcome.
Output is what the agent says it did. Outcome is what actually happened.
"I sent the email" is an output. The email arriving in the inbox is the outcome. Your grader should check the outcome when possible - not trust the agent's self-report.
Creating your own graders
A grader is just a question and a way to check the answer. If you can define what "right" looks like, you can build a grader for it.
You notice your agent sometimes responds in English when the user wrote in French. No standard grader catches this. So you create one:
Your eval suite just got smarter because you defined what matters to your specific use case.
More examples:
| Grader | What it checks |
|---|---|
| brand_voice | Does the output sound like your brand? Rubric with your specific tone rules. |
| no_hallucination | Did the agent only reference real data? Cross-check claims against the source. |
| tool_order | Did the agent search before creating? Check the trace: was lookup called before write? |
| cost_guard | Did this task stay under budget? Total tokens x price per token < threshold. |
brand_voice
Does the output sound like your brand? Rubric with your specific tone rules.
no_hallucination
Did the agent only reference real data? Cross-check claims against the source.
tool_order
Did the agent search before creating? Check the trace: was lookup called before write?
cost_guard
Did this task stay under budget? Total tokens x price per token < threshold.
Standard graders cover 80% of what you need. Custom graders cover the 20% that makes your agent yours.
Deterministic vs Non-deterministic
If these terms are new to you, here's the simplest way to think about it:
Deterministic means there's one right answer and you can check it automatically - like checking if 2+2 equals 4. Did the agent save the file? Yes or no. Did it pick the right category? Exact match. Same question, same answer, every time. No opinion involved.
Non-deterministic means the answer can vary and you need judgment to evaluate it - like grading an essay. Does the response sound natural? Is the recommendation useful? Did it handle the edge case well? These need a rubric and a judge, because "good" isn't a single fixed answer.
You need both. Deterministic evals catch broken behavior. Non-deterministic evals catch degraded quality.
When your evals are non-deterministic, run them multiple times:
An agent that passes 4 out of 5 times is very different from one that passes 5 out of 5. Both metrics matter. Know which one you're measuring.
Hill climbing: the improvement loop
Hill climbing is a term from optimization. The idea is simple: you're standing somewhere on a hill, but there's fog everywhere. You can't see the top. All you can do is feel the ground under your feet and take a step in whichever direction goes up.
In agent development, the "ground" is your eval score. The "step" is a change you make. The "fog" is the fact that you can't predict how a change will affect everything.
This is where evals become powerful.
Run your eval suite. Get a score. See what failed. Fix one thing. Run again.
Each cycle, the score goes up - or it doesn't, and you revert the change. That's one step on the hill. You don't need to see the top. You just need to know if this step went up or down.
The key is "fix ONE thing." Not five. Not a rewrite. One change, one measurement. If the score went up, keep it. If it went down, undo it and try something else. This is how you avoid breaking things that work while you improve things that don't.
Anthropic shared their own numbers from a production agent: 62% to 83% after restructuring. 83% to 92% after fine-tuning. Not one big leap - many small steps, each one measured.
Without evals, you make changes and hope they help. With evals, you know.
Where to start
You don't need a big eval suite on day one. Start with three:
- One exact_match eval for your most common task - does it route, label, or categorize correctly?
- One action_taken eval for the most important side effect - does it actually save, send, or create what it should?
- One llm_judge eval for output quality - does it match the tone and format you want?
Those three cover the basics: is the answer right, did the action happen, and is the quality acceptable.
Run those three after every change. When you find gaps, add more tests. Every bug you fix becomes a new regression test. That's the natural growth of an eval suite - driven by real failures, not theoretical coverage.
Here's what the growth looks like in practice:
- Week 1: 3 evals. You catch the obvious stuff.
- Week 3: 10 evals. You've added regression tests for bugs you found.
- Month 2: 20 evals. You start adding capability tests for new features.
- Month 3: You notice your agents are more reliable than they've ever been - and you can prove it with numbers.
As your agents take on more work and run more autonomously, your eval suite grows with them. It's the only way to trust what you can't watch.
References and further reading
These are the resources I've been studying while building this into my own workflow:
- Demystifying Evals for AI Agents - Anthropic's deep dive into eval types, grader design, and the pass@k/pass^k framework
- Agent Decomposition Workshop - Anthropic's hands-on workshop showing how to go from 62% to 92% with evals and skills
- An Introduction to Evals - Vercel's practical guide to building eval systems with dataset, runner, and scorer
- OpenAI Graders Guide - OpenAI's grader taxonomy: string check, text similarity, score model, python, and multi-graders
- OpenAI Trace Grading - How to grade the full trace of an agent's decisions, not just the final output
*These are notes from my own learning and implementation process. I'm actively building this into my workflow and will share what works and what doesn't as I go. If you're building agents and want to compare notes, I'd love to hear what you're measuring.*