Most teams ship an agent after a few good demos and find out it's broken from angry users. Knowing how to evaluate AI agents is what separates a prototype from something you can trust with real work.
Agents are harder to test than a single prompt. They plan, call tools, loop, and recover from errors. The same input can take ten different paths. You need evals built for that.
Why Agent Evaluation Is Different
Testing a chat completion is simple: input in, text out, compare. An agent is a multi-step system. A wrong answer might come from a bad plan, a wrong tool choice, a malformed argument, a misread result, or a loop that never ends.
If you only grade the final answer, you can't tell which step failed. If you only grade steps, you miss whether the job got done. You need both views.
- Outcome evals: did the agent achieve the goal?
- Trajectory evals: did it get there sensibly, with the right tools and no wasted steps?
- Safety evals: did it stay inside its limits?
Start With a Small, Real Dataset
Don't wait for a perfect benchmark. Public agent benchmarks are useful for comparing models, but they don't tell you if your agent works on your tasks.
Build your own set instead:
- Pull 20 to 50 real tasks from logs, support tickets, or your own usage.
- Include easy cases, hard cases, and weird edge cases.
- Add every production failure as a new test the day it happens.
- Write down what a correct result looks like before you run anything.
Twenty honest examples beat a thousand synthetic ones. The set grows as your agent meets reality.
The Metrics That Matter
Pick a few numbers and track them on every change. Too many metrics and nobody looks at any of them.
Task success rate
The share of tasks completed correctly. This is your headline number. Define success concretely: the file was created, the record was updated, the answer matches the reference.
Tool-call accuracy
Did the agent pick the right tool and pass valid arguments? This is often checkable with plain code. Compare the calls against an expected list, or validate arguments against a schema. No model needed.
Cost and latency per task
An agent that succeeds but burns 40 steps and a pile of tokens is a bad agent. Track steps, tokens, dollars, and wall-clock time per task, and watch the tail, not just the average.
Consistency
Run each task several times. Agents are non-deterministic, so a task that passes 6 out of 10 runs is not passing. Report pass rates, not single results.
Grading: Code, Humans, and LLM Judges
There are three ways to score a run. Use the cheapest one that works.
- Code-based graders: exact checks. Did the database row change? Does the output parse? Is the test suite green? Fast, cheap, and objective. Use these wherever you can.
- LLM-as-a-judge: a model scores open-ended output against a rubric. Good for tone, completeness, and reasoning quality. Keep rubrics specific, ask for a short justification, and use a different model than the one being tested when possible.
- Human review: slow but the ground truth. Use it to calibrate your judge and to review the failures that matter most.
Never trust an LLM judge blindly. Spot-check its scores against your own on a sample. If it disagrees with you often, fix the rubric before you trust the numbers.
Make Evals Part of the Loop
An eval that runs once is a report. An eval that runs on every change is a safety net.
- Run on every prompt, model, or tool change. Swapping a model or editing a system prompt can quietly break tasks that worked yesterday.
- Gate releases. If task success drops past a threshold, the change doesn't ship.
- Log full traces in production. Capture every step, tool call, and result so you can replay failures and turn them into tests.
- Sample live traffic. Score a slice of real runs continuously to catch drift your dataset doesn't cover.
Sandbox anything with side effects. Evals should never send real emails or touch real customer data. Use mocked tools or test environments.
Common Mistakes
- Grading only the final answer and missing broken intermediate steps.
- Testing each task once and calling the result stable.
- Tuning the prompt until the test set passes, then shipping an agent that only works on the test set.
- Ignoring cost until the invoice arrives.
- Skipping failure analysis. Read the traces. The patterns are usually obvious within an hour.
The loop is simple: build a small real dataset, score outcome and trajectory, automate it, and feed every production failure back in. Do that and your agent improves with every change instead of drifting.