Denshin / Blog / AI

How to evaluate AI agents: evals that catch real failures

It looked good in the demo is not a test. How to build an eval set from real traffic and bug reports, the four types of eval and what each one catches, why agents need trajectory evaluation and not just final-answer scoring, and how to upgrade a model without gambling.

Denshin Engineering · Engineering Team · 27 August 2026 · 8 min read

Every AI feature demos well. You picked the examples, you knew the phrasing that worked, and you quietly retried the one that went sideways. Then it meets real users, who paste in a half-finished sentence with an order number from a different system, and the thing confidently does the wrong job. Evals are how you stop finding this out from customers. This post is about the kind that catch real failures in agents, not the kind that produce a number for a slide.

What an AI eval actually is

An eval is a repeatable test that runs your AI system over a fixed set of inputs and scores the outputs against a definition of correct. It is a test suite, with the awkward property that the system under test is non-deterministic, so a single pass or fail on one input tells you very little and you are always reasoning about rates. That is the whole idea. Everything else is detail about how you get the inputs, how you score, and what you do with the number.

Two things follow immediately. First, you cannot write evals until you can state what "correct" means for your task, which is why the definition of done should be settled before you build, as we argued in agentic AI explained: what actually works in production. Second, an eval that always passes is not reassuring, it is uninformative. A good eval set contains cases you currently fail.

Why the demo lies

Demo inputs are clean, in-distribution and chosen by someone who knows the system. Real inputs are truncated, contradictory, in three languages, contain an ID that was deleted last week, and occasionally contain instructions aimed at the model. The gap between those two populations is where the failures live, and no amount of staring at a happy path closes it.

The other reason is subtler: you remember the successes. Without a fixed set and a recorded score, "it seems better since we changed the prompt" is a feeling. Half the changes we have made to prompts that felt like clear improvements turned out to be neutral or slightly worse once we could measure them, and we would not have known either way without a set to run against.

Building an eval set from real traffic

Do not write your eval cases from imagination. Mine them:

Real requests from logs. Sample across the whole distribution, not just the median, and include the boring ones so you notice regressions on easy cases.
Every bug report. This is the highest-value source you have. When someone says "it did the wrong thing here", that case goes into the set with the correct behaviour recorded, permanently. That is a regression test, and it is how the suite earns its keep.
Incidents and near misses. Anything that made someone say "how did it even do that".
Adversarial and malformed input. Empty strings, enormous inputs, wrong-language input, content that contains instructions. Defensive only: you are checking the system refuses or ignores, not building an attack kit.

Strip or pseudonymise personal data before anything lands in a test fixture that will sit in your repository and get copied to laptops. Store each case as input plus expectation plus a note on why it is in the set, because in six months nobody will remember why case 17 matters.

Four kinds of eval, and what each one catches

Type	How it scores	Catches	Watch out for
Golden answer	Exact or near match to a known-correct output	Regressions on well-defined tasks	Only works where one right answer exists
Assertion based	Code checks on structure, fields, tool calls, side effects	Malformed output, wrong tool, missing write	Passes on output that is valid but useless
Rubric or LLM as judge	A model scores against written criteria	Tone, completeness, helpfulness	Judge bias, drift, cost, false comfort
Regression suite	Reruns past failures, pinned to a model version	Old bugs coming back after a change	Silently rotting if nobody adds to it

Assertion based evals are underrated

For anything with structure, plain code is the best judge you have: it is free, fast, deterministic and never flatters you. If a step must produce JSON matching a schema, validate it. If a refund task must end with exactly one refund of the right amount, assert it against the database. If the answer must cite a document, check the citation exists and that the quoted text is actually in it.

def check_refund_task(trace, db):
    assert trace.called("get_invoice"), "must read the invoice before refunding"
    assert trace.count("refund_invoice") == 1, "exactly one refund, no retries"
    assert db.invoice("inv_123").status == "refunded"
    assert trace.turns <= 8, "took too many turns"

LLM as judge, with your eyes open

For open-ended output you need a judge, and a model with a written rubric is the practical option. It is genuinely useful, and it has known weaknesses: judges tend to favour longer and more confident answers, they can favour output that looks like their own style, and their scores drift when the judge model changes underneath you. Mitigations that work in practice: write the rubric as concrete yes or no criteria rather than a 1 to 10 feel, show the judge the reference answer where one exists, pin the judge model and version, and spot check a sample of judgements against human review every time you change anything about the judge. Never let the judge be the same model instance that produced the answer.

Trajectory evaluation: judge the path, not just the destination

For agents, final-answer scoring hides too much. An agent can reach the right answer having called a destructive tool twice, burned forty turns, or guessed after its search failed. Next week the guess will be wrong and the score will not have warned you.

So evaluate the trajectory as well. Useful checks, none of which need a model:

Did it call the required tools, and in a sane order (read before write, verify before commit)?
Did it call anything it should not have, especially destructive tools on a read-only task?
How many turns, tokens and rupees, and how does that compare with the previous run of the same case?
Did it recover from the injected failure? Run some cases with a tool stubbed to error and check the agent reports the problem rather than inventing around it.
Did it stop cleanly, or hit a budget limit?

Silent partial completion is the failure trajectory evals exist to catch: the run says success, the assertion on the side effect says only three of five records were updated. Your trace has to record every tool call, its arguments, its result and its cost for any of this to be possible, which is one more reason to design the loop the way we describe in AI agent architecture: tools, memory and the loop.

Offline evals and online monitoring do different jobs

Offline evals answer "is the new version better than the old one on cases we already know about". They run in CI, on a fixed set, before you ship. Online monitoring answers "what is happening to real users right now", and it catches the inputs your set has never seen.

Both are needed, and the online side is mostly ordinary observability: trace every run, record cost and latency per task with percentiles rather than averages, track the rate of budget-limit hits, tool error rates, and the rate at which users retry or escalate to a human. A rising retry rate is a quality signal available without any labelling at all.

Upgrading a model without gambling

Model upgrades are the moment evals pay for themselves. Pin the model version in config, never float it. When a new one appears, run the full offline set on both versions and diff the results case by case, because the aggregate score can stay flat while the failures move to different, worse cases. Then canary: send a slice of real traffic, watch cost, latency and your online signals, and be able to roll back with a config change. Prompts that survive this kind of transition tend to share a few properties, which we cover in prompt engineering best practices that survive model upgrades. The same discipline applies when you are choosing between models in the first place, which is a decision framework, not a leaderboard, as we argue in choosing an LLM.

How small can a useful eval set be

Smaller than people expect. Twenty to fifty real cases, curated with care, will catch most of what breaks in a small product feature, and a set that size runs in minutes and gets used. A set of two thousand synthetic cases costs a fortune to run, takes an afternoon to interpret, and nobody looks at it after the second week.

Keeping it honest is the harder discipline:

Add every real bug as a case, the same day, with the expected behaviour written down.
Keep a holdout you do not tune against, so you can tell tuning from overfitting.
Run the set on a schedule, not only when you remember, and treat a drop as a broken build.
Delete cases that no longer represent anything real, deliberately and with a note, rather than letting them rot.
Report pass rates per category, not one aggregate number. The aggregate hides exactly the thing you need to see.

What to do next

Pull twenty real inputs from your logs this week and write the expected behaviour for each.
Write assertion-based checks first, on structure, tool calls and side effects, before reaching for a judge model.
Record trajectories, not just final answers, and assert on turns, tools and cost.
Pin your model version, and require an eval diff before any upgrade ships.
Make every bug report a permanent case.

If you have an AI feature in production and no way to tell whether last week's prompt change helped, that is a fixable problem and usually a small piece of work. Talk to us and we will help you get a first honest eval set in place.

Tags: AI Evals, AI Agents, LLM, AI Testing, AI Engineering

All posts · Work with Denshin