Most teams think they already know what a correct answer looks like. They borrow a generic rubric, scrape the internet for examples of good, or read the agent's codebase and treat what it was built to do as the spec. That works for public tasks and young products. Inside an enterprise, it breaks.
We learned this building an eval loop for a customer-facing AI agent we worked on. On the first pass, our judge read a few hundred masked production conversations. It called some good and some bad. On a large share of them, it couldn't decide. Here's what that taught us.
01The internet gives you the questions, not the answers
One undecided conversation had the bot volunteering a specific fact about the company. Fluent and plausible. The judge's note, in effect: I can't verify this. Nobody outside the company could; someone inside could in seconds.
The usual shortcuts all miss this. When a customer asks whether their plan includes something, a generic rubric rewards a warm “plans like this usually include it.” The client's experts flag it, because the real answer depends on the customer's plan, start date and terms. The internet can show you what people commonly ask about products like this, but not what this company should say. And the codebase only tells you what the agent was built to handle. Customers don't read the codebase. They ask whatever they need, and much of it was never designed for.
- Customers
- Ask whatever they need, in their own words
- The gap
- Where domain experts define correct, case by case, with the client
- The codebase
- Shows only what the agent was built for
That gap is why we don't start from a scraped definition of correct. We work with each client to define what correct looks like, case by case. Our domain experts have spent 10+ years in the industry and can tell what correct looks like for a given question. And we bring industry data from past chatbot testing, so we know which behaviors usually need to change, and in what direction.
02Experts can't write the rubric up front. They can react.
Try a thought experiment. Pick something you know well: your job, your city, a sport you play.
Take a minute and write down what makes advice about it good. Most people get three or four points, and they're generic: be accurate, be clear, be practical.
Now imagine someone hands you one specific piece of advice about that thing, and it's subtly wrong. You'd spot the problem in seconds. And the reason you'd give probably isn't on your list.
Experts work the same way. Ask them for rules and you get the ones they've been asked about before. Show them a specific answer and the real rules come out.
So we don't ask experts to write a rubric. We build a simulation environment where they watch the agent handle realistic conversations and brain-dump what they notice, in their own words.
For each conversation we capture two things: a verdict, and the reason. The verdict is a label. The reason is the asset.
03Spend expert time only where the judge is unsure
Every verdict comes with a confidence score, and each team sets its own threshold. Above it, the verdict stands. Below it, the conversation goes to an expert, who confirms or corrects it and says why. There's no universal number; where you set the line depends on your domain and how much risk you can carry.
The target is 20 to 30 reviews every few days, aimed at intents the judge hasn't seen. Expert time is the scarcest resource in the loop, so it only goes where the judge is unsure. A judge that knows when it's wrong is worth more than one that's slightly more accurate but can't tell you where.
And the grey bucket keeps shrinking. Every expert reason teaches the judge something, so the share of conversations it can't call goes down with each round of reviews. Expert time moves on to whatever is genuinely new.
04Teach the judge in plain language
Expert reasons go straight back into the judge's instructions and examples, and we tune the weights the judge gives each criterion so its scores track what the experts care about most.
One corrected case, with its reason, covers the next hundred like it.
The experts' bar wins, even when it's stricter than ours. A team can decide that an answer which merely could have been better counts as a failure. No generic rubric sets the bar there.
05Don't close an issue until production says so
A confirmed failure moves through the same steps every time:
- A diagnosis agent reads the masked conversation, trace and code, and raises a ticket with root cause, proposed fix and test cases.
- Engineers build the fix and write evals for it, including the original conversation.
- The fix ships only if those evals and the full regression suite pass in UAT and SIT.
- The issue closes only when the judge sees the fix working on live traffic. The evals stay in regression.
Every expert lesson ends up encoded twice: in the judge, and in the test suite that gates every future release.
Why the humans don't leave (yet)
At a startup of under 100 people that started two years ago, the shortcuts mostly work. The codebase is close to the whole business logic, the people who wrote the rules are still in the room, and the data is young enough to be clean.
At an enterprise with 20+ years of history, none of that holds. Products, regulations and exceptions have piled up for decades.
Data is scattered across systems and contradicts itself. The most important rules were often never written down, or live in the heads of the people who've been there longest.
In principle, a model with clean data and a big enough context window could infer most of it. Enterprises aren't there yet.
That's also where the value is. These firms have the most customers, the most riding on each conversation, and the most knowledge locked in people.
They have the most to gain from getting this right, and the most to lose from an eval that pretends the experts aren't needed. Without them, an eval measures how closely your agent matches the internet's average: a precise-looking number for the wrong thing.
Summing it up
- Generic rubrics, the internet and your codebase tell you what's common or what was built. Clients and domain experts tell you what's correct.
- Don't ask experts for a rubric. Let them react to real conversations, and capture their reasons.
- Spend expert time only where the judge is unsure, and keep the ask small.
- Feed expert reasons back into the judge, and tune its weights to match.
- Close issues on production evidence, and keep every eval in regression.
If your judge can't decide on a large share of your traffic, that isn't a failure of the judge. It's a map of where your experts need to look.