How to check model output quality instead of taking it on faith
A small set of test cases, automated graders and a regression run before every prompt change, so a fix in one place doesn't break another.
A familiar story: someone complains about a weird answer, someone tweaks the prompt, three examples look better, the change ships. A week later the complaints are about something else, and it turns out that same tweak broke a case nobody remembered. A prompt is code, just without tests.
Evals are tests for features built on a model. The word sounds grand, but it starts as a spreadsheet with thirty rows.
Build a small set
You don’t need thousands of examples. Twenty to fifty per feature is enough to start, if you pick them well:
- typical requests: what people do most often, taken from real logs, not invented;
- awkward cases: empty input, very long input, another language, typos, ambiguity;
- known failures: every user complaint you fixed becomes a test case so it can’t quietly come back.
That last one is the most valuable. The set grows on its own from real mistakes and over time covers exactly what matters to your users.
For each example, record the input and what a good answer looks like. Not necessarily a reference text; criteria are often enough.
input: "refund order 4812, wrong size arrived"
expect: category = refund; order number = 4812; polite tone; no promised timelines
How to grade
There are three ways, and they combine well.
Checks in code. Anything you can check programmatically, check programmatically: the output is valid JSON, the required field is filled, length is within bounds, no banned phrases, the order number matches the input. Fast, free and indisputable.
A model as grader. For things code can’t catch, such as “the tone is polite,” “it didn’t invent terms missing from the document,” “it answers the question asked,” you can ask another model for a verdict. A couple of rules make this work:
- ask about one criterion at a time and request a yes/no with a short reason, not a score from one to ten;
- grade the grader: take a couple dozen answers, judge them yourself and compare. If you disagree often, the grader can’t be trusted, and its prompt needs fixing just like the main one.
Human eyes. No automation replaces reading fifty random production answers once a week. It’s also where new test cases come from.
Run it before every change
The main rule: any change to the prompt, the model, the parameters or what goes into the context runs through the set before it ships. You’re not asking “is it good now?” but “what changed since the last run?”: which cases started passing and which stopped.
If even one case from the known-failures group stops passing, that’s a reason to stop, even if the average went up.
Store each run’s results next to the prompt version. A month from now, “when did it start answering like this?” becomes a search, not a guess.
About randomness
A model can answer the same input differently. So:
- use a lower temperature for evals if the feature allows it;
- run borderline cases several times and look at the pass rate, not a single attempt;
- don’t draw conclusions from a difference of one or two cases; that may be noise.
Where to start tomorrow
Open the logs, write down twenty real requests to one feature, and add a line or two about what each answer should look like. Write a script that runs them and prints the output next to the expectation. It’s an hour of work, and after that, prompt changes stop being a lottery.
The upshot
Without evals, every prompt change is a bet. With a small set and a run before each release, it’s ordinary engineering: change, check, see the difference, decide.
SUBXCAT can build an eval set for your AI features and wire the run into your release process.