How to tell if it actually got better

You reword a prompt, run it once, read what comes back, and it seems better. That is not a result. The fix has a name — evals — and the version worth having is smaller and duller than the word makes it sound.

Published 25 August 2026 · Last verified 25 August 2026 · One outside source, read that day. No figures on this page

Why "it seemed better" isn't a result#

There is a page here on getting better results, and it is a list of things to change. This is the page about how you would know that changing them worked, which is a separate job, and the one people skip.

The loop most people run: something comes back badly, you rewrite the prompt, you run it again, the new answer reads better, you keep the change. Three things are wrong with that, and they compound.

One run is not a sample. Ask the same question twice and you get two different answers, which is exactly the property that makes asking twice a useful check on a claim. Here that same variation is what ruins your evidence. The second answer might be better because the prompt is better, or because this run happened to go well.

You chose the case, and you chose it badly. The input you re-ran is the one that just failed, which is the least representative input you own. A change that fixes it can quietly damage the ordinary cases you did not re-run, and nothing will tell you: a prompt change is not local, and an instruction added to stop one annoying behaviour applies to every output, including all the ones that were already fine.

You are grading with the brain that wrote the change. Sixty seconds after deciding a rewrite should help, you read the output looking for evidence that it did, and fluent text is very accommodating. The register does not move with the quality. → Confidence tells you nothing

Write down what good looks like first#

Before you run anything, write the standard down. Anthropic's guidance on building evaluations puts this first and is blunt about the difference between a criterion and a wish: a target has to be specific enough to name the behaviour and measurable enough that two people scoring the same output reach the same verdict. Wanting the answers to be good is not a standard. Wanting every answer to name the source document it came from is.

Write it before you look at output, not after. A standard written afterwards describes the output you happen to have, and will be met by definition.

Most real standards have several dimensions at once: accuracy, tone, length, how it handles an input it cannot answer, what it costs, how long it takes. The useful part of writing them down is being forced to say which ones you would trade. A change that makes answers more accurate and twice as long is not obviously an improvement, and if length was never written down as something you cared about, you will find out that it was from somebody else.

Cases you didn't choose to flatter it#

An eval is a fixed set of inputs, run again after every change, scored the same way each time. The fixed part is what makes two runs comparable, so the set is the thing to get right.

Take them from work that already happened. Invented test cases come out of the same head that wrote the prompt, and they share its assumptions about what the input will look like. Real inputs are messier in the specific ways that break things.

Include the awkward ones on purpose. The empty input. The one that is far too long. The one that is genuinely ambiguous, where two competent people would disagree about the right answer. The one where somebody is trying it on. The guidance above lists these as the cases to remember rather than the cases to avoid, and they are where the difference between two prompts actually shows up.

Hold some back. If you adjust the prompt against every case you have, what you learn at the end is how the prompt performs on those cases, which is not the question you asked. Keep a portion sealed and score against it only when you think you are done. That is the oldest idea in this corner of the field, a held-out test set, and it is routinely skipped by people who would never skip it anywhere else.

More cases beats better grading#

This is the piece of advice most worth carrying away, because it runs against instinct. Anthropic's design principles for evals say to "prioritize volume over quality": a larger set of cases with rough automatic scoring beats a small set graded carefully by hand.

The reason is that careful grading is what caps the number of cases, and the number of cases is what decides whether a difference you observed is real. A dozen hand-scored cases produce a precise measurement of something too small to see past luck. Scoring that is somewhat crude but costs nothing to repeat lets the set be big enough to have an opinion.

There is a second reason, about people rather than statistics. A check that requires judgement gets run once, when you are enthusiastic. A check that runs itself gets run every time, including the time you were sure the change was too small to matter. Running it every time is the entire point.

Grade it the cheapest way that works#

The same guidance ranks the three ways of scoring, and the ranking is by speed and reliability rather than by sophistication. Code first, because it is fastest and gives the same verdict every time. People second, and it says to avoid grading by hand where you can, not because people score badly but because they are slow and expensive, which in practice means the grading stops happening. A model third, with a condition attached that the next section is about.

If you are not writing code, the first tier still applies, and it is worth translating. A code-based check is any check with exactly one right answer that anybody would score identically. Does the answer contain the client's name. Is it under two hundred words. Did it come back as valid JSON. Does every link in it resolve. Did it refuse the thing it was supposed to refuse. These feel too crude to be worth writing down, and between them they catch most of the ways a prompt change quietly makes things worse.

Spend the judgement on what genuinely cannot be reduced to a rule, which is usually much less of the standard than it first appears.

A model grading itself is an instrument you haven't calibrated#

Having a model score the output is fast, scales, and handles the judgement calls a rule cannot. The guidance recommends it, and attaches a condition that is easy to read past: establish that the grading is reliable before you scale it up. That condition is not a footnote, it is the whole technique.

A grader is a measuring instrument, and you calibrate an instrument against a known standard before you trust what it reads. So score a batch by hand yourself, have the model score the same batch, and look at where the two disagree. If they disagree often, the number you are about to track for the next month is a measurement of the grader.

Three things the guidance is specific about, all of which make the grader more repeatable: give it a detailed rubric, specific enough that a stated rule decides the verdict rather than an impression; make it emit something constrained, a verdict or a point on a fixed scale, because prose cannot be tallied and comparing two piles of prose is the problem you were escaping; and let it reason first and then discard the reasoning, which improves the score without inviting you to read it as an explanation. → Thinking and reasoning

The blind spot a grader inherits

What a self-check can and cannot reach is set out on How to check an AI's answer, and is not repeated here. What belongs to measuring rather than to checking is the consequence: a grader that shares the blind spots of the thing it grades does not register them as failures. It agrees. So the shared mistake arrives as a passing score, the metric moves the way you hoped, and the one number you are now watching has quietly become a measure of how alike the two of them are.

If you're not building anything#

Everything above comes from documentation written for people building software on top of a model, and it is worth saying so rather than borrowing the authority quietly. The tooling in that document does not transfer to someone using an assistant at work. The discipline does, and it fits in a spreadsheet.

The minimum honest version: one document, your prompt at the top, ten real inputs underneath, and a column for the output each one produced. Paste each input into a fresh session and keep what comes back. Change the prompt. Do it again, into the second column. Then read the two columns against each other, rather than reading the new one against your memory of the old one.

The fresh session matters for the same reason it does when you are checking a claim: a conversation that has already seen the earlier answer is not an independent run of anything. → Context windows

That is an afternoon at most, and it is inelegant, and it answers the question. An impression does not.

When it isn't worth it#

The threshold is repetition, not importance. If you are going to run a prompt three times and then never again, read the output, judge it, and get on with your day: building a test set costs more than the errors it would catch, which is the same trade as knowing where to stop checking.

If the prompt runs every day, sits inside something other people use, or produces output that reaches somebody before you see it, you already need this. The only open question is whether you find the regression in your own cases or in theirs.

Between those, there is a cheap middle that is worth more than its reputation: keep five real inputs in a file and re-run them whenever you change something. That is not evals in any serious sense. It is also the difference between noticing a change broke something and not noticing.

One last argument for owning a set of cases, and it is the reason this page exists on this site rather than a different one. A vendor's claim that a new model is better, a colleague's claim that a new prompt is better, and this site's claim about anything are all the same kind of claim: unverifiable until you have something specific to run it against. Your own cases are how you stop taking any of it on trust.

Sources#

One outside document, read on 25 August 2026 and listed on Sources with the pages that depend on it: Anthropic's Define success criteria and build evaluations. It is cited here for the shape of a success criterion, the three eval design principles including the volume-over-quality one quoted above, the ordering of code, human and model grading, and the tips for grading with a model.

What that document contains and this page deliberately does not: worked code, named metrics from machine learning, and example targets expressed as numbers. Those belong to the context they were written in, where they can be read with their assumptions attached. Nothing on this page is a figure, a model name, or a benchmark score, so there is nothing here to go stale in the way Model facts does. What is here is the argument, and the argument is checkable by trying it.