Prompt evaluation

You've shipped a feature built on a prompt. It worked perfectly when you tested it. A week later, a user submits something you never considered — an emoji in a field you assumed was ASCII, a two-sentence question you thought would be one — and the whole flow falls over. This is not a story about writing a better prompt. It's a story about not having a way to measure whether a prompt is good.

Two different skills

Prompt engineering is how you write prompts. Prompt evaluation is how you know they work. They are complementary, and most engineers learn the first one, skip the second, and then wonder why production keeps surprising them.

Prompt engineering gives you a toolkit: clarity, structure with XML tags, few-shot examples, explicit constraints, multishot patterns. These are the techniques for getting Claude to produce what you want in the first place. You'll spend Module 5 on them.

Prompt evaluation is different. It's the practice of measuring a prompt's effectiveness against a dataset of realistic inputs, scoring the outputs objectively, and using those scores to decide whether version B is actually better than version A. It answers one specific question: how good is this prompt, really?

Three paths after drafting a prompt

Once you've written a prompt, you face a choice:

Option 1 — Test it once and ship it. You run the prompt on the input you had in mind. It works. You push to production. This is the default, it feels efficient, and it is the most expensive option in the long run because users will absolutely find a case you didn't think of.

Option 2 — Test it a few times and tweak for the edge cases you noticed. Better, in the sense that you're thinking about edge cases at all. Still wildly insufficient for anything production-facing. Real users will produce inputs you could not have imagined during manual testing — not because you're bad at your job, but because there are more of them than there are of you.

Option 3 — Run it through an evaluation pipeline. You build a dataset of realistic inputs (tens, hundreds, or thousands), feed each one through the prompt, score the outputs automatically, and iterate based on objective metrics. This is more work upfront. It is also the only option that actually answers "is this prompt good enough to ship."

Option 3 is the option that makes you an engineer instead of a prompt hobbyist. The rest of this module teaches you how to build it.

Why engineers consistently fall into the first two traps

We all do this. Including the people writing this course. There are three honest reasons:

• Confirmation bias in testing. When you write a prompt, you test it on the input that was in your head when you wrote it. That input is not a random sample of user behaviour — it's the input that led to the prompt in the first place.
• Underestimation of input variance. Real traffic has a long tail. A prompt that handles 95% of inputs perfectly can still be a bad prompt if the other 5% is where your revenue, compliance, or user trust lives.
• Evals feel like overhead. Building a test dataset and a grader takes a day. Shipping without one takes an hour. The tax on evals is paid upfront; the tax on skipping them is paid in production, in incidents, and in trust.

Naming the trap is most of getting out of it. If you catch yourself about to ship a prompt on "I tested it and it looked fine," that is the moment to stop and build the eval.

Evaluation-first development

Evaluation-first development is the practice of treating the eval as part of the prompt, not a thing you might get to later. The eval exists before the "final" prompt exists. You draft a prompt, score it, iterate, score again, and pick the version with the best metric.

This sounds like more work. It is, at first. What you get in return:

• Objective comparison. You can tell whether version B is really better than version A instead of trusting your intuition after two manual runs.
• Regression safety. When you change a prompt three months from now, the eval tells you immediately whether the change broke something you weren't thinking about.
• Confident iteration. You can try weird ideas — swap in a new technique, reorder instructions, add or remove context — because the eval will catch bad changes before they reach users.
• Reliability that compounds. The next feature built on the same prompt inherits the eval. The feature after that inherits it too.

The systematic approach removes guesswork from prompt engineering. You stop arguing about which version sounds better and start measuring which version is better. Over a few iterations, that difference is the difference between a feature that works and a feature that holds up in production.

The next lesson walks through the five-step eval workflow concretely, with a small worked example you can run.