A typical eval workflow

You've bought into the idea that prompts need evals. Now what does an eval actually look like in code? Strip away the fancy tools and dashboards and you're left with a surprisingly simple loop: a dataset, a prompt, some Claude calls, a grader, and an average score. This lesson walks through the whole thing with a toy example small enough to hold in your head.

The five steps

Every eval workflow — from a notebook prototype to a production pipeline — follows the same five steps:

1. Draft a prompt.
2. Create an eval dataset.
3. Feed each dataset item through Claude.
4. Grade the responses.
5. Change the prompt and repeat.

That's it. Everything else — parallelism, caching, grading sophistication, drift detection — is optimisation on top of this core loop.

Step 1: Draft a prompt

Write an initial version of the prompt you want to evaluate. It doesn't need to be good yet. The whole point of an eval is that you don't have to guess whether it's good — you can measure.

prompt = f"""
Please answer the user's question:

{question}
"""

This is deliberately basic. It gives Claude a question and nothing else. It will be our baseline.

Step 2: Create an eval dataset

An eval dataset is a collection of sample inputs that represent the kind of traffic your prompt will handle in production. For a prompt that answers user questions, the dataset is a list of questions. For a classification prompt, it's a list of inputs with their true labels. For a code generation prompt, it's a list of task descriptions.

For this example, three questions:

• "What's 2+2?"
• "How do I make oatmeal?"
• "How far away is the Moon?"

Three is absurdly small. Real datasets have tens to thousands of items. You can assemble them by hand, pull them from production logs, or — as you'll see next lesson — have Claude generate them. For now, three is enough to illustrate the shape of the loop.

Step 3: Feed each item through Claude

Interpolate each question into the prompt template and send it to Claude. Each question becomes a complete prompt:

Please answer the user's question:
What's 2+2?

Claude responds to each one:

• To the math question: "2 + 2 = 4"
• To the oatmeal question: a cooking explanation of varying detail
• To the Moon question: a distance figure with context

Collect each response alongside the input that produced it. You're going to need both in the next step.

Step 4: Feed responses through a grader

A grader is a function that looks at an input and an output and returns a score. At this point you don't have to care how the grader works — it could be another Claude call, a regex, a deterministic check against a known answer, or a human rubric. Scores are typically on a 1–10 scale, where 10 is "perfect" and lower numbers flag progressively worse output.

For our three questions, a grader might return:

| Question | Score | |---|---| | What's 2+2? | 10 | | How do I make oatmeal? | 4 | | How far away is the Moon? | 9 |

The average is (10 + 4 + 9) ÷ 3 = 7.66.

That number — the average score — is the object of the whole exercise. It's the single scalar you're trying to move up. It's what you'll compare across prompt versions.

Step 5: Change the prompt and repeat

Now you have a baseline: 7.66. You can change the prompt and run the exact same dataset through the exact same grader to see whether your change moved the number.

prompt = f"""
Please answer the user's question:

{question}

Answer the question with ample detail
"""

Run the same three questions through this new prompt. You might see the average climb to 8.7. That's a real, measurable improvement — not "feels better to me," but a 1.04-point bump on the same dataset under the same grader.

Or it might drop. That's also information. A drop means the "more detail" instruction made some other class of answers worse. You now know something about the prompt you didn't know before.

Why this works

The value of the whole workflow is that the comparison is objective. You are no longer arguing about which prompt sounds better. You have a number. Numbers compound across iterations:

• You can try ten prompt variants in an afternoon and pick the best one.
• You can catch regressions when you come back to the prompt in three months.
• You can justify prompt changes to your team with something other than vibes.

The toy example is small, but everything larger is built out of the same five steps. The next lesson replaces the hand-written dataset with one generated by Claude itself — because three hand-written questions is a demo, not an eval.