Running the eval

You have a dataset sitting in dataset.json and a prompt template you want to evaluate. The next step is the scaffold: three small functions that load the dataset, run each case through Claude, and collect results in a structured shape you can grade later. This lesson builds that scaffold end to end. The grader is still a placeholder — we'll replace it in the next two lessons — but the pipeline around it is real code you'll keep using.

Three functions, one pipeline

The evaluation pipeline splits into three functions, each with a single responsibility:

• run_prompt(test_case) — merge a single test case into the prompt template and call Claude.
• run_test_case(test_case) — call run_prompt, grade the result, and package everything into a result dict.
• run_eval(dataset) — iterate over the whole dataset calling run_test_case for each entry, and return the full list of results.

Three layers, each one a thin wrapper around the one below it. Keeping them separate means you can swap any layer independently later — a new grader, a parallel executor, a different prompt template — without touching the rest.

run_prompt

def run_prompt(test_case):
    """Merges the prompt and test case input, then returns the result"""
    prompt = f"""
Please solve the following task:

{test_case["task"]}
"""

    messages = []
    add_user_message(messages, prompt)
    output = chat(messages)
    return output

Three things happening:

• The prompt template is an f-string with {test_case["task"]} interpolated in. This is the baseline v1 prompt — deliberately minimal, no formatting instructions, no role, nothing.
• A fresh messages list is created for each call. Eval cases are independent — you don't want state leaking from one to the next.
• The function returns the raw text output, nothing more. Grading happens one layer up.

Because the prompt has no formatting guidance, the output at this stage will be verbose and inconsistent — Claude might wrap code in markdown, add explanations, or preface everything with "Here's a solution:". That's fine. The whole point of the eval is to measure the baseline before improving it.

run_test_case

def run_test_case(test_case):
    """Calls run_prompt, then grades the result"""
    output = run_prompt(test_case)

    # TODO - Grading
    score = 10

    return {
        "output": output,
        "test_case": test_case,
        "score": score,
    }

A few notes on this shape:

• The score is hardcoded to 10 for now. This is a deliberate placeholder. You need the pipeline to run end-to-end before you can build the grader — seeing real outputs flow through tells you what the grader will actually be looking at.
• The returned dict keeps everything together. Each result is {output, test_case, score}. That triple is enough for grading, debugging, and later analysis. You can trace any score back to the exact input that produced it and the exact text Claude returned.
• The TODO is there on purpose. Explicit placeholders beat hidden assumptions. When you replace the hardcoded 10 in the next lesson, the comment tells you exactly where to do it.

run_eval

def run_eval(dataset):
    """Loads the dataset and calls run_test_case with each case"""
    results = []

    for test_case in dataset:
        result = run_test_case(test_case)
        results.append(result)

    return results

Dead simple: iterate, call, collect, return. This is the layer that you'll later parallelise — swap the loop for ThreadPoolExecutor.map or asyncio.gather and you've cut eval latency dramatically — but the signature stays the same. Everything downstream continues to work.

Wiring it up

To actually run the eval, load the dataset and pass it in:

import json

with open("dataset.json", "r") as f:
    dataset = json.load(f)

results = run_eval(dataset)

Expect this to take a while even with Haiku — roughly 30 seconds for a small dataset, more as it grows. Eval runs are slower than you think, which is one reason why you want a stable dataset file on disk instead of regenerating it.

Inspecting results

print(json.dumps(results, indent=2))

You'll see one object per test case, each containing the original task, Claude's (probably verbose) output, and the placeholder score of 10. Look at the outputs — they're the failure modes you're about to grade against. The verbosity, the stray markdown, the extra explanations — these are exactly the kinds of things a grader will penalise once it exists.

What you've actually built here is most of an eval pipeline. The grader is still a hole — and a big one — but the structure around it is real. The next two lessons fill that hole: first with a model-based grader that uses Claude itself to judge outputs, then with a code-based grader that uses deterministic checks for things like "does this parse as valid JSON."