Model based grading

You have a pipeline that runs prompts against a dataset and collects outputs. The hole in the middle is the grader — right now every case scores 10 regardless of whether the output is brilliant or nonsense. This lesson fills that hole with a model grader: a second Claude call that reads the original task and the generated response and returns a structured judgment.

Three ways to grade

Before diving into code, know the three approaches you can mix and match:

• Code graders — deterministic checks written in Python. Output length, required substrings, JSON-parses-cleanly, regex matches. Cheap, fast, and perfect for anything with a mechanical definition of "correct."
• Model graders — a second AI call that judges the output. Slower and costlier, but flexible enough to evaluate things like "did the answer address the user's actual question" or "is this response helpful."
• Human graders — a person reviews outputs against a rubric. The most flexible, by far the slowest, and the only option when the thing being judged is subjective in a way neither code nor models can capture.

For most production evals, you'll combine code graders for mechanical properties with a model grader for quality. This lesson builds the model grader; the next lesson adds the code grader.

Defining criteria first

You cannot write a grader until you've written down what "good" means. For our AWS code-generation prompt, the criteria are:

• Format — the response should be only Python, JSON, or regex, with no explanatory prose or markdown fences.
• Valid syntax — whatever's returned should actually parse as the intended language.
• Task following — the response should address what was asked, and the code should be approximately correct.

Format and valid-syntax are best checked programmatically (that's the next lesson). Task following is what you'll use a model grader for — it's a judgment about content quality that doesn't reduce cleanly to a regex.

The grader prompt

A model grader is just a prompt with very specific structure. Here's the shape:

def grade_by_model(test_case, output):
    eval_prompt = f"""
    You are an expert code reviewer. Evaluate this AI-generated solution.

    Task: {test_case["task"]}
    Solution: {output}

    Provide your evaluation as a structured JSON object with:
    - "strengths": An array of 1-3 key strengths
    - "weaknesses": An array of 1-3 key areas for improvement
    - "reasoning": A concise explanation of your assessment
    - "score": A number between 1-10
    """

    messages = []
    add_user_message(messages, eval_prompt)
    add_assistant_message(messages, "```json")

    eval_text = chat(messages, stop_sequences=["```"])
    return json.loads(eval_text)

Three things worth understanding about this prompt:

1. The persona matters. "You are an expert code reviewer" puts Claude in the right mode for the task. Without it, you get more hedging and less opinionated grading.

2. The structured output uses the prefill + stop_sequence pattern. Same technique from Module 3 — prefill ```json, stop at ```, parse with json.loads. The grader's output has to be machine-consumable or the rest of the pipeline can't use it.

3. Reasoning comes before the score. This is the important one. If you only ask for a number, Claude defaults to middling values — everything drifts toward 6. Asking for strengths, weaknesses, and reasoning first forces the model to think about the output, and the score that comes out the other side is dramatically more discriminating. Reason first, score second is a pattern you'll use any time you're asking a model to evaluate something.

Wiring the grader into the pipeline

Replace the hardcoded score in run_test_case with a call to the grader:

def run_test_case(test_case):
    output = run_prompt(test_case)

    model_grade = grade_by_model(test_case, output)
    score = model_grade["score"]
    reasoning = model_grade["reasoning"]

    return {
        "output": output,
        "test_case": test_case,
        "score": score,
        "reasoning": reasoning,
    }

Two additions: score now comes from the grader, and reasoning is included in the result so you can read why a case got the score it got. When you're debugging a prompt change, those reasoning strings are the single most useful thing in the output — they tell you what the grader saw that drove the number.

Computing the average

One more change, in run_eval:

from statistics import mean

def run_eval(dataset):
    results = []

    for test_case in dataset:
        result = run_test_case(test_case)
        results.append(result)

    average_score = mean([result["score"] for result in results])
    print(f"Average score: {average_score}")

    return results

The average across all cases is the scalar you'll be optimising. Print it at the end of every run so you can see the number move as you iterate.

What you've got now

A working end-to-end pipeline: dataset in, scored results out, with the average printed at the bottom. Run it on v1 of the prompt and write down the number. That's your baseline — the value every subsequent version needs to beat.

A caveat worth knowing. Model graders are somewhat capricious. The same output run through the same grader can get slightly different scores on different runs, because the grader is itself sampling from a distribution. Set the grader's temperature low to reduce this, average over multiple runs if you want more stability, and treat small score differences (0.1–0.3) as noise rather than signal. For bigger differences, they're real.