Code based grading

Your model grader tells you whether the content of Claude's response is any good. What it won't reliably tell you is whether the response actually parses as Python, or JSON, or a regex. For that kind of yes/no mechanical question, you don't want a probabilistic grader — you want a deterministic one. That's what a code grader is for.

What code graders are good at

A code grader is any Python function that takes an output and returns a score. It can do anything you can write in code:

• Parse the response and check the syntax
• Count tokens or characters
• Check for required substrings or forbidden ones
• Match against regex patterns
• Compute readability metrics
• Compare to a gold-standard answer with string similarity

The requirement is only that it returns a usable signal — usually a number between 1 and 10.

Code graders are fast, cheap, and deterministic. The same input always gets the same score. This makes them the right tool for anything with a mechanical definition of "correct": syntax validation, format compliance, structural checks. Save model graders for the things code graders can't do.

Three syntax validators

For our AWS code-generation prompt, the mechanical criterion is simple: the output has to parse as whatever format the task called for. Python's standard library gives us one-liner validators for each:

import ast
import json
import re

def validate_json(text):
    try:
        json.loads(text.strip())
        return 10
    except json.JSONDecodeError:
        return 0

def validate_python(text):
    try:
        ast.parse(text.strip())
        return 10
    except SyntaxError:
        return 0

def validate_regex(text):
    try:
        re.compile(text.strip())
        return 10
    except re.error:
        return 0

Each follows the same shape: try to parse, return 10 on success, return 0 on failure. Binary scoring is appropriate here — either the output parses or it doesn't, there's no middle ground to calibrate.

A note on the libraries:

• json.loads — parses JSON strings. Raises JSONDecodeError on any failure.
• ast.parse — the Python AST module. Parses arbitrary Python source code and raises SyntaxError if it can't. This catches everything the Python interpreter would catch at compile time without executing the code, which is exactly what you want.
• re.compile — compiles a regex pattern. Raises re.error for malformed patterns. Note that re.compile will happily accept patterns that match nothing — it's checking syntax, not semantic correctness.

The dataset schema: adding format

For the code grader to pick the right validator, each test case needs to specify which format it expects:

{
  "task": "Create a Python function to validate an AWS IAM username",
  "format": "python"
}

Update your generate_dataset() prompt to emit this field on every task — add it to the inline example shown in the generator prompt, and Claude will include it in future dataset generations.

Your code grader dispatches on format:

def grade_syntax(output, test_case):
    fmt = test_case["format"]
    if fmt == "python":
        return validate_python(output)
    if fmt == "json":
        return validate_json(output)
    if fmt == "regex":
        return validate_regex(output)
    return 0  # unknown format: treat as failure

Tightening the prompt

While we're updating the pipeline, now is the right moment to tighten the prompt itself so the format grader has a fair chance. Version 1 was deliberately minimal. Version 2 tells Claude exactly what we want:

* Respond only with Python, JSON, or a plain Regex
* Do not add any comments or commentary or explanation

And you can also prefill the assistant turn with a generic code fence so Claude starts in code-mode without having to decide which language:

add_assistant_message(messages, "```code")

Pair that with stop_sequences=["```"] on the chat() call and you get back raw code every time, no markdown, no preamble.

Combining scores

Now you have two scores per case: a model-grader score on content and a code-grader score on syntax. Merge them. A simple average gives equal weight to both:

model_grade = grade_by_model(test_case, output)
model_score = model_grade["score"]
syntax_score = grade_syntax(output, test_case)

score = (model_score + syntax_score) / 2

You can weight them differently if one matters more for your use case. For a code-generation prompt, some teams weight syntax heavier — wrong syntax is a hard failure, and the model grader's 8/10 on content doesn't help if the code won't run. For a classification prompt, you'd drop syntax entirely and lean on the model grader. The right weighting is a call you make based on what your prompt is actually doing.

What matters now

Run the pipeline. Write down the baseline score. That number is not inherently good or bad — it's a starting point. What matters is whether you can make it go up by iterating on the prompt, and whether the improvements you make survive rerunning on the same dataset. You have replaced the subjective question "does this prompt feel better?" with the objective question "did the score move?" That is the whole point of the exercise.