Generating test datasets

Three hand-written questions is not an eval. A real dataset has dozens or hundreds of cases covering the corners of your input space, and hand-authoring that is the kind of work that makes you skip the eval entirely. The fix: have Claude generate the dataset for you, use a fast cheap model to keep costs sane, and save it to disk so you never have to regenerate it.

The goal: AWS code-generation prompt

For the rest of this module, you'll evaluate a prompt that helps users write AWS-related code. Specifically, it should be able to produce three types of clean, machine-consumable outputs:

• Python functions
• JSON configuration objects
• Regular expressions

The requirement is that Claude return only the code — no explanatory prose, no markdown headers, no trailing commentary. Here's our starting prompt (version 1):

prompt = f"""
Please provide a solution to the following task:
{task}
"""

That's deliberately minimal. The whole point of the eval is to iterate from this baseline, so we want something simple enough to have lots of room for improvement.

The dataset shape

An eval dataset for this task is just an array of JSON objects, each describing one task we want Claude to solve:

[
  { "task": "Write a Python function that lists all S3 buckets." },
  { "task": "Generate an IAM policy JSON that grants read-only access to a specific bucket." },
  { "task": "Write a regex that matches AWS access key IDs." }
]

Each entry has a task field. That's the input we'll interpolate into the prompt.

Use Haiku for generation

Dataset generation is the canonical use case for a fast cheap model. You're going to generate tens or hundreds of items, each call costs you, and you don't need Opus-grade reasoning to come up with realistic task descriptions. Point your generator at Haiku and pocket the difference.

The helpers, extended

First, re-establish the helper functions from earlier lessons. Note that chat() now takes an optional stop_sequences argument — you'll need it for JSON extraction:

def add_user_message(messages, text):
    user_message = {"role": "user", "content": text}
    messages.append(user_message)

def add_assistant_message(messages, text):
    assistant_message = {"role": "assistant", "content": text}
    messages.append(assistant_message)

def chat(messages, system=None, temperature=1.0, stop_sequences=[]):
    params = {
        "model": model,
        "max_tokens": 1000,
        "messages": messages,
        "temperature": temperature,
    }
    if system:
        params["system"] = system
    if stop_sequences:
        params["stop_sequences"] = stop_sequences

    response = client.messages.create(**params)
    return response.content[0].text

This is the same shape as before, with stop_sequences added as a list with a default of []. The generator function uses it to clip the response at the closing code fence.

The generator function

def generate_dataset():
    prompt = """
Generate an evaluation dataset for a prompt evaluation. The dataset will
be used to evaluate prompts that generate Python, JSON, or Regex
specifically for AWS-related tasks. Generate an array of JSON objects,
each representing a task that requires Python, JSON, or a Regex to
complete.

Example output:
```json
[
  {
    "task": "Description of task"
  },
  ...additional
]

• Focus on tasks that can be solved by writing a single Python function, a single JSON object, or a single regex
• Focus on tasks that do not require writing much code

Please generate 3 objects. """

messages = [] add_user_message(messages, prompt) add_assistant_message(messages, "json") text = chat(messages, stop_sequences=[""]) return json.loads(text)

Two things this is doing:

- **Prefill + stop sequence.** Exactly the pattern you learned in Module 3. Assistant message ends with ` ```json `, and `stop_sequences=["```"]` clips the response at the closing fence. You get back a raw JSON array, no markdown, no preamble.
- **Inline example in the prompt.** The prompt shows the model the shape you want. This is a one-shot pattern — you tell Claude not just "generate JSON" but "generate JSON that looks like this."

## Running it

```python
dataset = generate_dataset()
print(dataset)

You should see a Python list of three dicts, each with a task field describing a realistic AWS coding problem. Run it a few times — the tasks will vary, and that's desirable. You want breadth, not repetition.

When you're building a real eval, bump the count up. Start with "Please generate 30 objects." Chunk into multiple calls if you want diversity — one call is correlated output; several calls get you more coverage of the input space.

Persist to disk

The one thing you absolutely want to do is save the dataset to a file. Regenerating it on every eval run is slow, expensive, and destroys your ability to compare prompt versions — because you're comparing against a different test set each time.

with open("dataset.json", "w") as f:
    json.dump(dataset, f, indent=2)

Now dataset.json is a stable artifact. Every eval run loads the same file, meaning score differences across prompt versions reflect real improvements, not dataset drift. When you want a new dataset — say, because you've discovered a new failure mode in production — you regenerate it deliberately and version the file.

The dataset is part of your eval infrastructure. Treat it like code — check it in, diff it, and know which version produced which score.