Prompt engineering

You have an eval pipeline. You now also have the thing most engineers never get: a number that tells you, objectively, whether a prompt is any good. This module is about using that number — iterating on a real prompt and watching the score climb as you apply one systematic technique at a time. The running example across all five lessons is a meal plan generator for athletes, starting from a deliberately bad prompt scoring around 2.3 out of 10.

Prompt engineering, finally defined

Prompt engineering is the practice of taking a prompt you've written and improving it through iterative refinement, measured against an eval. Without the eval, "prompt engineering" is a loose bag of tricks and personal taste. With one, it's a tight loop with objective feedback.

The loop has five steps:

1. Set a goal. What is this prompt actually for?
2. Write an initial prompt. Deliberately basic. Don't optimise the first draft.
3. Evaluate it. Run it through your eval and write down the score.
4. Apply a technique. One change at a time. Clarity, specificity, XML structure, examples.
5. Re-evaluate. Did the score move?

You repeat steps 4 and 5 until you hit the quality bar or stop getting returns on further changes. One change at a time is non-negotiable — if you apply three techniques in a single iteration and the score goes up, you don't know which one did the work.

The running example: meal plans for athletes

Across the next four lessons, you'll iterate on a single prompt: generate a one-day meal plan for an athlete given their height, weight, goal, and dietary restrictions. It's a good example because the output has real quality dimensions — caloric total, macro breakdown, meal timing, restriction compliance — and a naive prompt will miss most of them.

The eval harness: PromptEvaluator

To keep the focus on engineering technique and not on eval plumbing, the course uses a helper class:

evaluator = PromptEvaluator(max_concurrent_tasks=5)

PromptEvaluator wraps everything you built in Module 4 — dataset generation, running prompts, grading, score aggregation — with a few quality-of-life additions:

• Concurrency control. max_concurrent_tasks runs multiple grader calls in parallel so iteration isn't painfully slow. Start at 3 if you're on a tight rate limit, bump up to 5 or 10 once you know you have headroom.
• Structured dataset generation. You pass a spec of the inputs your prompt takes, and it generates matching test cases.
• HTML reports. Each run produces a browseable report with the output, the reasoning, and the score for every case — the single most useful artifact for debugging why a change didn't help.

Generating the dataset

dataset = evaluator.generate_dataset(
    task_description="Write a compact, concise 1 day meal plan for a single athlete",
    prompt_inputs_spec={
        "height": "Athlete's height in cm",
        "weight": "Athlete's weight in kg",
        "goal": "Goal of the athlete",
        "restrictions": "Dietary restrictions of the athlete",
    },
    output_file="dataset.json",
    num_cases=3,
)

Three cases is a tiny dataset — deliberately so. During iteration, keep the dataset small. Running 100 cases on every edit kills your turnaround time and discourages experimentation. Get to a good prompt on three cases first, then scale up to 30 or 100 to validate.

The baseline prompt

Now write the worst prompt you can defensibly call a first draft:

def run_prompt(prompt_inputs):
    prompt = f"""
What should this person eat?

- Height: {prompt_inputs["height"]}
- Weight: {prompt_inputs["weight"]}
- Goal: {prompt_inputs["goal"]}
- Dietary restrictions: {prompt_inputs["restrictions"]}
"""

    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

"What should this person eat?" is deliberately terrible. It's a question instead of an instruction, it doesn't specify the output format, and it gives Claude nothing about what "good" looks like. We want the baseline to be bad so we have room to show real improvement.

Running the first eval

results = evaluator.run_evaluation(
    run_prompt_function=run_prompt,
    dataset_file="dataset.json",
    extra_criteria="""
The output should include:
- Daily caloric total
- Macronutrient breakdown
- Meals with exact foods, portions, and timing
""",
)

extra_criteria gives the grader additional rubric points beyond what it would infer on its own. This matters — without it, the model grader may score based on general quality (is the response coherent?) rather than your specific requirements (does it list calories per meal?).

Run it. The score will come back low. In the demo, the baseline scores 2.32 out of 10. Don't be discouraged by low initial numbers — a score in the low single digits is typical for a deliberately weak baseline, and it's the starting line, not the finish line.

The HTML report

After every run, open the HTML report. For each test case it shows the input, the output, the grader's reasoning, and the score. Read the reasoning strings. They tell you exactly what the grader thinks the prompt is missing — the output is too short, it doesn't include calorie counts, the foods don't fit the restriction, and so on. That list of complaints is your roadmap for the next iteration.

The iteration discipline

From here, the rest of this module is about applying one engineering technique per lesson and watching the score climb:

• Being clear and direct — rewriting the opening line
• Being specific — adding output guidelines and process steps
• Structuring with XML tags — giving Claude clean boundaries
• Providing examples — one-shot and multi-shot patterns

Each technique will produce a measurable score jump on the same dataset and the same grader. You'll watch 2.3 turn into 4, then 8, then higher. That progression is prompt engineering — not a bag of tricks but a disciplined, measurable practice that turns "I think this is better" into "the score went up 2.4 points."