4Prompt evaluation
Exercise on prompt evals
3 min read411 words
Time: 15–20 minutes
Wire together everything from the last four lessons into a working end-to-end eval pipeline: dataset generation, prompt execution, model grading, and code-based syntax grading — running against the AWS code-generation prompt to produce an objective baseline score.
Step 1—Set Up
Open a new notebook. Import json, ast, re, and statistics.mean. Bring in the add_user_message, add_assistant_message, and chat helpers from the previous lessons — make sure chat accepts system, temperature, and stop_sequences.
Point your module-level model variable at Haiku for dataset generation and grading. You can swap in a larger model later if you want to see how grader quality changes.
1 / 3
You can now apply end-to-end eval pipeline in your own projects.