// Evals 101 · Start today

Your first eval, this afternoon

Seven steps, in order. Tick them off as you go, then fill in the eval spec below and download it as a Markdown file you can share with your team.

0 of 7 steps done

  1. Choose one real job your AI does, like “summarise a support ticket” or “draft a refund reply”. One task, not the whole product.

  2. Pull real inputs from logs or your inbox, and include the cases where the AI already got it wrong. Real failures make the best test cases.

    Source: Anthropic Engineering (2026)

  3. List what a good answer must do: “mentions the order number”, “doesn’t promise a refund date”. A 1–10 score is harder to pin down: one study found AI graders squeezed scores toward the middle of the scale.

    Source: Zhang et al. (2026)

  4. Use code where a rule can decide (is the date in the right format?). Use an AI judge for judgement calls, and check it against your own labels. Use a person when the stakes are high.

    Source: Anthropic Engineering (2026)

  5. AI can answer differently each run. Run each case a few times and record the pass rate, not the best try.

    Source: Yao et al. (2024)

  6. Open the failed cases and sort them into types before changing anything. Fix the most common type first, with a prompt, tool or data change.

  7. Run the same set after every prompt, tool or model change, and add each new real failure as a case. That’s your eval habit started.

// Template

Write your eval spec

One page that says what the AI must do, how you’ll check it and how you’ll run it. The preview updates as you type.

Grader

A script checks each rule (format, required fields, forbidden phrases). Fast, cheap and repeatable.

Saves as eval-spec-my-task.md. Made in your browser; nothing you type is sent or saved.

Preview

# Eval spec: My task

> Made with Evals 101 (mirquadri.com/learn/evals). Keep it short, keep it real, run it often.

## Task

_(fill this in)_

## Pass checks (yes/no)

- [ ] _(fill this in)_

## Test cases

Aim for 20 to 50 real examples, including past failures.

1. _(fill this in)_

## Grader

**Code.** A script checks each rule (format, required fields, forbidden phrases). Fast, cheap and repeatable.

## How to run

- Run every case several times and record the pass rate, not the best try.
- Read the failures and sort them into types before changing anything.
- Re-run the whole set after every prompt, tool or model change.
- Add each new real failure as a test case.

## Results log

| Date | Change | Pass rate | Notes |
| --- | --- | --- | --- |
|  |  |  |  |