Myth busters

Reality

Same spirit, different beast. AI output can change from run to run, so you need repeated runs, clear rubrics and sometimes a judge — not one exact-match assertion.

Analogy

A unit test is a light switch: on or off. An eval is a free-throw record: you need several shots to know the shooter.

Source: Anthropic Engineering (2026)

Reality

One study found AI graders squeezed scores toward the middle of the scale; another found small wording changes can shift them. Yes/no checks are easier to repeat.

Analogy

Ask friends to “rate the film out of 10” and one person’s 6 is another’s 8. Ask “would you watch it again?” and you get a clear answer.

Source: Zhang et al. (2026) · Li et al. (2025)

Reality

Public benchmarks test general skills, and models may have seen the test questions during training. Only your own evals, on your own tasks, tell you what works for you.

Analogy

Being top of the class in general knowledge doesn’t make you good at your specific job.

Source: Sainz et al. (2023)

Reality

AI judges have known biases, such as favouring longer answers or the first answer they see. Check your judge against human labels before you trust it.

Analogy

A talent-show judge who always prefers the longest speech. Useful, but you’d want to know that.

Source: Zheng et al. (2023)

Reality

Domain experts decide what “good” means. Engineers wire it up. The best checklists are written by the people who know the work.

Analogy

The head chef writes the recipe card; the kitchen porter doesn’t.

Reality

Models, prompts, tools and user needs keep changing. Evals go stale. Review and update them like any other code.

Analogy

A map of a growing city. Last year’s map gets you lost.

Reality

Start small with real cases. One practitioner guide suggests 20–50 simple tasks drawn from real failures is a great start.

Analogy

You don’t need a full gym to start exercising. A few push-ups today beat a perfect plan next year.

Source: Anthropic Engineering (2026)

Reality

Research on agents found simpler, cheaper setups were often about as accurate as costly ones. Test cost and quality together, task by task.

Analogy

You don’t hire a surgeon to put on a plaster.

Source: Kapoor et al. (2024)

Reality

Evals are a stand-in for the real world. Real users find new failures, so keep tracing production and turning failures into new tests.

Analogy

Passing a driving test doesn’t mean you’ll never have an accident. That’s why cars have dashcams and insurance.

Reality

Start with better instructions, tools and model choice. They’re faster and cheaper to change. Fine-tune later, when evals show a clear, narrow gap.

Analogy

Before sending someone on a course, check you gave them clear instructions and the right tools.