ORCHESTRATE AI · Evals 101

AI evals, made simple

An eval is a driving test for AI. You don’t trust a new driver because they sound confident — you watch them drive. This guide shows you how to watch AI work, from your first check to a loop that keeps it improving.

1 Simple words
2 Analogy
3 Example
4 Why it matters

// Journey map

Three levels, like a video game

Start with one task and one check. Level up to test environments and pick models. Finish with the loop that keeps an agent improving in production.

Level 1Easy ModeOne task and one verifier0 of 5 concepts understood
Level 2Hard ModeEnvironments, suites and model choice0 of 8 concepts understood
Level 3God ModeThe production improvement flywheel0 of 7 concepts understood

Press “Got it” on a concept to fill the map.

Progress stays while you move around these pages. Nothing is saved.

// Level 1

Easy Mode

One task and one verifier

Try the Easy Mode demos →

Eval

Simple words: A check that the AI did its job correctly.
Analogy: A driving test for the AI. You don’t trust a new driver because they sound confident; you watch them drive.
Example: You ask an AI assistant to book a meeting. The eval checks: is the meeting on the calendar, at the right time, with the right people?
Why it matters: Without evals you are guessing. With them you can see whether a change made the AI better or worse.

Task

Simple words: One small job the agent must do, which you can check afterwards.
Analogy: One exam question. Small enough to mark right or wrong.
Example: “Add a follow-up note to the Acme Co record in the CRM.”
Why it matters: Big, vague jobs are impossible to grade. Small, clear tasks give you a clear yes or no.

Source: Anthropic Engineering (2026)

Verifier

Simple words: The code or checklist that decides right or wrong.
Analogy: The answer key plus the person marking the exam.
Example: A few lines of code that look in the CRM and confirm a new note exists on Acme Co, dated today.
Why it matters: A task without a verifier is just a demo. The verifier is what turns “looks fine” into “passed”.

Source: Anthropic Engineering (2026)

Checklist (binary rubric)

Simple words: Yes/no questions instead of a 1–10 score.
Analogy: Pass/fail checkboxes versus “rate this from 1 to 10”. Everyone agrees whether a box is ticked; nobody agrees what a 6 means.
Example: “Is the note saved? Yes/No. Is the email under 3 paragraphs? Yes/No. Was nothing actually sent? Yes/No.”
Why it matters: One study found AI graders squeezed numeric scores toward the middle of the scale; another found small wording changes can shift them. Yes/no checks are easier to agree on and to repeat.

Source: Zhang et al. (2026) · Li et al. (2025)

Agent

Simple words: An AI that takes actions with tools, not just one that chats.
Analogy: A new employee, not a search box. You hand it a job, and it clicks, types and files things for you.
Example: Instead of telling you how to update a CRM record, the agent logs in and updates it.
Why it matters: Actions have consequences. An agent that sends the wrong email causes real harm, so it needs real checks.

// Level 2

Hard Mode

Environments, suites and model choice

Try the Hard Mode demos →

Environment

Simple words: A safe copy of real tools and data where the agent can practise.
Analogy: A flight simulator. Pilots crash there so they don’t crash for real.
Example: A fake CRM filled with made-up companies, plus a fake email outbox that never sends.
Why it matters: You can’t test an agent on your real customers. An environment lets it make mistakes safely.

Sandbox (container)

Simple words: A sealed, repeatable snapshot of the environment, often built with a tool like Docker.
Analogy: A terrarium. Everything the agent needs is inside, and nothing leaks out.
Example: Every test run starts from the exact same fake CRM. When the run ends, the box is thrown away.
Why it matters: If every run starts from the same state, a different result means the agent changed, not the world.

Eval suite

Simple words: A tagged collection of tasks. Each task has an instruction, an environment and a verifier.
Analogy: A full exam paper, split into subjects.
Example: A set of CRM tasks tagged “notes”, “email drafts” and “data cleanup”.
Why it matters: One task can pass by luck. A suite shows where the agent is strong and where it breaks.

Model catalog

Simple words: The list of AI models you might use for the job.
Analogy: A shortlist of job candidates. Same interview for everyone.
Example: Model A (large), Model B (medium), Model C (small and cheap) all run the same suite.
Why it matters: New models arrive all the time. A catalog plus a suite turns “should we switch?” into a quick test.

Cost vs. quality trade-off

Simple words: A cheaper model may be good enough for the job.
Analogy: Economy vs. business class. Both reach the same destination; one costs far more.
Example: If the small model passes every check on internal notes, you may not need the large one there.
Why it matters: Research on agents has argued that cost should be measured alongside accuracy, not ignored.

Source: Kapoor et al. (2024)

Reward hacking (cheating)

Simple words: The agent finds a loophole to pass the test without doing the work.
Analogy: A student who finds the answer key instead of learning the material.
Example: The verifier only checks that “a note exists”, so the agent saves an empty note and passes.
Why it matters: A passing score means nothing if the agent gamed it. Good verifiers check the real outcome.

Source: Amodei et al. (2016)

Environment leakage

Simple words: The answer is accidentally left somewhere the agent can see it.
Analogy: A test paper handed out with the answers printed on the back.
Example: The fake CRM has a column called “expected_answer” that the agent can read.
Why it matters: Leaks make a weak agent look brilliant. You only find out when it fails on real data.

Open-source eval frameworks

Simple words: Shared, free standards for packaging tasks, so people can reuse each other’s evals.
Analogy: Standard shipping containers. Any port can load any box.
Example: Harbor, from the team behind Terminal-Bench, runs agent evals in containers using a shared task format.
Why it matters: You don’t have to build every test from scratch, and your results are easier to compare.

Source: Harbor Framework Team (2026)

// Level 3

God Mode

The production improvement flywheel

Try the God Mode demos →

Trace

Simple words: A step-by-step log of everything the agent did.
Analogy: A flight recorder, or a stack of receipts for every step.
Example: Step 1: searched the CRM for “Acme”. Step 2: opened the record. Step 3: wrote the note. Step 4: drafted the email.
Why it matters: A score says that something failed. A trace shows where and why.

Source: Anthropic Engineering (2026)

Tracing

Simple words: Switching that step-by-step logging on, in testing and in production.
Analogy: Turning the dashcam on before you drive, not after the crash.
Example: Every live agent run saves its trace, so any complaint can be replayed later.
Why it matters: Real users will do things your tests never tried. Without tracing, you can’t see what went wrong.

Harness engineering

Simple words: Improving the prompts, tools and model choice around the agent.
Analogy: Tuning the workplace, not the worker: better instructions, better tools, the right person for the job.
Example: The trace shows the agent keeps picking the wrong Acme. You add a tool that searches by company ID.
Why it matters: It’s usually the cheapest place to start, so try it first. Note: people also say “eval harness” for the machinery that runs tests. Different thing.

Fine-tuning

Simple words: Training a model further on your own examples, for one narrow job.
Analogy: Sending a specialist on a short course.
Example: You train a small model on your approved CRM notes so it writes them in your house style.
Why it matters: It can make a small, cheap model great at one job. It costs more effort, so try harness changes first.

Improvement flywheel

Simple words: Production traces become new evals, which make a better agent, which you run again.
Analogy: A gym routine that adapts to your last workout.
Example: A customer complaint → find the trace → turn it into a new task → fix the prompt → the new task now passes.
Why it matters: Your eval suite grows from real failures, so the agent gets better at exactly what your users need.

Source: Anthropic Engineering (2026)

Human in the loop

Simple words: Humans decide what “good” means; agents do the grinding.
Analogy: An editor sets the style guide; the writers follow it.
Example: A sales lead writes the checklist for a good follow-up email. The agent drafts; the checklist grades.
Why it matters: Domain experts know what good looks like. Evals turn their judgement into something repeatable.

Commercial eval platforms

Simple words: Paid products for storing traces, running evals and reviewing results.
Analogy: A managed gym versus building your own.
Example: LangSmith, from LangChain, is one such platform. Others exist; you can also build the basics yourself.
Why it matters: They save setup time. The ideas — tasks, verifiers, traces — are the same whichever tool you pick.

Source: LangChain (2026)

// Beyond the video

What the video didn’t say

Seventeen more ideas — online evals, AI judges, RAG checks, red teaming, metrics — in the same plain-words format.

Read the 17 extras →

How this guide stays honest

Research claims link to their source. Examples use made-up companies and generic names like “Model A”. Every interactive demo carries an “Illustrative example — not real data” label, because its numbers are made up for teaching.