Quiz: test yourself — AI Evals, Made Simple

01What is an eval?
A repeatable check of whether an AI system did its job correctlyA one-off demo that shows the AI at its bestA leaderboard ranking of AI modelsA user survey about how the AI feels to use
02Why do yes/no checklists usually beat a 1–10 score from an AI grader?
Checklists are faster to typeYes/no answers are usually easier to keep consistent; numeric scores can bunch toward the middle and shift with wordingAI graders can’t produce numbersA 1–10 score is only for humans
03A support agent must handle the same request correctly for every customer. Which measure fits best?
pass@k: did at least one of k tries succeed?pass^k: did all k tries succeed?The single best run you sawThe average length of its answers
04What is a good first eval dataset?
Thousands of synthetic questions generated overnightA public benchmarkA small set of real tasks, especially ones the AI has failed beforeNothing until the product launches
05Which of these is a known bias of AI judges?
They always pick the shortest answerThey refuse to grade codeThey can favour longer answers, or whichever answer is shown firstThey grade every answer the same
06During an eval, the agent can read the file that holds the expected answers. What is this called?
Environment leakageFine-tuningDistillationObservability
07What does a trajectory eval check?
Only the final answerThe steps the agent took to get there, such as which tools it called and in what orderHow fast the model loadsHow much the model costs per month
08Why can a top leaderboard score mislead you?
Leaderboards are always out of dateModels may have seen the test questions in training, and the tests may not match your tasksLeaderboards only test speedHigher scores always mean worse models
09You changed a prompt. What should happen next?
Ship it if one example looks betterRe-run the eval suite to catch anything that used to work and now breaksWait for users to complainSwitch to a bigger model
10Two models are equally good on your evals. One is cheaper and faster. Which do you pick?
The more expensive one, to be safeThe cheaper, faster one: it gives the same quality for lessWhichever is newerNeither; evals can’t help choose

0 correct · 0 of 10 answered