// Evals 101 · Sources
Sources
Every research claim in this guide links to one of these. Each entry says what we use it for and how it was checked.
Anthropic Engineering · 2026
Demystifying evals for AI agents (opens in a new tab)
Used for: Definitions of task, trial, grader, transcript and eval harness; code, model and human graders, including code-based tool-call verification; "20-50 simple tasks drawn from real failures is a great start".
Checked on , read in full
Zheng et al. · 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)
Used for: LLM judges show position, verbosity and self-enhancement biases; strong judges reached over 80% agreement with human preferences.
Checked on , checked by title, authors, date and abstract
Zhang et al. · 2026
Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring (opens in a new tab)
Used for: One study (multimodal LLMs scoring a clinical test on an ordinal scale): all three model families compressed scores toward the middle of the scale.
Checked on , checked by title, authors, date and abstract
Li et al. · 2025
Evaluating Scoring Bias in LLM-as-a-Judge (opens in a new tab)
Used for: Score-based LLM judges are swayed by rubric order, score labels and reference answers.
Checked on , checked by title, authors, date and abstract
Chen et al. · 2021
Evaluating Large Language Models Trained on Code (opens in a new tab)
Used for: Popularised the pass@k metric: at least one of k attempts succeeds.
Checked on , checked by title, authors, date and abstract
Yao et al. · 2024
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (opens in a new tab)
Used for: Introduced pass^k to measure how reliably an agent succeeds across repeated trials; tests agents in conversations with a language-model-simulated user.
Checked on , checked by title, authors, date and abstract
Kapoor et al. · 2024
AI Agents That Matter (opens in a new tab)
Used for: Agent evaluations should weigh cost alongside accuracy; simple baselines were often on the cost-accuracy Pareto frontier.
Checked on , checked by title, authors, date and abstract
Es et al. · 2023
Ragas: Automated Evaluation of Retrieval Augmented Generation (opens in a new tab)
Used for: RAG metrics for faithfulness, answer relevance and context relevance.
Checked on , checked by title, authors, date and abstract
Sainz et al. · 2023
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (opens in a new tab)
Used for: When a model has seen a benchmark’s test data in training, its score on that benchmark is overestimated.
Checked on , checked by title, authors, date and abstract
OWASP · 2025
OWASP Top 10 for LLM Applications 2025 (LLM01: Prompt Injection) (opens in a new tab)
Used for: Prompt injection is the top-listed risk; direct and indirect forms.
Checked on , checked by title, authors, date and abstract
Greshake et al. · 2023
Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (opens in a new tab)
Used for: Instructions hidden in retrieved content can hijack an LLM-integrated app.
Checked on , checked by title, authors, date and abstract
Amodei et al. · 2016
Concrete Problems in AI Safety (opens in a new tab)
Used for: Names reward hacking as a practical AI safety problem.
Checked on , checked by title, authors, date and abstract
Hinton, Vinyals & Dean · 2015
Distilling the Knowledge in a Neural Network (opens in a new tab)
Used for: Knowledge distillation: compressing a large model’s knowledge into a smaller one.
Checked on , checked by title, authors, date and abstract
Hu et al. · 2021
LoRA: Low-Rank Adaptation of Large Language Models (opens in a new tab)
Used for: LoRA freezes the base model and trains small added matrices instead.
Checked on , checked by title, authors, date and abstract
Ouyang et al. · 2022
Training language models to follow instructions with human feedback (opens in a new tab)
Used for: RLHF: fine-tuning a model using human rankings of its outputs.
Checked on , checked by title, authors, date and abstract
McHugh · 2012
Interrater reliability: the kappa statistic (opens in a new tab)
Used for: Measuring how much two raters agree beyond chance (Cohen’s kappa).
Checked on , checked by title, authors, date and abstract
OpenTelemetry · 2026
OpenTelemetry Semantic Conventions for Generative AI (opens in a new tab)
Used for: An open, still-in-development standard for naming GenAI spans, metrics and events.
Checked on , read in full
Andrej Karpathy · 2025
Post introducing the term “vibe coding” (opens in a new tab)
Used for: Origin of the term “vibe coding” (2 February 2025).
Checked on , checked by title, authors, date and abstract