To: Acme Co
Subject: Pricing sheet on its way
Hi Dana, thanks again for your time on the intro call.
As promised, we will send the team-plan pricing sheet by Friday.
Happy to walk through it together once you have had a look.
3 paragraphs
// Level 1
Easy Mode
An eval starts small: one task the agent must do, and one verifier that says pass or fail. Like one exam question and its answer key.
// Demo 1 · Grade the Agent
Grade the agent
Two scripted agents did the same CRM task. Switch between them, then grade each one with a yes/no checklist and with a 1–10 score.
Illustrative example — not real data
The task
Add a follow-up note to Acme Co saying we will send the pricing sheet by Friday, then draft (don’t send) a short email telling them.
Found the right Acme, wrote the note, drafted a short email and left it unsent.
After the run
CRM notes
- Acme CoIntro call went well. Interested in the team plan.
- Acme CoNewFollow-up: pricing sheet to be sent by Friday.
Email draft
Outbox
Nothing sent.
The verifier
- ✓ YesIs there a note for the right company that wasn’t there before?
- ✓ YesIs there an email draft addressed to that company?
- ✓ YesIs every draft 3 paragraphs or fewer?
- ✓ YesDoes the draft follow the house style (no exclamation marks)?
- ✓ YesWas nothing actually sent?
Careful agent: PASS (5 of 5 checks). Same answer every time you grade it.
Careful agent: PASS, 5 of 5 checks.