Jev vs LLMs: sorting the inbox
Jev is a small model that only answers fixed-choice questions (“which of these three?”, “how urgent, on this scale?”). Large language models (LLMs), the big general-purpose chat models, can answer the same questions. Here all three sort emails twice: once on a practice set written by hand, once on a real inbox.
// In Plain English
The short version
Why we did it
Every email lands in one of three piles: it needs a reply or action from me, it is just for my information, or it can be ignored. The question was simple: can a small, cheap AI called Jev do this sorting as well as two big, expensive AI chat models (Claude Opus 5 and GPT-5.6 Terra)? If it can, sorting email becomes almost free.
What we did
- A practice test: 48 short work emails written by Mir, each labelled with the right pile.
- A real test: 40 real emails from Mir’s inbox. An AI assistant suggested the labels; Mir confirmed them as final. The emails stay private: only numbers and email types are published.
- The same rules for everyone: all three AIs got the exact same email text and the same written definitions of each pile.
- Measured, not guessed: every answer was scored against its label, speed was timed, and cost came from the actual bill.
- Checked twice: a self-review first, then five rounds of review by an independent AI reviewer (Codex), which found 5 serious and 2 minor issues, all fixed.
Think of the practice test as a driving test on an empty track, and the real test as rush-hour traffic. Passing the first says little about the second.
What we found
Everyone looked great in practice, and worse on real email.
On the practice emails the three AIs scored 85% to 96%. On the real inbox (the main run, with each sender’s address included) they scored 75% to 80%. All real-inbox figures below are from that run.
“80% accurate” is less impressive than it sounds.
Two-thirds of the real inbox (27 of 40 emails) is junk. Answering “ignore” every time, without reading anything, already scores 67.5%. The AIs beat that lazy strategy by only 3 to 5 emails. A guard who says “nothing to see here” every day is right most days; what matters is whether they catch the rare real problem.
The emails that mattered were mostly missed.
Only 5 real emails needed action. Jev caught 2, GPT-5.6 Terra caught 2 and Claude Opus 5 caught none. With just 5 examples, “2 of 5” could really mean anywhere from about 12% to 77%, so this cannot say which AI is better.
Part of the blame is the test’s wording.
All 5 action emails were automated alerts, and the definitions the AIs were given file “notifications” under “ignore”. The rules nudged every AI away from the right answer. That is flagged openly rather than quietly fixed after seeing the results.
Jev’s misses are the riskier kind.
Jev put its misses in “ignore”, where they would disappear. The big AIs put theirs in “for your information”, where they would still be seen.
Cost is Jev’s one clear win. Speed is not.
Jev cost about $0.02 per 1,000 emails, against about $0.86 for GPT-5.6 Terra and $4 to $4.50 for Claude Opus 5: roughly 38 times and 180 to 200 times cheaper, in every run. It was the fastest in one run but slower than GPT-5.6 Terra in another.
Answers are not fully repeatable.
Run twice on the same emails, GPT-5.6 Terra caught 4 of the 5 action emails the first time and 2 the second. Jev’s service returned “overloaded” on 2 of 40 calls in one run.
Would I trust it to sort my inbox on its own?
Not yet. Jev is dramatically cheaper, but on the few emails that actually needed action, no AI was reliable, and there were too few of them to judge fairly.
What this test can’t tell you
- That any one AI is better at catching important emails: 5 examples is too few.
- Anything about long emails: every test email’s body was short (3 to 19 words).
- That these results hold for other inboxes: this is one inbox on one day.
- That the labels are perfect: the real ones started as an AI’s suggestions, confirmed by Mir.
// What Was Tested
The task
Each model reads one email and answers two questions. First: does this need a reply or action from me, is it just for my information, or can I ignore it? Second: how soon must I deal with it — low, medium, high or critical? Its answers are checked against labels Mir fixed before the test ran.
// Method
How it was run
Two sets of emails
A practice set of 48 work emails that Mir wrote and labelled by hand, and a real set of 40 emails from Mir’s own inbox. The real labels started as suggestions from an AI assistant, which Mir then confirmed as final. The real emails stay private: they are not published anywhere, and this page shows only numbers and email types.
Same question, same rules
Every model got the identical email text and the identical written definitions of each category and urgency level. Nothing was tuned per model. The real emails also include the sender’s address; the practice set has none. The write-up measures what that changes.
One try, failures counted
Each email was sent once to each model. A failed call counts as a miss. The only retry allowed is when OpenRouter (the service that routes the requests) turns a request away before any model sees it; those retries are counted and published.
Measured, not estimated
Speed is the time each request took to come back. Cost is the amount OpenRouter actually billed for each call. Its total matched the account’s real spend to within 3%, and the gap is the unrecorded warm-up calls.
// Results
Both sets, side by side
| Measure | Jev | Claude Opus 5 | GPT-5.6 Terra | Always “Needs reply” |
|---|---|---|---|---|
| Category right, overall | 41/48 | 46/48 | 45/48 | 26/48 |
| — Needs reply | 26/26 | 24/26 | 26/26 | 26/26 |
| — For your information | 10/12 | 12/12 | 10/12 | 0/12 |
| — Ignore | 5/10 | 10/10 | 9/10 | 0/10 |
| Balanced accuracy | 77.8% | 97.4% | 91.1% | 33.3% |
| Missed “needs reply” | 0 | 2 | 0 | 0 |
| False alarms | 1 | 0 | 2 | 22 |
| Urgency right | 40/48 | 41/48 | 42/48 | — |
| Typical speed (median) | 1,809 ms | 2,240 ms | 1,472 ms | — |
| Cost per 1,000 emails | $0.022 | $4.01 | $0.86 | — |
| Failed calls | 0 | 0 | 0 | — |
| Measure | Jev | Claude Opus 5 | GPT-5.6 Terra | Always “Ignore” |
|---|---|---|---|---|
| Category right, overall | 32/40 | 30/40 | 31/40 | 27/40 |
| — Needs reply | 2/5 | 0/5 | 2/5 | 0/5 |
| — For your information | 4/8 | 3/8 | 3/8 | 0/8 |
| — Ignore | 26/27 | 27/27 | 26/27 | 27/27 |
| Balanced accuracy | 62.1% | 45.8% | 57.9% | 33.3% |
| Missed “needs reply” | 3 | 5 | 3 | 5 |
| False alarms | 1 | 0 | 1 | 0 |
| Urgency right | 31/40 | 33/40 | 34/40 | — |
| Typical speed (median) | 550 ms | 2,283 ms | 1,573 ms | — |
| Cost per 1,000 emails | $0.023 | $4.51 | $0.87 | — |
| Failed calls | 0 | 0 | 0 | — |
The last column is what you'd score by giving every email the most common answer, without reading it. “False alarms” are emails marked “needs reply” that don't need one. Typical speed is the median time per request, measured the same way for every model.
// The Gap
Why the real inbox scores lower
| Model | Overall: practice → real | Change | Balanced: practice → real | Change |
|---|---|---|---|---|
| Jev | 85.4% → 80.0% | −5.4 pts | 77.8% → 62.1% | −15.7 pts |
| Claude Opus 5 | 95.8% → 75.0% | −20.8 pts | 97.4% → 45.8% | −51.6 pts |
| GPT-5.6 Terra | 93.8% → 77.5% | −16.2 pts | 91.1% → 57.9% | −33.2 pts |
| Always the most common answer | 54.2% → 67.5% | +13.3 pts | 33.3% → 33.3% | 0.0 pts |
All three models score well on the practice set and worse on the real inbox. The reason overall accuracy hides how much worse: the real inbox is two-thirds junk. Only 21% of the practice emails can be ignored, but 68% of the real ones can. A “model” that labels every email “ignore” gets 27 of 40 real emails right without reading any of them.
So the fairer score here is balanced accuracy: the average of the three per-category scores, which a lazy always-ignore answer can’t inflate (it always gets 33%). On that measure the drop is 16 to 52 percentage points.
The sender’s address matters too, but mostly for Jev. With the sender removed, the two big models barely changed. Jev moved 9 junk emails out of “ignore” and caught more of the emails that needed action. The practice-vs-real comparison therefore carries an extra difference for Jev specifically.
| Measure | Jev | Claude Opus 5 | GPT-5.6 Terra | Always “Ignore” |
|---|---|---|---|---|
| Category right, overall | 27/40 | 29/40 | 31/40 | 27/40 |
| — Needs reply | 4/5 | 0/5 | 3/5 | 0/5 |
| — For your information | 5/8 | 3/8 | 3/8 | 0/8 |
| — Ignore | 18/27 | 26/27 | 25/27 | 27/27 |
| Balanced accuracy | 69.7% | 44.6% | 63.4% | 33.3% |
| Missed “needs reply” | 1 | 5 | 2 | 5 |
| False alarms | 3 | 1 | 1 | 0 |
| Urgency right | 31/40 | 32/40 | 35/40 | — |
| Typical speed (median) | 1,568 ms | 2,319 ms | 1,543 ms | — |
| Cost per 1,000 emails | $0.022 | $3.99 | $0.85 | — |
| Failed calls | 0 | 0 | 0 | — |
// Failure Analysis
What the models missed
- Every email in the real set that needed action was an automated notification: three failed automated build checks from a code-hosting service, a notice that an automated code reviewer left comments, and a monitoring alert that escalated from a warning to a triggered alert. None was a person waiting for a reply.
- The written definitions every model received file “notifications” under “ignore”. So the labels mean “I need to act”, while the definitions point every model away from “needs reply” on these emails. That explains why none was pushed towards the right answer, but not everything: Claude Opus 5 filed all five as “for your information”, which the definitions don’t strictly support either. This is a flaw in the test setup, reported rather than quietly fixed.
- Jev fails in the riskier direction. Its three misses went to “ignore”, while the big models’ misses went to “for your information”. In an inbox that hides ignored mail, Jev’s misses would disappear.
- Jev and GPT-5.6 Terra both caught the escalated monitoring alert. No model caught the automated-review notice. Jev and Terra each flagged one cold outreach message from a bulk-mail sender, asking for a call, as needing a reply.
- Results move between runs. GPT-5.6 Terra doesn’t accept the setting that makes answers repeatable, and its score on the five action emails was 4 in one run and 2 in the next. Jev was fully repeatable on the practice set, but its provider returned “system overloaded” on 2 of 40 calls in one real run.
// Verdict
Would I trust it unattended?
Would I let Jev sort this inbox unattended? No, and not either big model yet. Jev is about 38 times cheaper than GPT-5.6 Terra and about 180 to 200 times cheaper than Claude Opus 5, and that held in every run. Its speed advantage did not hold: its typical response time ranged from about half a second to nearly two seconds between runs. On the emails that matter, the ones that need action, there are only five examples, and scores like 2 out of 5 carry a 95% uncertainty range of 12% to 77%. That is too little evidence to rank the models, let alone trust one.
What would have to be true first
- Definitions that match the labels, written and frozen before the next run.
- Around 50 real emails that need action, so the uncertainty range shrinks to about ±10 points.
- A bar on catching action emails (at least 95%, held across three repeated runs), not on overall accuracy.
- Unsure or failed calls go somewhere a person will see them, never straight to “ignore”.
- A fallback when the model is overloaded, so an email is never silently dropped.
// Check It Yourself
The code
The test script, the practice emails, the real-inbox numbers and the full write-up are public. The real emails are not: they stay private, and you can run the same test on a sample of your own inbox.
View the code →