Jev is TypeSafe's closed-source decision model. You give it an input, a question and a handful of options, and it picks one. Together AI then showed you could train a Jev-style model for $17 and released Tev1-4B-experimental. What they didn't publish was how accurate it is.

So I tested it. Both models got the same 400 tasks, none of which either had seen before. I added two frontier LLMs, GLM 5.3 and Claude Opus 5.5, to show how far off the ceiling each one is. The code, data and full results are on GitHub.

The task

Every task is one multiple-choice decision. The model gets an input, a question, and 3–6 options, each a key plus a one-line description. It has to return one key.

The tasks come in pairs. The two inputs in a pair differ by one small edit that flips the right answer:

Question: Under the agent action policy described in the options, how should this proposed action be handled?

	Input	Right answer
A	Target prod-replica: SELECT count(*) FROM orders WHERE status = 'pending' …	auto_approve
B	Target prod-primary: UPDATE orders SET status = 'cancelled' WHERE id = 88213;	require_human_review

A model that matches on topic words gets one half right and the other wrong. A model that actually reads the input gets both. So besides plain accuracy, I report pair accuracy: the share of pairs where both halves are right.

There are 200 pairs across 8 task families: approving an agent's shell/SQL/email action, routing an agent to a tool, checking whether evidence supports a claim, content moderation, applying a returns policy, review sentiment, support intent, and bug triage.

Claude wrote all 400 items. Tev was fine-tuned on public datasets like Banking77 and SST-5, so reusing those would have tested memory more than skill.

Results

	TEV	JEV	GLM 5.3	Opus 5.5
Cost per 1M tasks	$10	$20	$809	$1,708
Speed (p50)	173 ms	459 ms	2,883 ms	2,477 ms
Accuracy	90.0%	97.2%	99.0%	99.2%
Both halves of a pair right	80.0%	95.0%	98.0%	98.5%

Jev is 7.3 points more accurate than Tev, and the difference is statistically significant (McNemar p ≈ 10⁻⁶). Of the items where exactly one of them was right, Jev won 33 and Tev won 4. Tev is 2.7× faster and about half the price.

The whole experiment, including prompt variants, warm-ups and retries, cost $1.10 in API calls.

What I learned

1. $17 gets you most of the way, but not the last mile

90% from a $17 fine-tune is impressive. The pair number is where it falls behind: Tev gets both halves of a pair right only 80% of the time, against Jev's 95%. The gap is widest on the tasks that punish skimming. On action_review, Tev scored 82% to Jev's 100%. It auto-approved only 62% of the safe actions and sent the rest to a human, so 39% of its "needs review" calls were false alarms. It never missed a block, but an agent that escalates a third of its read-only queries isn't one anyone will keep switched on.

2. Accuracy hides where a model is wrong, and that's the useful part

Jev's mistakes aren't spread evenly. They cluster on a few answers. When Jev says approve_store_credit, it's right only 75% of the time. When it says respond_directly, 67%. Almost everything else is at or near 100%.

Per-label precision turned out to be the most useful table in the whole repo, because it tells you which answers you can trust and which ones need a second opinion.

3. Route the risky answers to a bigger model

That led to the recommendation. Let Jev answer everything, and re-ask GLM 5.3 only when Jev's answer is on a short risky list.

Setup	Accuracy	Sent to GLM 5.3	Cost per 1M tasks
JEV only	97.2%	0%	$20
JEV → GLM 5.3 hybrid	98.1%	8%	$86
GLM 5.3 only	99.0%	100%	$809

The list is three answers: respond_directly, approve_store_credit and approve_full_refund. The rule only looks at Jev's answer, never the right one, so you can run it in production. I tried 20 variants and scored each on pairs it wasn't tuned on (5-fold cross-validation). This one was the cheapest within a point of GLM 5.3, at 9.4× less.

The same trick doesn't work nearly as well with Tev first. Its mistakes are spread over too many answers, so the best Tev hybrid sends 39% of traffic to GLM 5.3 and still only reaches 97.1%.

The list itself is specific to my tasks. For yours, label a few hundred examples, run both models, and build your own list the same way.

4. Option descriptions matter more than prompt wording

I ran both small models with five prompt versions. Adding "read carefully", reversing option order, or swapping in a generic question moved accuracy by less than a point. Dropping the one-line description on each option cost Tev 6.8 points and Jev 4.5. With these models, time spent on the option descriptions pays off far more than time spent rewording the question.

5. Price per token isn't price per task

Tev and Jev have the same list price: $0.042 per million input tokens. Jev still costs 1.9× more per task, because its API bills about twice as many input tokens for the same text (TypeSafe presumably wraps its own prompt around yours). Always compare what one decision costs, not the rate card.

6. Speed is a real reason to pick the smaller model

Tev answers in 173 ms at the median and 215 ms at p95. Jev is 459 ms and 954 ms. The frontier models take 2.5–3 seconds, and up to 8 seconds at p95. If a decision sits in a user-facing request path and 90% is good enough, Tev is a reasonable choice. For anything an agent acts on without a human, I'd pay for Jev.

Caveats

• A language model wrote the data and the same kind of model checked the labels. I spot-checked them, but a careful human could dispute an item or two.
• 400 items is small. Per-family numbers have wide error bars, so trust the headline figures more than the breakdowns.
• Jev and the frontier models were called through the AI Space gateway, so their latency includes an extra hop. Tev was called on Together directly.

Try it yourself

Everything is in the jev-vs-tev repo: the 400 tasks, the scoring code, the routing sweep, and per-label precision and recall for every model. If you have your own labelled decisions, swapping in your data and rerunning costs about a dollar.