---
title: "DecideBench v2: How Well Can Decision Models Replace Frontier LLMs in Judgements?"
description: "DecideBench v2 runs 26 decision models (JEV, Drex Max, REAX 1, pplx-decider, Clef, GPT-6 Luna and open alternatives) through 450 production-format decisions that Claude Opus 4.8 gets 99.3% right. The best scores 94.4%."
canonical: "https://choy.in/blog/decidebench-v2"
author: "Cho Yin Yong"
published: "2026-11-09"
last-updated: "2026-11-09"
category: "AI"
reading-time: "8 min read"
keywords: "decidebench, decision model benchmark, jev alternatives, drex max, reax 1, pplx-decider, clef, gpt-6 luna decisions, self-hosted decision model, llm benchmark, claude opus 4.8"
---

# DecideBench v2: How Well Can Decision Models Replace Frontier LLMs in Judgements?

> DecideBench v2 runs 26 decision models (JEV, Drex Max, REAX 1, pplx-decider, Clef, GPT-6 Luna and open alternatives) through 450 production-format decisions that Claude Opus 4.8 gets 99.3% right. The best scores 94.4%.

When I [benchmarked the open-source JEV alternatives](/blog/open-source-jev-alternatives) in September, [DecideBench](https://github.com/choyiny/decidebench) was a side effect: 400 decisions I needed to compare a dozen models fairly. Since then I've kept adding models to it, and by v1.1 most of them scored above 90%. Drex Max, REAX 1, s1-pro and JEV were within a point of each other. A benchmark where everything passes can't tell you which model to pick.

So I rebuilt it. DecideBench v2 has 450 new items, written the way a decision model actually sees its input in production. I ran 26 decision models through it, plus Claude Opus 4.8 as a reference.

The short version: Opus 4.8 gets 99.3% right. The best decision model, Nace.AI's **Drex Max**, gets 94.4% at about 1/290 of the cost. JEV, which scored 98.0% on v1, scores 86.2%.

## What changed in v2

v1's inputs were short summaries: "a customer asks for a refund 31 days after purchase". In production, nobody hands a decision model a summary. It gets the runtime's JSON payload for a tool call, a raw email with its headers, a release controller's metrics, or a retrieval result with its sources. v2 is written in those formats, and when the decision depends on a written policy, the full policy is in the question.

That makes the inputs about three times longer (a median of 893 characters, against 279 in v1), and items now have 3 to 10 options instead of 3 to 6. There are 9 task families:

- approving an AI agent's proposed tool call
- routing an analytics agent to a tool
- checking a drafted claim against its sources
- moderating a community post
- ruling on a refund under a written policy
- gating a staged release (new)
- tagging a customer's intent
- triaging an inbound email (new)
- setting a bug report's severity

The items still come in pairs. The two halves of a pair differ by one small, realistic edit that changes the right answer, like a file attached to the chat instead of a database table, or a sponsorship pitch instead of a bug report from the same customer. A model that matches on topic gets one half wrong, so I report **pair accuracy** (both halves right) next to plain accuracy.

The other change is how I set the bar. v2 is built so that a frontier LLM solves it. Claude Opus 4.8 answers 447 of the 450 items correctly, and of its three misses, one has an arguable label. That turns the question into one of price, not capability: Opus costs about $26,500 per million decisions here and takes 1.3 seconds each. How close do models that cost 100 to 1,000 times less get?

I also tried Claude Opus 5.5 as the reference. Anthropic's cyber safeguard declined 7 of the agent-routing items before it answered, and it got 442 of the other 443 right. I kept Opus 4.8, which answered everything.

## The results

![Cost vs accuracy on DecideBench v2. Claude Opus 4.8 sits at 99.3% for $26,543 per million tasks. Drex Max is the best decision model at 94.4% for $91, followed by REAX 1, pplx-decider self-hosted and s1-pro around 92%, and JEV at 86.2% for $75.](https://choy.in/content/blog/decidebench-v2-cost-vs-accuracy.png)

| Model | Hosting | Accuracy | Pair accuracy | Cost per 1M decisions | Latency p50 |
|---|---|---:|---:|---:|---:|
| Claude Opus 4.8 (reference) | API | **99.3%** | **98.7%** | $26,543 | 1,336 ms |
| Drex Max (Nace.AI) | API | 94.4% | 89.3% | $91 | 258 ms |
| REAX 1 27B (Decision Models) | API | 92.4% | 87.1% | $120 | 545 ms |
| pplx-decider v1.2 27B (Perplexity) | self-hosted | 92.2% | 86.2% | $386 | 1,528 ms |
| s1-pro (Decision Models) | API | 91.6% | 85.3% | $121 | 418 ms |
| JEV (TypeSafe) | API | 86.2% | 72.9% | $75 | 334 ms |
| Clef 27B (Cloudflare) | self-hosted | 85.3% | 72.4% | $216 | 928 ms |
| GPT-6 Luna (OpenAI Decisions) | API | 84.0% | 72.0% | $140 | **196 ms** |
| s1-fast (Decision Models) | API | 82.0% | 68.9% | **$30** | 238 ms |
| Drex 2.0 (Nace.AI) | API | 80.9% | 65.8% | $87 | 264 ms |
| yev1-4b (XY Space) | self-hosted | 79.3% | 62.7% | $71 | 275 ms |
| OpenJev Flash 9B | self-hosted | 75.3% | 55.6% | $41 | 174 ms |
| imajev-4b | self-hosted | 74.0% | 53.8% | $58 | 248 ms |
| Random guessing | – | 25.1% | 4.4% | – | – |

That's the top half. The [README](https://github.com/choyiny/decidebench) has all 26 entries, per-family and per-option-count breakdowns, and the [leaderboard](https://huggingface.co/spaces/choyiny/decidebench-leaderboard) lets you sort them. Hosted entries were timed from one client machine, network included. Self-hosted entries ran on an NVIDIA RTX PRO 6000 Blackwell, priced by GPU time at $3.36 an hour, and were timed on the GPU box itself, so their latency isn't directly comparable.

## What stood out

### The gap is real, and it's in the rules

No decision model matches Opus 4.8 yet. Drex Max comes closest, 4.9 points behind, and that is still 22 more wrong answers out of 450.

Most of those misses come from two families. On routing, intent and moderation, the top four decision models score 96–100%, level with Opus. On **rollout gating**, Opus scores 100% and no decision model scores above 74%. Those items need the model to work out a crash-free rate or an error ratio from the controller's numbers, then check it against an ordered list of rules, where the first rule that applies decides. Refunds under a written policy, which mix dates and amounts, are the next widest gap.

The per-option breakdown makes it concrete. With worked examples, none of the ten most accurate decision models picked the right answer on a single `pull_update` item, and JEV recognised only 2 of the 6 rollbacks. If a decision model is choosing whether to ship your release, check what it does when the answer is "stop".

### JEV dropped 12 points

On v1, JEV scored 98.0%. On v2 it scores 86.2%, and only 72.9% of pairs fully right. Of the four models that scored 98% or more on v1, it fell furthest: Drex Max lost 4.4 points, REAX 1 6.1 and s1-pro 6.6. It's still on the cost frontier at $75 per million decisions, but Drex Max is 8 points more accurate for $16 more. It stays at 94–96% on agent actions, routing, claim checking, moderation and intent, and drops to 76% on refunds, 78% on bug severity and 60% on release gating.

### An open model ties the second-best API

Perplexity's **pplx-decider 27B** scores 92.2% self-hosted, next to REAX 1 and ahead of s1-pro. It's the most accurate model you can run on your own GPU, so the data never leaves your hardware.

It's not cheap to run naively, though. Perplexity ships it as a Python library, not a server, so I wrote a small wrapper that puts it behind JEV's API. Unbatched, it cost $875 per million decisions at 3.4 seconds each. Batching queued requests into one call cut that to $386 and 1.5 seconds, with the same accuracy within one item. If you self-host it, batch.

### The provider can matter as much as the weights

Cloudflare's Clef scored 68.4% (Clef 27B) and 57.6% (Clef-Flash 9B) on v2 through AI Space. Running Cloudflare's published weights myself, the same models scored 85.3% and 66.9%. I don't know what AI Space does differently, but 17 points is too big to be noise, so v2 reports the published weights. If a hosted model disappoints you, try the weights before you rule out the model.

### Last time's pick dropped 21 points

In September I said I'd start with [imajev-4b](https://huggingface.co/mohit67890/imajev-4b) if I needed open weights. It scored 95.0% on v1. On v2 it scores 74.0%. Most of the small open models lost 16 to 25 points going from summaries to real inputs, and Kev and Decider-2B lost more than 30. On v1, imajev-4b was 3.8 points behind Drex Max. On v2 it's 20.4 behind. A model that looks close on a clean test can be far off on what your system actually sends it.

### The encoders are near chance

Laya, GLiNER2.5-Decide, CLM and Julia-1 score 26–41%, against 25% for random guessing. They were already weak on v1. With three times the input and up to 10 options, they mostly pick up the topic and guess.

![Latency vs accuracy on DecideBench v2. GPT-6 Luna on OpenAI Decisions is the fastest hosted entry at 196 ms; Drex Max answers in 258 ms at 94.4%.](https://choy.in/content/blog/decidebench-v2-latency-vs-accuracy.png)

## Which decision model I'd use now

For decisions an agent acts on without a person checking, I'd use **Drex Max**. It's the most accurate decision model, it's on both the cost and latency frontiers, and at $91 per million decisions it costs a rounding error next to the LLM it's trying to replace. For a release gate or anything with arithmetic in the policy, I'd still send it to a frontier LLM.

If the data can't leave my hardware, **pplx-decider 27B** behind a batching server. If I needed a small model that fits on a cheaper GPU, I'd look at yev1-4b (79.3% at $71) or OpenJev Flash 9B (75.3% at $41 and 174 ms). If cost per decision is the only thing that matters, **s1-fast** gets 82.0% for $30.

Whichever you pick, test it on your own policy-heavy decisions, not your routing ones. Routing is where every model looks good.

## What's not in v2, and the caveats

Some v1 entries didn't make it. Drex 1.5 is gone: Nace.AI now serves Drex Max under its name, and all 450 answers matched. Quyet-1.0-Large was retired by its provider. The Jeff models' pinned code is gone upstream. JEV without examples couldn't run because AI Space had no gateway credits for it during the runs. I also left out the general-purpose LLMs other than the reference; they were the ceiling on v1, and Opus 4.8 is the ceiling now.

A few caveats:

- Claude wrote the items, the worked examples and the policies they cite. Every label was checked by Claude Opus, Sonnet and Haiku before the runs, but a careful reader could still dispute a few.
- 450 items is small, and each family has only 50. Trust the overall numbers more than the per-family ones.
- Self-hosted models ran one at a time at 4 requests in flight. A busier server would cost less per decision.
- The items are public, so a model trained on them later may score higher. Every line carries a canary string so they can be filtered out of training data.

## Try it yourself

The 450 items, the 382 worked examples, every model's raw answers and the scoring code are on [GitHub](https://github.com/choyiny/decidebench), and the dataset is on [Hugging Face](https://huggingface.co/datasets/choyiny/decidebench) under CC BY 4.0. v2 is now the default config; v1 still loads as `v1`, and the leaderboard keeps a v1 tab. If you maintain a decision model I haven't covered, open an issue and I'll run it.