Decision API benchmark: Jev, Celeris, Clef and OpenAI
2026-10-12 · by Andreas Vilinski · two test sets, run via the jev CLI with one question per call: 330 items from the public jev-bench (15 per dataset) and my own 60 labelled items × 3 runs. OpenAI Decisions = gpt-6-luna on /v1/decisions (public beta); OpenAI chat (DIY) = gpt-4.1-mini read through token log-probabilities
← All charts · Interactive scatter plot: accuracy against cost and speed
Speed
Every call on my 60-item set, then median, tail and batching. First call of each run left out (cold start).
Latency of every call
Each dot is one of the 180 calls per model (log axis, so the fast cluster is spread out and slow outliers stay visible). Bars mark p50 and p95. further left is better
Latency p50
median single call, incl. process start + TLS lower is better
Latency p95
slowest 5% of calls lower is better
10 questions, one request
all questions about one state; OpenAI runs them as 10 parallel calls lower is better
10 questions, 10 calls in sequence
what you pay if you do not batch lower is better
Accuracy on public data: jev-bench sample
Real, public data in the System One format: Praveenrajus/jev-bench (22 datasets such as MNLI, BoolQ, Banking77, MMLU, SST-5), 15 test items per dataset. Same questions to every model, scored by accuracy of the top option and the Brier score.
| Model | Accuracy (95% CI) | Brier | p50 / p95 | Failed or refused |
|---|---|---|---|---|
| Jev | 76.4% (72%–81%) | 0.329 | 465 / 554 ms | 0 of 330 |
| Celeris | 83.9% (80%–88%) | 0.220 | 545 / 661 ms | 0 of 330 |
| Clef | 77.6% (73%–82%) | 0.313 | 656 / 1312 ms | 0 of 330 |
| OpenAI Decisions | 72.1% (67%–77%) | 0.432 | 408 / 528 ms | 1 of 330 |
| OpenAI chat (DIY) | 57.5% (52%–63%) | 0.588 | 766 / 1067 ms | 75 of 330 |
Accuracy per dataset
| Dataset (15 items each) | Jev | Celeris | Clef | OpenAI Decisions | OpenAI chat (DIY) |
|---|---|---|---|---|---|
| arc_challenge | 100% | 100% | 100% | 100% | 100% |
| banking77 | 73% | 80% | 80% | 80% | 1% |
| boolq | 100% | 100% | 93% | 87% | 100% |
| chaosnli | 73% | 87% | 60% | 47% | 60% |
| civil_comments | 80% | 100% | 93% | 80% | 93% |
| clinc150 | 100% | 100% | 100% | 100% | 1% |
| fever_evidence | 93% | 100% | 100% | 100% | 93% |
| go_emotions | 33% | 80% | 73% | 40% | 4% |
| helpsteer2_helpfulness | 33% | 33% | 27% | 33% | 23% |
| helpsteer2_verbosity | 40% | 67% | 47% | 33% | 27% |
| ledgar | 87% | 87% | 87% | 93% | 1% |
| massive | 80% | 87% | 87% | 67% | 2% |
| measuring_hate_speech | 53% | 100% | 67% | 40% | 47% |
| mmlu | 93% | 80% | 93% | 93% | 87% |
| mnli | 93% | 100% | 87% | 93% | 93% |
| paws | 87% | 93% | 73% | 80% | 80% |
| sms_spam | 87% | 93% | 87% | 93% | 87% |
| sst5 | 73% | 47% | 53% | 67% | 63% |
| strategyqa_closed | 100% | 100% | 100% | 90% | 100% |
| strategyqa_grounded | 100% | 93% | 93% | 90% | 87% |
| stsb | 53% | 73% | 47% | 27% | 63% |
| yelp5 | 47% | 47% | 60% | 53% | 53% |
Public benchmark in charts
jev-bench accuracy
330 items: 15 random test items from each of the 22 datasets higher is better
jev-bench Brier score
sum over options of (probability − label)²; lower is better lower is better
Latency p50 on real inputs
inputs from 30 to 2,800 characters lower is better
Cost per 1,000 decisions on this mix
list price × tokens measured on the 330 items lower is better
Cost
Cost per 1,000 decisions
list price × tokens measured in this benchmark, one question per call lower is better
List price per 1M input tokens
output free except OpenAI ($1.60 per 1M) lower is better
Input tokens per call
each vendor counts with its own tokenizer, so this is billing size, not content size
| Model | Input / 1M tokens | Output / 1M tokens | Input tokens per call | Per 1M decisions | Per 1,000 decisions |
|---|---|---|---|---|---|
| Jev | $0.042 | free | 348 | $14.61 | $0.0146 |
| Celeris | $0.040 | free | 146 | $5.82 | $0.0058 |
| Clef | $0.038 | free | 199 | $7.55 | $0.0076 |
| OpenAI Decisions | $0.100 | free | 179 | $17.86 | $0.0179 |
| OpenAI chat (DIY) | $0.400 | $1.60 | 104 | $43.39 | $0.0434 |
Prices are list prices checked on 2026-10-12: Celeris ($0.04), Jev ($0.042, no official pricing page found; taken from third-party write-ups of the launch), Cloudflare clef-flash ($0.038, i.e. 3,455 neurons per M at $0.011 per 1,000; 10,000 neurons per day are free), OpenAI Decisions ($0.10, no output charge), OpenAI gpt-4.1-mini ($0.40 in, $1.60 out). Cost per decision = input tokens measured here × price. The four decision APIs read the state once for N questions in a request; the DIY chat route sends the state again for every question.
Accuracy (my 60-item set)
Accuracy: yes/no
24 items × 3 runs, vague items excluded higher is better
Accuracy: routing
20 items × 3 runs, 6 options incl. “unclear” higher is better
Accuracy: severity (exact level)
16 items × 3 runs, 4 levels higher is better
Severity error
mean distance to the labelled level, in levels lower is better
Calibration and trust
Calibration: yes/no (Brier)
squared error of the probability lower is better
Calibration: routing (Brier)
squared error over all options lower is better
Confidence when right vs wrong
a useful confidence is clearly lower when wrong higher is better solid = right, faded = wrong
Accuracy when confidence ≥ 0.8
how far you can trust a confident answer higher is better
Accuracy when confidence < 0.8
a good model is worse here, so these can go to a human lower is better
Says “can’t tell” on vague input
share of 4 vague yes/no items answered 0.25–0.75 higher is better
Decision flips between runs
items whose decision changed across 3 runs lower is better
Input tokens per call
as reported by the provider; sizes are not comparable across tokenizers
More numbers
| Metric | Jev | Celeris | Clef | OpenAI Decisions | OpenAI chat (DIY) |
|---|---|---|---|---|---|
| Failed calls | 0 / 180 | 0 / 180 | 0 / 180 | 0 / 180 | 0 / 180 |
| Overall accuracy | 92.9% | 93.5% | 96.4% | 96.4% | 95.8% |
| Severity within ±¾ level | 88% | 100% | 94% | 100% | 88% |
| Latency max | 685 ms | 667 ms | 7530 ms | 3329 ms | 1469 ms |
| Run-to-run std dev | 0.003 | 0.005 | 0.000 | 0.000 | 0.001 |
| Output tokens per call | 33 | 0 | 0 | 0 | 1 |
| Items answered with conf ≥ 0.8 | 151 of 168 | 133 of 168 | 78 of 168 | 156 of 168 | 159 of 168 |
Items where a model was wrong (my set, majority of 3 runs)
| Text | Gold | Jev | Celeris | Clef | OpenAI Decisions | OpenAI chat (DIY) |
|---|---|---|---|---|---|---|
| Extraction: 'Meeting moved to Tuesday 3 Nov 2026 at 14:00 CET'. Extracted: 2026-11-03T14:00+01:00. | True | yes (0.89) | yes (0.84) | no (0.21) | yes (0.95) | yes (0.99) |
| Charts render but legends overlap on narrow windows; resizing fixes it. | 1 | L1 · 0.95 | L1 · 0.58 | L1 · 1.00 | L1 · 0.96 | L1 · 1.00 |
| Nobody in the EU region can log in; no alternative path. | 2 | L3 · 2.68 | L3 · 2.66 | L2 · 2.03 | L3 · 2.75 | L3 · 3.00 |
| Invoice PDF generation fails for every customer; finance cannot bill. | 2 | L3 · 2.84 | L3 · 2.64 | L2 · 2.15 | L2 · 2.46 | L2 · 2.32 |
| Mobile app crashes on launch for all iOS 19 users. | 2 | L3 · 2.70 | L3 · 2.65 | L2 · 2.05 | L3 · 2.65 | L3 · 3.00 |
| Customer data from tenant A visible to tenant B. | 3 | L2 · 2.11 | L3 · 2.78 | L2 · 2.03 | L3 · 2.54 | L3 · 2.95 |
Reading the results
Public benchmark. On a 330-item sample of jev-bench, Celeris is clearly the most accurate (84%, 95% CI 80%–88%) and best calibrated (Brier 0.220). Clef (78%) and Jev (76%) follow, then OpenAI Decisions (72%); the DIY chat adapter is far behind because it cannot handle questions with more than 26 options (five datasets). The 95% intervals are about ±4.5 points wide, so Celeris leads OpenAI Decisions clearly, while Clef and Jev are statistically tied.
Speed. Jev (467 ms) and OpenAI Decisions (414 ms) have the lowest median on my own set, Celeris is steady at 535 ms, Clef is slower with a long tail (p95 1572 ms, slowest call 7.5 s). Celeris advertises 67 ms from AWS us-east-1; from here, through the CLI with a new TLS handshake per call, every number is dominated by the network path. The first call of each run is left out as a cold start.
My own 60-item set is easy: all five models are within a few points of each other (93% to 96%), and one answer is worth almost two points. OpenAI Decisions has the best calibration here (Brier 0.0023 yes/no, 0.0026 routing), Jev is the only model that answers every vague input with “can’t tell” (100%), and the DIY chat adapter is nearly as confident when wrong (0.94) as when right (0.97). On the harder public data the ranking changes, which is the reason to trust the larger set more.
Cost. All four dedicated decision APIs cost between $0.0058 and $0.0179 per 1,000 single-question decisions on my short inputs, and $0.0144 to $0.0417 on the longer public-benchmark inputs, so a million decisions costs roughly $6 to $42. OpenAI Decisions is priced at $0.10 per million input tokens with no output charge; the DIY chat route costs about 7x Celeris on short inputs. Cost is a minor factor at small scale; accuracy and calibration matter more. Tokens are counted by each vendor’s own tokenizer.
Caveats. The public sample is 15 items per dataset, one run, so differences under about 5 points are inside the error margin. My own set is synthetic, small and easy, with 3 repeats. Latency comes from the jev CLI on one machine. Severity labels on my set are my own judgement. Scoring on jev-bench counts a failed or refused request as a uniform answer. jev-bench is published by a third party and has a mixed licence; I used only a small sample of its test splits. The OpenAI Decisions API is in public beta. No sponsorship or affiliation with any of these vendors.