Decision Models: Accuracy vs. Cost and Speed
Typed yes/no, choice and score decisions from five hosted models on 330 items of the public jev-bench. Bubble area is proportional to median latency, or to cost when latency is on the x-axis. Measured 2026-10-12 · full report
Method and data
Same questions to every model through the jev CLI (one question per request, sequential). Accuracy counts the top-probability option; Brier is the sum over options of (probability − label)²; a failed or refused request scores as uniform, following the method published with Celeris-1. Prices: Celeris $0.04, Jev $0.042, Cloudflare clef-flash $0.038, OpenAI Decisions $0.10 per 1M input tokens (output free); gpt-4.1-mini $0.40 in, $1.60 out. Latency includes process start and a fresh TLS connection per call, so it is higher than vendor-published figures. Full tables, per-dataset results and caveats: /jev-benchmark/.