Evidence
Jev benchmarks: what the numbers actually show
The speed and cost advantages hold up across independent tests — one to two orders of magnitude, repeatedly. The accuracy story is more complicated, and the headline claim rests on a definition worth reading carefully.
Vendor-reported numbers
A custom TypeSafe evaluation across four workflows — security incident response, agent-trace observability, invoice processing and customer service — scoring each model against the average predictions of GPT-6 Astra and Anthropic Fable.
| Model | Accuracy | Cost / case | Latency |
|---|---|---|---|
| Jev | 67.8% | $0.0004 | 0.4s |
| GPT-5.6 Terra | 67.9% | $0.0304–$0.1761 | 10–38s |
| Opus 5 | 73.1% | $0.0304–$0.1761 | 10–38s |
| GPT-5.6 Sol | 74.1% | $0.0304–$0.1761 | 10–38s |
TypeSafe's headline framing: 193.6x faster and 444.6x cheaper than frontier LLMs in peak in-house testing. Latency is quoted as 70–500ms end-to-end against 3–329 seconds for frontier models, with output tokens priced at zero — “too cheap to meter.”
Independent reproductions
These matter more. Jev is days old, so the sample is small and mostly single-author — but the direction is consistent.
Phishing classification — 2,000 emails vs Claude Haiku 4.5
Jev's raw verdict loses clearly on accuracy against Claude Haiku 4.5 (McNemar p < 0.0001), while winning decisively on speed and cost.
The interesting part: The striking result: a cross-validated logistic regression over five signal questions asked in the same Jev call reached 95.1% accuracy, AUROC 0.988, ECE 0.027 — i.e. asking several cheap questions and combining them beat asking one big one.
Caveat: Latency was wall-clock measured from a machine in France against US-hosted services.
Passage extraction
Found Jev roughly 25x faster and 580x cheaper than Claude Fable 5.1 — 0.35s versus 8.83s per passage.
Caveat: Speed/cost oriented; not an accuracy study.
LLM model routing
Replacing an LLM classifier with Jev's Choice primitive was faster than either original configuration and significantly cheaper.
Caveat: Used one sample per tier, so it is not an accuracy verification and says little about borderline cases.
Agent tool-call risk classification
A reproducible harness measuring Jev on tool-call risk classification, including whether the confidence score is worth routing on.
Caveat: Measures Jev alone — provider adapters exist but no API key was available, so comparison columns are empty. The author explicitly states it neither supports nor refutes the vendor multipliers.
The most useful finding so far
Buried in the phishing benchmark is the single most actionable result in the whole Jev corpus. The model's own one-shot verdict lost to Claude Haiku 4.5. But asking five orthogonal signal questions in the same call and feeding those probabilities into a plain logistic regression reached 95.1% accuracy with an AUROC of 0.988.
That reframes how to use Jev. It is not “a classifier you query once” — it is a cheap, calibrated signal generator you query broadly, because the marginal cost of another question is close to zero. Ask ten questions and do the combining yourself.
How to read any Jev benchmark
- Check what accuracy means. Agreement with other models is not ground truth.
- Check where latency was measured from. The phishing benchmark timed from France to US-hosted endpoints — real network, not a fair model-to-model comparison.
- Check the sample size. The routing test used one sample per tier. Directional, not conclusive.
- Check whether a comparison actually ran. The tool-call risk harness has empty comparison columns because no API key was available — its author says so explicitly.
- Then run your own. On your data, the answer may differ, and at this price testing is nearly free.