Six Open Jev Clones, Four Different Jev Scores

TypeSafe AI opened early access to Jev on 15 September 2026, a decision-only model that returns typed choices with probabilities instead of text, priced at $0.042 per million input tokens with output unmetered. Within 48 hours, six open reproductions had shipped. AINews counted them on 19 September: Laya, SemIf, Bespoke Nimble, Kev-0.5B, Jevlike and DiffusionGemmaJev, built on five different architectures by five unrelated teams, most of them in a single day.

Every one of them publishes a number showing it matches or beats Jev. The problem is that Jev does not have one score. Four independent evaluations of the same model on the same public benchmark, Banking77, put its accuracy at 87.0%, 83.2%, 77.8% and 76.3%. That is a spread of 10.7 points on a 77-label classification task that has been standard since 2020. The clone wave is not a race to reproduce a model. It is six teams reproducing six different targets.

Six reproductions in forty-eight hours

The reproductions divide cleanly by what their authors assumed Jev is. Two treat it as an encoder problem, three as a fine-tuned decoder, one as a diffusion model. None of them know, because Jev's weights and architecture are closed.

The cheapest of these to reproduce is instructive. Kev-0.5B was trained on an Apple M5 laptop in about 1.75 hours, drawing 30 to 40 W for roughly 0.06 kWh, on 9,000 records drawn from six public datasets. It scores 0.799 accuracy on its held-out split and serves TypeSafe's public /v1/systemone API contract, so it is a drop-in for code already written against Jev. Its author, Jared Palmer, is VP of AI at Vercel, which is also the gateway that reported Jev as its fastest-adopted model ever at roughly 13% of teams on day one.

Jev's own score moves by 10.7 points depending on who measures it

Banking77 is the one benchmark all of these evaluations share. It is 77 intent labels over customer banking queries, public since 2020, with a fixed test split. There is no legitimate reason for four teams to get four different numbers from the same closed API. They did anyway.

The two most rigorous entries are the two lowest. The nibzard benchmark froze its protocol before testing, published raw logs, took no vendor sponsorship and included negative results, and it puts Jev mid-pack at 76.3%, behind gpt-oss-120b at 81.3% and glm-5.3 at 80.4%. The ickma2311 evaluation pre-registered two experiments with formal kill criteria and sha256-pinned documents before collecting data, and returned an explicit verdict of AMBIGUOUS on both. The highest number, 87.0%, comes from the comparison table of a competing product.

Two things about Jev do replicate cleanly across evaluators, which makes the accuracy gap more conspicuous rather than less. Latency agrees: nibzard measures p50 at 264 to 276 ms, Convai measures 236 to 276 ms. Calibration error agrees exactly: both independently report an expected calibration error of 0.246. The disagreement is confined to accuracy, which is the only figure any of the clones advertise.

Every clone benchmarked against a different Jev

Because the target moves, each reproduction picked the version of Jev that suited its own evaluation, and each one wins on the benchmark it built.

Laya reports 0.766 against Jev's 0.727 on a 2,000-item typed-decisions set, 0.950 against 0.910 on AG News, and 0.595 against 0.480 on DAIR Emotion. Bespoke Nimble reports 90.1% against Jev's 93% on its own curated evaluation, taking base Qwen from 66%, at 115 ms on an H100. SemIf reports 84.5% balanced accuracy for its 4B build against a published Jev figure of 88.3%. Kev-0.5B does not benchmark against Jev at all and reports only its own held-out 0.799.

Three of those four comparisons use an evaluation set the comparing team built itself. None of the four use the same one. A creator reading any single model card would conclude that reproduction is roughly solved, at between 94% and 97% of Jev's quality. Reading all four together, the honest summary is that nobody has measured the same thing twice.

A 2023 encoder beat all of them

The pre-registered evaluation included a control that none of the clone announcements mention. Alongside Jev, a nano-class LLM and a frontier LLM, it tested a frozen bge-small-en-v1.5 encoder with logistic regression fitted on the 10,003 Banking77 training examples. That is a 2023-era embedding model and a scikit-learn classifier, with no decision-model architecture of any kind.

The encoder wins by 10.1 points over Jev and 5.8 over a frontier model, because it is the only entrant that saw labelled training data for the task. That is exactly the situation most production classification work is in. If you have a few thousand labelled examples of your own routing decisions, the evidence here says fit a classifier on them before you shop for a decision model. The zero-shot column is where Jev and the clones are genuinely competitive, and it is the column the marketing does not distinguish.

The variable nobody controls for is the label count

Read the tables by number of options and the disagreement largely resolves. Every flattering result in this category is measured at two to six labels. Every unflattering one is measured at 77 or more.

Laya scores 0.950 on AG News, which has 4 labels, and 0.595 on DAIR Emotion, which has 6. On Banking77, with 77 labels, its own card reports 0.425 against Jev's 0.870, and calls it a known limitation. That is the single largest gap in any table published this week, and it runs against the clone.

Jev degrades more gracefully but has a hard ceiling. The nibzard cardinality sweep ran from 2 to 512 options and found Jev's latency flat across 2 to 255, which is the architectural claim working as advertised. At 256 options the API returns 400 Too many choices and stops. Every LLM in the comparison handled 512. Jev also changed 13% of its choices when option order was permuted, against 37% for the worst LLM, so position bias is real but comparatively mild.

The practical reading: this class of model is strong in the regime it was designed for, which is a handful of options decided fast and cheaply, and it either collapses or refuses outside it. Count your labels before you read anyone's benchmark.

Calibration is the weakest measured claim, not the strongest

Calibrated probability is the category's core pitch, and it is the attribute that survives independent measurement least well. The nibzard benchmark puts Jev's expected calibration error at 0.246, the worst in a field where the LLMs range from 0.039 to 0.122. On forced-uncertainty items, where the correct behaviour is to decline, Jev admitted ignorance 49.7% of the time against 97.3% to 100% for the LLMs.

The open models are candid that calibration is a post-hoc fix rather than a property of the architecture. Convai states that Laya ships intentionally overconfident, and that fitting a single scalar temperature per question type on your own distribution moves expected calibration error from 0.466 to 0.081. Kev-0.5B reports 0.065 falling to 0.031 after temperature scaling at T=1.47. Both numbers beat Jev's 0.246, and both require you to hold out your own labelled data to fit the temperature. Calibration is not something you download. It is something you fit.

How to pick one for a real pipeline

If you are wiring a classifier, router or guardrail into a content or agent pipeline, the published numbers cannot make this decision for you. A short evaluation can, and every model above is cheap enough to test.

1. Count your labels first. Under about 10, the whole category is viable. Over 50, assume every headline benchmark you have read is inapplicable and test directly. Over 255, Jev is ruled out by the API.
2. Check whether you have labelled data. If you can assemble even a few thousand examples, fit a supervised encoder as your baseline before anything else. It won by 10.1 points in the only evaluation that tried it.
3. Build one evaluation set from your own traffic, a few hundred items, and run every candidate against it. This is the step the entire clone wave skipped, and it is why six models all claim to have won.
4. Measure calibration on your data, then fit temperature. Both open models document the procedure and both land near 0.03 to 0.08 afterwards. Out of the box, treat every confidence score as uncalibrated.
5. Price the floor, not the model. Jev costs $0.07 per 1,000 decisions against $0.19 for the cheapest LLM tested. Laya, SemIf, Nimble and Kev are self-hosted, so the comparison is your GPU time, and Kev's full training run cost 0.06 kWh.
6. Permute your option order once and see how many answers change. It took Jev 13%, and it is a five-minute test that no model card runs for you.

For local work, Kev-0.5B and the 639 MB SemIf build are the two that genuinely run on a laptop, and SemIf's desktop default is MiniCPM5-2B, which is already a known quantity for local agent work. Laya is the fastest at 32.8 ms p50 and the only one with a 51-language sweep. Nimble is the strongest on its own evaluation and the heaviest to serve.

Frequently asked questions

What is a System One model?

A model that gives up text generation entirely and returns a typed decision with a probability instead: a choice among supplied options, a boolean, or a rank on a rubric. Because the output space is constrained to the options you pass in, it cannot hallucinate a value outside them. Jev is the first commercial model marketed as this class, covered in our breakdown of its accuracy claims.

Are any of the Jev clones actually usable in production?

Laya, SemIf, Bespoke Nimble and Kev-0.5B all ship weights under Apache 2.0 or MIT, with working inference code. Nimble needs a CUDA GPU with BF16 support and caps at 2,048 prompt tokens and 26 choices per field. Kev-0.5B and the smaller SemIf builds run on a laptop. Jevlike and DiffusionGemmaJev are the least documented of the six and should be treated as demonstrations.

Why do the benchmark numbers disagree so much?

Different sample sizes, different prompt formats, different subsets, and in several cases an evaluation set built by the team doing the comparing. Sample sizes range from 208 paired items to 2,000. There is no standard benchmark for typed decisions yet, which is the underlying gap all six reproductions inherited.

Is Jev's calibration better than an LLM's?

On the independent measurement, no. Its expected calibration error of 0.246 was the worst in the nibzard field, and it declined to answer forced-uncertainty items only 49.7% of the time. Two of the open reproductions reach 0.03 to 0.08, but only after fitting a temperature parameter on held-out data.

How many options can these models handle?

Jev's latency is flat from 2 to 255 options and the API rejects 256 with 400 Too many choices. Nimble caps at 26 choices per field. Laya has no hard cap but drops to 0.425 accuracy at 77 labels. If your task has more than a few dozen categories, benchmark before committing.

Should I use a supervised classifier instead?

If you have labelled examples, test one first. A frozen bge-small encoder with logistic regression on 10,003 training examples scored 93.3% on Banking77, beating Jev at 83.2% and GPT-5.6 Terra at 87.5%. The decision models earn their place on zero-shot tasks, where no training data exists, and on latency.

What does Jev cost compared to running a clone?

Jev is $0.042 per million input tokens with output unmetered, which worked out to $0.07 per 1,000 decisions in the independent test, against $0.19 for the cheapest LLM measured. The open models cost nothing per call and cost you a GPU instead. Kev-0.5B's entire training run consumed about 0.06 kWh on a laptop.