CLM-8B, an open decision model from researchers at Stanford University and NVIDIA Research, was released on 23 September 2026 under Apache 2.0. It picks between options instead of writing text, and its authors say it matches TypeSafe's Jev on computer-use, gaming and tool-calling tasks at up to 9x lower latency. Fine-tuned as a verifier, it sets new highs of 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1.
The GitHub repository passed 1,154 stars in two days. The numbers deserve a closer look than the chart gives them. The 81.6% is 31 of 38 coding tasks, and on 25 of those 38 the verifier's choice could not change the result. Here is what the model does, what the headline scores count, and how to run it as a verifier for your own coding agent.
What Stanford and NVIDIA Released on 23 September
CLM stands for Contrastive Language Model. The authors call it a "System One" model: you give it a state and a closed set of candidate actions, and it returns a probability for each one. It never generates a sentence. The team is Jacky Kwok (project lead), Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré and Azalia Mirhoseini.
The architecture is simple. A frozen Qwen3-8B turns the state and each candidate into embeddings, and two small projection heads, 20M trainable parameters in total, map them into a shared space. The score is the similarity between the state and each action, and a softmax over those scores gives you the answer. Because states and actions are encoded separately, a fixed menu of actions is embedded once and cached. The blog says that at roughly 1,000 candidates, CLM is 13x faster than Jev.
What you download is small. The reference head on Hugging Face is a 75 MB file, and the Python package went up on PyPI as contrastive-lm 0.1.0 on 24 September. Training ran in three stages: about 60M question and answer pairs from Nemotron, about 30M hard negatives written by Gemini 2.5 Flash-Lite, then about 1M agent trajectories. The blog says a full pre-training run takes about an hour on one RTX 4090, because the frozen encoder's embeddings are computed once and reused.
How CLM Compares With Jev
Jev is the closed decision model TypeSafe AI launched on 15 September, which we covered in our analysis of its 0% hallucination and 67.8% accuracy claims. CLM's server speaks the same request format, so code written against Jev can point at CLM instead. The authors ran four zero-shot tests against Jev with no task-specific training:
| Task | CLM-8B latency | Jev latency | CLM-8B result | Jev result |
|---|---|---|---|---|
| T-Rex game | 16.5 ms | 149.8 ms | 5/5 | 5/5 |
| Tool calling (BFCL v4) | 76.8 ms | 125.5 ms | 95.2% | 99.2% |
| WikiRacing | 79.8 ms | 225 ms | 26/30 | 30/30 |
| Super Mario | 33.5 ms | 132.6 ms | 5/5 | 5/5 |
CLM is faster on every row. It is not more accurate on any of them: it ties on the two games and trails Jev by four points on tool calling and four races on WikiRacing. The "up to 9x" figure comes from one row, the T-Rex game, and the speed-up on the other three is between 1.6x and 4x.
Read the T-Rex example's own README before you quote that row. A physics planner labels each action in the prompt ("jump: Safe. Clears the 2 large cacti. Best."), and a shield replaces any unsafe answer with the model's most probable safe one. The README says so plainly: the survival number "measures the combined system". It is a latency test, and a fair one, but it is not evidence the model can play the game.
What 81.6% on DeepSWE Actually Counts
The coding result is the one that matters for builders. For each DeepSWE task, the team sampled up to four candidate solutions from Claude Opus 5, and a verifier picked one. The fine-tuned DeepSWE head ships with a verification file, and it gives the whole picture:
| DeepSWE, 38 held-out tasks, best of 4 | Tasks solved | Rate |
|---|---|---|
| Pick a candidate at random (pass@1) | 28 | 73.7% |
| Jev as the verifier | 27 | 71.1% |
| CLM-8B fine-tuned as the verifier | 31 | 81.6% |
| Perfect verifier (any candidate passed) | 34 | 89.5% |
The file also reports 13 "mixed" test tasks, where some candidates passed and some failed. Those are the only tasks where picking matters. On 21 tasks every candidate passed, and on 4 none did, so any verifier scores the same on those 25. By our arithmetic, CLM chose correctly on 10 of the 13 mixed tasks, against about 7 expected from a random pick. That is three extra tasks solved, or half of the six that a perfect verifier could have recovered. It was also quick about it: 79 ms per verification on an H100, against 449 ms for Jev, or 5.7x faster.
That is a real gain, and the split is honest: 75 training tasks and 38 held-out tasks, with no overlap, published with checksums. It is also a small sample. One task more or less moves the score by 2.6 points. Jev's result, one task below random, is the more surprising finding, and the blog puts it directly: Jev "fails to serve as a verifier for long-horizon tasks".
The Terminal-Bench Number You Cannot Check Yet
The second headline, 87.6% on Terminal-Bench 2.1, is harder to pin down. It covers 30 held-out tasks, five candidates each from Claude Fable 5, against a pass@1 of 84.0% and 83.1% for Jev. But 87.6% of 30 is 26.28 tasks. A single run cannot produce that figure (26 tasks is 86.7%, 27 is 90%), so it must be averaged over runs or folds, and the repository does not say which.
As of 25 September, the Contrastive-LM Hugging Face account holds one fine-tuned head, the DeepSWE one, and no Terminal-Bench head or verification file. The gap between 87.6% and 84.0% is 3.6 points, a little over one task out of 30. Treat it as promising and unconfirmed. Neither score is a full leaderboard submission. Both are held-out subsets chosen by the authors. That is the same caution we applied to Step Code's 80.9% Terminal-Bench claim earlier this week.
How to Run CLM-8B as a Verifier for Your Coding Agent
You need Linux and an NVIDIA GPU with 24 GB of memory. The repository was tested on one RTX 4090, and its requirements file notes that it pulls in torch and vLLM. These steps use the commands from the README:
- Install the package. Run
pip install contrastive-lmin a fresh virtual environment. - Start the encoder. Run
vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 2048 --port 8090 &. The 2,048-token limit is what lets vLLM fit on a 24 GB card. - Start the CLM server. Run
clm-serve. On first run it downloads the 75 MB head into~/.cache/clm/and serves the API and a playground athttp://localhost:8700/. - Try it in the playground. Write a state, add a question, and look at the full probability distribution. The Rank tab ranks any list of candidates, and every request is shown as JSON, curl and Python.
- Rank your agent's candidates. Send the task as
contextand your agent's candidate patches or next steps asanswerstoPOST /v1/rank, or callEngine.rankin Python. You get them back best first, with probabilities. - Swap it in for Jev, if you use Jev.
client.system_one(state, questions)replays a TypeSafe request unchanged. SetCLM_API_KEYto require a key; cross-origin browser requests are off unless you pass--cors. - Fine-tune a head on your own rollouts. The zero-shot head is not what scored 81.6%. That came from a head fine-tuned on agent rollouts from 75 DeepSWE training tasks, kept separate from the 38 it was tested on. To verify your own agent's work, collect passing and failing runs from your repository and train with
train/finetune.py --task clm, starting from the reference head.
Three things will trip you up. States longer than 2,048 tokens are truncated, and coding trajectories are long: the DeepSWE head was trained on 8,192-token states. To match that, raise --max-model-len 8192 on vLLM and clm-serve --max-tokens 8192 together, which needs more GPU memory. A 502 error means clm-serve cannot reach the encoder on port 8090. And the README's command to reproduce the DeepSWE result points at Contrastive-LM/deepswe-clm-embeddings-8k, which returned a 401 when we checked. The training embeddings card links the evaluation rollouts under a co-author's account, tarsur385/deepswe-prm-embeddings-8k, which is public. We have not run that evaluation.
Limits to Know Before You Wire It In
CLM only chooses. It cannot write the patch, the command or the reply, so it always sits beside a generator such as Claude Code or Codex, never in place of one. A verifier also cannot beat its candidates. On DeepSWE, four of the 38 tasks had no passing candidate, and no verifier can recover those.
The zero-shot head is general, and the coding results needed task-specific training. The repository's fine-tuning guide is written as instructions for a coding agent, and it ends with "LOOP FOREVER". If you hand it to an agent, give that agent a time and GPU budget. There is no CPU route for the encoder. The repository's playground mock runs without a GPU, but with a fake encoder, so it shows you the interface and not real answers.
CLM is text-only for now. The team says a multimodal CLM-35B is in training for release early next month. For robotics, it points to an earlier variant described in the CoVer-VLA paper. MarkTechPost's launch write-up tabulates every chart number, and it is a useful companion to the repository. Keep the task counts next to the percentages when you read it: the "state of the art" label on DeepSWE rests on three extra tasks.
Who Should Try It Now
Try it if you already run best-of-N with a coding agent and pick the winner by hand, by test pass or by asking a large model. A local 8B verifier that answered in 79 ms on an H100 in the DeepSWE test is cheap enough to score every candidate, and you own the weights. It also fits hot loops where a fixed set of actions repeats: game bots, UI automation and tool routing, where cached actions do the most work.
Skip it for now if you have no Linux GPU, if your states run past 2,048 tokens and you cannot spare the memory for 8,192, or if tool-calling accuracy matters more than milliseconds. Jev still leads there. Everyone else should wait for the Terminal-Bench head and the CLM-35B release before reading the 87.6% as settled.
Frequently Asked Questions
What is CLM-8B?
CLM-8B is an open decision model from researchers at Stanford University and NVIDIA Research, released on 23 September 2026. It scores a set of candidate actions against a state and returns probabilities. It does not generate text. It pairs a frozen Qwen3-8B encoder with a 75 MB projection head.
Is CLM-8B free to use commercially?
The code and the CLM-8B weights are released under Apache 2.0. The Qwen3-8B encoder it runs on is also Apache 2.0. The fine-tuned DeepSWE head and its training embeddings are released under MIT.
How is CLM-8B different from Jev?
Jev is TypeSafe AI's closed, hosted decision model. CLM-8B uses the same request format but runs on your own GPU. In the authors' tests, it was faster on every task and tied or trailed Jev on accuracy, with the biggest gaps on tool calling (95.2% against 99.2%) and WikiRacing (26/30 against 30/30).
What hardware does CLM-8B need?
Linux and an NVIDIA GPU. The quickstart is tuned to fit a 24 GB card such as the RTX 4090 by capping states at 2,048 tokens. Longer states need more GPU memory. The published verifier latencies were measured on an H100.
Does 81.6% on DeepSWE mean CLM solves 81.6% of coding tasks?
No. CLM does not solve anything. It picked the best of four Claude Opus 5 attempts on 38 held-out tasks and got 31 right, against 28 for a random pick and 34 for a perfect picker. The score measures the choice, not the coding.
Can I use CLM-8B with Claude Code or Codex?
Yes, as a separate scoring step. Have the agent produce several candidate patches, send them to CLM's rank endpoint with the task as context, and apply the top one. For results like the DeepSWE ones, fine-tune a head on passing and failing runs from your own repository first.
