Before our search pipeline retrieves anything, it decides how wide to search. “NVDA margins last two quarters” is about one company, so a narrow search over NVIDIA’s documents is enough. “Which companies are decreasing payroll?” names no company, so it needs broad market research: a slower sweep across every company we cover. We used to make this call with a prompted LLM.

Decision models answer a yes/no question like this one with a probability, in a few hundred milliseconds. A decision model can replace the prompted LLM entirely, or it can sit in front of the LLM in a model cascade. In a cascade, the decision model handles most requests itself and escalates to the LLM only when it is uncertain. Either way, the developer has to pick one or more thresholds, and picking them usually means tuning on annotated data.

Calibration can remove that step. A decision model is calibrated when its probabilities mean what they say: of the queries it scores 0.7, about 70% really are yes (Guo et al., 2017). A calibrated model’s default threshold of 0.5 is a sensible starting point, and the scores near 0.5 are where it is most likely to be wrong, which is exactly what a cascade should escalate.

We tried three decision models that serve the same API: Jev from TypeSafe; DiffusionGemma-Jev (djev), an open server built on a diffusion language model; and OpenAI’s GPT-6 Luna Decisions (Luna). We scored them on 238 queries labelled by two annotators.

As a replacement for the LLM, Jev works out of the box. At its default threshold of 0.5 it lands within about 2 F1 points of the LLM, and its price per token is less than half of Luna’s. djev matches it only after we tune its threshold on annotated data. Luna needs no tuning but tops out about 5 points lower.

In a cascade, Jev knows when to escalate without annotated data. Because Jev is calibrated, when it is unsure it is often wrong: sending the 7.6% of queries it scores between 0.2 and 0.8 to the LLM brings the cascade level with the LLM alone. djev’s band can only be found with annotated queries, and Luna is often sure and wrong: a third of its mistakes come with a probability of exactly 0 or 1, where no band reaches.

The task: does this query need broad market research?

Decision models take two inputs: the state, here the query being classified, and a yes/no question, here a few sentences of criteria for what counts as a broad search. The criteria end with “when in doubt, false”, because sending a narrow query to the broad search is slow and expensive. The model returns the probability that the answer is yes, and we flag the query for broad search when that probability is at or above a threshold.

Figure 1. A decision model answers a yes/no question with a probability. We send the user’s query as the state and ask one yes/no question. The probability then routes the query: a low score goes to the narrow search, a high score to the broad search, and an unsure score can be sent to a larger model. The three example queries are illustrative, not from our evaluation set; the probabilities are Jev’s answers to them on 2026-10-08. The 0.2 and 0.8 zone edges are the escalation band we test later in this post.

We compare three models, Jev, djev and Luna, and give each the same inputs.

  • Jev (jev-1.13.0) is TypeSafe’s first “System One Model”. TypeSafe trains it with a method it calls Reinforcement Learning for Calibrated Decisions, which targets “answers with epistemically honest probabilities” (TypeSafe).
  • djev serves the same API from DiffusionGemma, a general-purpose diffusion language model, by reading its token probabilities over fixed answer slots (mmastrac/djev; we ran taeold/djev-run). The model was not trained for calibration, and the project makes no calibration claim.
  • Luna (openai/gpt-6-luna-decisions-20261006) is OpenAI’s GPT-6 Luna behind OpenAI’s Decisions API, called through OpenRouter. The listing does not say how it is trained or whether it is calibrated.

Methodology

Data. We used 238 search queries, each labelled by two annotators working independently. About 15% are positive, roughly 36 per annotator. We score each model against each annotator and report the mean.

Accuracy at a threshold. To turn probabilities into decisions, we pick a threshold and flag every query at or above it. We report F1 and Krippendorff’s α, a measure of agreement that corrects for chance (0 is chance, 100 is perfect agreement). We compare against two reference points: our previous LLM classifier (F1 79.1, α 75.6) and the two annotators scored against each other (F1 81.7, α 78.5), which is roughly the ceiling for this task.

Tuning on held-out data. Anything we fit to the annotations, such as a threshold or the share of queries to escalate, we fit on a random half of the queries (with the same share of positives in each half) and score on the other half. We repeat this over 2,000 random splits and report the average. Scores that involve no tuning, such as the default threshold of 0.5, use all 238 queries.

Ranking quality. AUROC is the chance that a randomly chosen positive query gets a higher score than a randomly chosen negative one. It does not depend on any one threshold: 1.0 means some threshold separates the two classes perfectly, and 0.5 is no better than chance.

Calibration. A model is calibrated when its probabilities match how often things happen: of the queries it scores near 0.3, about 30% are positive. We measure this with expected calibration error (ECE). We group queries by score and average the gap between the mean score and the share of annotator votes that were yes in each group, so 0 is perfect.

Ranking and calibration measure different things. Ranking depends only on the order of the scores. If we squared every probability, the order and the AUROC would not change, but every score would shrink and the model would no longer be calibrated. In the other direction, a model that gives every query 0.15, the base rate, is perfectly calibrated and useless for ranking. A good classifier needs both: good ranking, so that some threshold works, and calibration, so that we know which threshold that is without annotated data.

Jev works out of the box

Figure 2. Jev is close to the LLM at its default threshold; djev gets there only after tuning. Each model’s accuracy at the default threshold of 0.5 on all 238 queries (hollow), and with a threshold tuned on half the queries and scored on the other half, averaged over 2,000 random splits (filled). Tuning helps djev and does not help Jev or Luna. Dashed lines mark our LLM classifier and the two annotators scored against each other.

At the default threshold of 0.5, Jev scores F1 76.8, about 2 points below the LLM. Tuning does not improve on that. When we pick its threshold on half the queries, the choice moves between 0.2 and 0.6 from split to split, and scores 75.6 on the other half, a little below the default.

djev ranks queries as well as Jev, but at 0.5 it flags only 14 queries against roughly 36 positives and scores F1 54.5. Its scores are squeezed toward zero: the median query gets 0.000, and the queries both annotators called positive have a median of 0.49. Tuning its threshold on half the queries usually picks 0.01–0.03 and scores 76.7 on the other half, level with Jev’s default. That tuning needs annotated data, has to be redone whenever the model or the question changes, and does not carry over to other models: on Jev, a threshold of 0.03 flags 212 of 238 queries.

Luna puts almost every query near 0 or near 1, so the threshold barely matters: anything from 0.03 to 0.5 gives F1 71–75 on the full set. Tuned on half the queries, it scores 70.4 on the other half, no better than its default of 71.3 and about 5 points below Jev. Jev is ahead in 90% of our random splits at the default threshold, but on 238 queries the 5.5-point gap is not statistically significant (bootstrap 95% interval −2.6 to 14.2).

Jev is also the cheaper of the two hosted models, and faster at the median:

F1 at 0.5F1 with tuned threshold (held out)Price per million input tokensMedian latencyp99 latency
Jev76.875.6$0.042194 ms15.1 s
djev54.576.7self-hosted——
Luna71.370.4$0.10269 ms759 ms
LLM classifier79.1———

Luna has the steadier tail latency: five of our 238 Jev calls took over 2 seconds, and no Luna call took over 0.9 seconds. We called Jev through TypeSafe’s API on 2026-09-28 and priced it at its OpenRouter list price; we called Luna through OpenRouter on 2026-10-08. Latency is wall-clock time from our client.

Calibration explains the difference

The three models rank queries about equally well. Where they differ is calibration:

Ranking (AUROC, higher is better)Calibration error (ECE, lower is better)
Jev0.950.033
djev0.950.089
Luna0.910.062
Figure 3. Jev’s probabilities track how often the answer is yes. (A) Each dot groups the queries a model scored in one probability range, plotted at their average score against the share of annotator votes that were yes. A calibrated model sits on the dashed line; a dot above it means the model gave those queries too low a probability. djev’s dots sit well above the line. Bins with only a few queries, which covers most of djev’s and Luna’s above 0.05, are noisy. (B) How each model spreads its 238 probabilities across the same ranges. djev and Luna put almost nine in ten queries below 0.05, and some of those queries are positive.

Jev’s probabilities track the observed rates, and its average prediction (0.142) matches the share of positives (0.149).

djev is underconfident. Queries it scores 0.1–0.3 are positive 80% of the time, and it puts about a dozen positives below 0.05. Its ordering is right, which is why its AUROC is high, but its numbers understate how likely each query is to be positive. That is why its threshold ended up at 0.03.

Luna is overconfident. 92% of its answers are below 0.05 or at least 0.9. Its high scores are accurate: the 13 queries it scores 0.9 or above are positive 96% of the time. Its low scores are not: it puts 6 of the 29 queries both annotators called positive below 0.05, where Jev puts none. Lowering the threshold far enough to catch them also flags many negatives, so no threshold lifts Luna much above its default.

With calibrated probabilities, the threshold follows from the cost of each kind of mistake. If a false positive costs k times as much as a false negative, we flag queries scoring above k / (k + 1). “When in doubt, false” puts k a little above 1, so the threshold sits a little above 0.5. For Jev, that does as well as anything we tuned on annotated queries. For djev, we had to annotate queries and search for the threshold that scored best instead.

Calibration tells us which queries to escalate

A common pattern is to trust the small model when it is confident and send the query to a larger model when it is not. Here the larger model is our LLM classifier. This only helps if the model’s mistakes come with uncertain probabilities.

Figure 4. Jev’s mistakes sit where it is unsure; many of Luna’s sit at P ≤ 0.01. Every query each model gets wrong at threshold 0.5, placed at the probability the model gave it. Filled dots are clear mistakes, where both annotators disagree with the model. Hollow dots are queries the annotators split on, which count as half a mistake. The shaded band is the 0.2–0.8 range we send to the LLM. Our run, 2026-10-08, 238 queries.

Jev never gives one of its mistakes a score below 0.03 or above 0.8. Sending its 0.2–0.8 band to the LLM (7.6% of queries) catches 5 of its 8 clear mistakes and brings the combined system to F1 79.7, level with the LLM alone. Most of what remains are queries the annotators themselves split on.

Luna’s mistakes pile up at the extremes. The same band sends 4.2% of queries and catches 3 of its 10 clear mistakes. To reach the rest we would have to escalate queries the model calls near-certain, and 193 queries share P = 0.00, so the only way to reach the mistakes among them is to escalate all of them. djev’s mistakes also sit low, because its whole scale is squeezed toward zero. Its unsure zone is near 0.01–0.03, where its tuned threshold landed, not around 0.5, and we could only find that by checking its scores against annotated queries.

Calibration also lets us predict how many mistakes a model will make. Adding up min(P, 1 − P) over all queries gives the expected number of errors at threshold 0.5: Jev expects 17.5 and makes 14.5, within noise; Luna expects 5.3 and makes 16.5. The same sum over the queries outside the 0.2–0.8 band predicts what is left after escalating: Jev expects 11.4 mistakes there and makes 8; Luna expects 1.8 and makes 13. With Jev, we can size the band before annotating any queries and get a rough, if slightly pessimistic, count of the mistakes that will remain. If we instead escalate the most uncertain queries by rank and choose how many using annotated queries (tuned on one half, scored on the other), djev and Luna reach F1 78.9 and Jev 77.5. Jev’s fixed 0.2–0.8 band, which we set without any annotated queries, scores 79.7, as well as any of them.

Public benchmarks show the same split

Public results put Jev and djev at similar accuracy, with Jev ahead where calibration is scored. Luna launched on 2026-10-06, and both public comparisons so far have Jev ahead of it, including on calibration error. Hover over a benchmark or metric for details. The benchmarks use different items and some are self-reported, so we treat them as indicative.

BenchmarkMetricdjevJev 1.13.0Luna
djev author’s testAccuracy198/201191/201—
Public JevBench questionsAccuracy84.8%86.6%—
JevBench leaderboard v1.5.0Intelligence7272—
JevBench leaderboard v1.5.0Calibration8088—
chimera-agentAUROC—0.9030.862
chimera-agentECE—0.1200.157
eval-connectionsPuzzles won—14.89.7

Caveats

  • This is one task with 238 queries and about 36 positives per annotator. Differences of a point or two in F1 are within noise. We tune on half the queries and score on the other half, and a single half has only about 18 positives, so tuned scores vary by several points from split to split.
  • Jev returns two-decimal probabilities with a floor near 0.02, which limits how finely we can set low thresholds. djev’s probabilities are finer-grained, and once tuned it scores as well as Jev’s default.
  • djev is not a distillation or copy of Jev. It serves the same API from a different model that was not trained for calibration, so the comparison tells us about calibration, not about the diffusion architecture.
  • Our claims about skipping annotation assume the model’s calibration carries over to your task. We could only confirm that for Jev because we had annotated queries; a small annotated set is still worth having to check it.

Summary

  • Jev works out of the box. At its default threshold it is within about 2 F1 points of our LLM classifier, with no annotated data needed to set it up, and its price per token is less than half of Luna’s.
  • djev ranks as well as Jev but is underconfident. It matched Jev only after we tuned its threshold on annotated data.
  • Luna is overconfident. Tuning does not help it, and it tops out about 5 F1 points below Jev.
  • All three rank queries about equally well. Calibration is what separates them.
  • Jev’s uncertain band is where its mistakes are, so sending 7.6% of queries to the LLM matches the LLM alone. A third of Luna’s mistakes come at P = 0 or 1, where no escalation band reaches.

Sources