Building a verifier with Jev: designing Generator–Verifier performance improvements

Why put Jev in the verifier slot

This is Part 2 of the Generator–Verifier series. Part 1 explained that if a generator produces several candidates and a verifier picks one, performance can improve without changing the generator. It also showed that the more inference-time compute increases, the more the verifier becomes a bottleneck. This part designs how to improve performance when TypeSafe AI's decision model Jev, instead of an LLM, is placed in that verification slot.

The official TypeSafe documentation source explains it this way. LLMs are designed to produce text for people to read, so when asked for a judgment that code will consume, there is a mismatch: the text has to be parsed back into structured values. Jev does not generate text; it directly returns typed values and probability distributions. There are three question types. Choice picks one item from a list (choice·probabilities·confidence), Score assigns a score using a rubric (a scoring guide) (score·probabilities·confidence), and Noul answers whether a statement is true with a value from 0 to 1. The LLM-as-Jev study paper likewise categorized the same three formats as selection, binary verification and ordinal scoring, and noted that Jev's weights and training method have not been disclosed. According to the JEVal study paper, Jev is available only through a hosted API.

This article separates two kinds of statements. Content beginning with 'Evidence:' is fact found in papers or official documentation. Content beginning with 'Design proposal:' is the author's proposal drawn from that evidence and is not a verified result. Speed and pricing figures that TypeSafe itself published are noted separately as the provider's own measurements.

Core idea: fast first-pass screening, slow second-pass review

Think of an airport security checkpoint. Every bag passes quickly through the X-ray scanner, and only bags with unclear scans are opened by a person. Jev plays the scanner. It is applied cheaply to every candidate and outputs a score and confidence. The close inspection of opening the bag is handled by a strong LLM verifier or a human.

Evidence: There is a study paper on prefill-only decision models inspired by Jev. Prefill-only means the model reads the input once and does not decode, i.e. it does not write out text one token at a time. According to this study, such models score all options in a single forward pass, so one call is 1–2 orders of magnitude cheaper than a generative language model of the same size. The official TypeSafe documentation source also explains that multiple questions about the same state are evaluated at once, in parallel and independently, so adding questions barely increases response time. At the other end, the generative verifier GenRM paper generates its judgment as text. It can therefore use CoT (Chain-of-Thought, step-by-step reasoning) and majority voting, and it raised GSM8K accuracy under Best-of-N from 73% to 93.4%. In exchange, as seen in Part 1, each verification is costly.

Design proposal: Instead of making the two verifiers compete, stack them in layers. Jev is the first-pass verifier that sees every candidate, and the generative verifier is called only for cases Jev finds ambiguous. Evidence: In the experiments of the prefill-only study paper, at equal cost, a confidence-based cascade outperformed any approach that re-queried the same model. This is a signal that the layered structure is not far-fetched, but keep in mind that the result comes from that study's tasks and models.

How it works: reranking, step checks and routing

① Best-of-N reranking. Design proposal: When an LLM generates N candidates, feed each one into Jev Score as a 'question + candidate' state and receive a rubric score and probability distribution. Then sort the candidates by expected score or by the probability of the top grade. Evidence: The official TypeSafe documentation source recommends that each question ask about only one narrow, clear thing. Questions that require weighing several factors should be split into one question per factor, with the results combined in code. Following this, instead of asking a single 'Is this answer good?', you ask separately 'Does it follow the required format?', 'Does it address all the conditions in the question?' and 'Do the reasoning and conclusion fit together?', and set the weighted sum in code. When priorities change, you only change the coefficients instead of rewriting the prompt.

② Step and condition checks. Design proposal: Turn each step of a solution or execution plan into a statement such as 'The calculation in step 3 correctly follows from the result of step 2', feed it into Noul, and receive the probability that it is true. This effectively imitates, through questions and without training, what the process reward model (PRM) from Part 1 did. Evidence: The LLM-as-Jev study paper explains that such models are used for decisions that 'verify whether evidence satisfies a rule.' According to the JEVal study paper, decision models were competitive when the basis for judgment was contained in the input. In contrast, they fell behind in domains requiring expert knowledge, such as medicine, finance and law. So write Noul statements so that the facts needed for the judgment are included in the input.

③ Confidence routing. Design proposal: If confidence is high, pass the case through as is; if it is medium, escalate to a strong LLM verifier; if it is still ambiguous, send it to a human. Cases judged by humans are collected back into the evaluation set. Evidence: The TypeSafe homepage source advises users to set the thresholds for when to act automatically and when to request review. In the System Switch study paper, the 30% of decisions with the lowest confidence were handed to a reasoning model. The improvement over handing off at random was proportional to the first-pass model's AUROC (rank correlation 0.87). AUROC is a metric showing how well confidence separates right answers from wrong ones. This means the effect of routing depends on this 'separating power' more than on accuracy.

Calculating cost and latency

Design proposal: Total cost is roughly 'N generations + verification calls + (escalation rate × strong verifier cost) + human review cost.' If N candidates are each scored separately with Score, the state differs per candidate, so there are also N verification calls. However, the official TypeSafe documentation source explains that even if you ask several questions per candidate, questions about the same state can be bundled into one call. If you put all candidates into one state and have Choice pick one, the calls drop to one, but you must also weigh the option-label risk discussed later. Usually most of the cost comes from the N generator runs, and the variable that actually drives the budget is the escalation rate.

Evidence (provider's own measurements): The TypeSafe homepage source states that, based on a workflow for System One tasks, Jev is 193.6x faster and 444.6x cheaper than an LLM. As an example, it gave Jev at 0.114 seconds and $0.000081, and the LLM at 8.566 seconds and $0.013880. It states the price as $42 per billion input tokens, 238x lower in input price than Claude Fable 5.1. All of these are figures produced under conditions the provider chose. Evidence from independent research includes the 1–2 order-of-magnitude difference reported by the prefill-only study paper mentioned earlier, and the JEVal study's paper report that InnerJev-27B answered typical queries in about 0.1 seconds.

For latency, look at the tail rather than the average. Evidence: The JEVal study paper had Jev choose the next action in τ-bench. Median episode time fell, but task success rate dropped and tail latency rose. Design proposal: With routing, only ambiguous cases slow down, so record median and upper-percentile latency separately and manage the escalation rate as a budget variable.

There is also little reason to set N high. Evidence: The study on the limits of resampling paper reported that when the verifier has false positives, the optimal number of attempts is often fewer than 10. The cost of false positives outweighed the gains, and the inference scaling curve bent downward.

Checking calibration and setting thresholds

Calibration means that judgments the model labels as 80% are actually right about 80% of the time. The evidence does not point in one direction. In the JEVal study paper, decision models often got the most plausible answer right while greatly inflating its probability. In TypedBench paper, one hosted model followed the policy but was sensitive to wording and systematically reported low confidence. In situations with asymmetric costs, using those probabilities was sometimes worse than simply using the top-ranked answer. On the other hand, the LLM-as-Jev study paper reported that reading next-token probabilities from an untrained Qwen3.5-4B showed strong calibration. Since the direction differs by model, task and wording, the provider's description of 'calibrated confidence' source is only a starting point and does not replace checking.

Evidence: The prefill-only study paper predicted accuracy when only the option list was changed, using the probabilities from the first call. Predictions based on rankings were within 4.2 points, but predictions at the level of probability values were off by 21.0 points. In the same study, uniformly adding more calls to the same model improved calibration but barely raised accuracy. Design proposal: Jev can be brought in relatively early where only rankings are used, as in Best-of-N reranking. In contrast, places that use absolute probability values, such as 'pass if 0.9 or above', should be switched on only after checking calibration on your own data.

Design proposal: Here is how to check. Build an evaluation set sampled from real traffic with correct-answer labels. Bin confidence into ranges, compare the actual accuracy in each range, and also measure AUROC. Evidence: The System Switch study paper showed that accuracy, calibration and the power of confidence to separate right from wrong answers are distinct properties. Even models with similar accuracy differed widely in AUROC. Design proposal: Set thresholds by weighing 'the cost of one false pass' against 'the cost of one escalation call.' Recheck them on data not used to set them, and if you change the question wording, measure again from scratch. This is because of TypedBench's paper finding of sensitivity to wording.

Risks: inflated probabilities, label traps, error accumulation, over-optimization

Probability inflation. Evidence: The JEVal study paper reported that decision models tend to overestimate probabilities. A study on the limits of resampling paper found that a verifier's false positives do not go down with repeated sampling, which puts a ceiling on accuracy. Design suggestion: High-confidence false passes go straight through routing, so sample some cases from the high-confidence band as well and have people audit them.

Option label trap. Evidence: The Option-Channel attack study paper tested seven open-weight decision models as agent guardrails. It did not test Jev itself. Inserting six lines of server logs unrelated to the policy raised the rate of allowing a prohibited action from 0% to 63% on policies the models had originally judged correctly. When only a misleading name was attached to the allow option, the rate reached 93–100% in the four models that take labels as input. Verdicts flipped by the attack were no less confident than the original verdicts, so escalating the lowest-confidence verdicts did not help either. In the System Switch study paper, simply changing the order of the options changed accuracy for some models. Design suggestion: Keep option names and definitions as neutral wording fixed in code, and treat generated candidate text as untrusted input. Also add attack tests to the evaluation set that shuffle the order and check whether the verdict changes.

Error accumulation. Evidence: The JEVal study paper found that on τ-bench, the advantage of fast local decisions was offset by decision errors accumulating over long trajectories. Design suggestion: Passing many step checks does not mean the whole result is correct. Put a stricter threshold or escalated verification in front of irreversible steps, and keep a final-result check separate from the step checks.

Verifier over-optimization. As with the reward hacking covered in Part 1, if you select with a verifier and measure performance with the same verifier, candidates that fit the verifier's blind spots win. Evidence: The study on the limits of resampling paper pointed out that building training data by filtering with a verifier has the same false-positive limit. Design suggestion: Keep the Jev questions used for selection separate from the criteria used for final evaluation (human labels, or a different verifier). Treat 'the Jev score rises but accuracy on a separately held evaluation set stays flat' as a sign of over-optimization.

Starting small: experiment order

Everything below is a design suggestion. First, build an evaluation set. Have people attach correct answers or verdict labels to real inputs, and include easy cases, ambiguous cases, and the label and order attack cases discussed above. Second, measure baselines. The baselines are the accuracy of a single generator answer, and the accuracy, cost and latency of using only a strong LLM verifier. Third, attach Jev in 'shadow mode'. Record its verdicts without acting on them, and measure accuracy by confidence band, AUROC, and how much verdicts shift when the wording changes.

Fourth, set two thresholds. One is the threshold for passing a case through as is; the other is the threshold for sending it to a person. Set them by weighing the cost of a false pass against the cost of an escalation call, and check them on data not used to set them. Fifth, turn things on in stages. Start with Best-of-N reranking with a small N that uses only the ranking; once calibration is confirmed, add confidence routing, and finally add Noul step checks. Sixth, during operation, keep monitoring the escalation call rate, the false-pass rate from sample audits, and latency on the escalation path.

It is also worth adding an open model as a comparison. Evidence: The LLM-as-Jev study paper proposed a training-free method that reads decisions from an open LLM's next-token probabilities. The JEVal study paper reported that InnerJev-27B performed similarly to Jev on JEVal. Design suggestion: To avoid depending on a single hosted API, run such open alternatives on the same evaluation set to get a point of comparison.

Further reading

To first see which decisions Jev handles well and poorly, read the JEVal study paper. Its probability inflation and τ-bench error accumulation findings connect directly to the risks section of this article. Next, the prefill-only decision model study paper shows that 'rankings are stable but probabilities are less so' and includes a cascade comparison, which explains why reranking and routing are treated separately here.

For concrete methods of measuring calibration and thresholds, see TypedBench paper and the System Switch study paper. From a security perspective, be sure to read the Option-Channel attack study paper. If you plan to build a similar verifier with an open model, the LLM-as-Jev study paper is the starting point.

For background on using a generative verifier as a second-pass review, see GenRM paper; for why N should be kept small, see the study on the limits of resampling paper. For how to use the product in practice, start with the official TypeSafe documentation source. The Confidence and Patterns sections that the documentation points to next cover using confidence scores and configuration patterns.