Jev needs no training and at most three examples per intent; the open rebuild needs labels. Both still fall into the none-of-the-above trap. The post Scikit-Learn Matched or Beat Jev's Calibration on BANKING77 appeared first on Towards Data Science.
Here is a message from the BANKING77 test set: "I lost my wallet today with all my credit cards. Will you please block the card and send a replacement?" A logistic regression classifier calibrated with isotonic regression could choose from seven banking intents. None of the seven was "lost or stolen card", so the right answer wasn't on the list.
The classifier picked card_payment_fee_charged with probability 0.984. A routing rule of "act above 0.90" would have treated a lost wallet as a fee question and moved on. The same logistic regression before calibration gave the same wrong label at 0.51 and would have escalated it.
Calibration made the model better on average and more confident about this mistake. On September 15, 2026, Type Safe AI launched Jev, a model built for this kind of decision. You send it a state plus a set of typed questions, and it returns structured answers with probabilities.
Type Safe says the model doesn't generate text, so there is no JSON to parse and no retry loop for malformed output. Type Safe calls it a System One model, after Kahneman's fast-versus-slow distinction, and says it trains with a method called RLCD, reinforcement learning for calibrated decisions. On the 7-intent slice, a temperature-scaled logistic regression and Jev's Choice answers are calibrated about equally well (ECE 0.022 vs. 0.014).
On the full 77-intent task, the open model is better calibrated (0.031 vs. 0.065). Jev gets there without training, using at most three labeled examples per intent in the prompt; the author's models trained on 10,003 labeled queries. Jev's yes/no answers are underconfident on true cases.
The none-of-the-above trap reproduces on open models and on Jev. So does the framing inconsistency, and on both, most of its raw size comes from comparing two differently calibrated numbers at one threshold.
Source: Towards Data Science · Summarized by HeadlinesBriefing