HeadlinesBriefing HeadlinesBriefing.com

How accurately calibrated is Jev?

Hacker News •
×

Jev is Type Safe’s new “System One” classifier model. The name is inspired by Daniel Kahneman’s book Thinking, Fast and Slow, in which he distinguishes between fast, instinctive, System One thinking and slower, conscious, System Two thinking. Large Language Models (LLMs) like Claude or GPT are System Two models and “Decision Models” like Jev and its predecessors (e.g. Laya) are System One.

Jev takes a pretrained transformer and bolts a classifier on the end, allowing it to use new context immediately while acting as a classifier. You give it context and a multiple-choice question, and it returns a probability distribution over the multiple choices. The entire below series of experiments cost less than $4.00.

I tested Jev's calibration using well-understood physical distributions, like the Maxwell-Boltzmann distribution. I picked 10 candidate distributions, 5 prompt templates per distribution, and 20 variations per prompt, giving 1,000 settings total. GPT-6 Astra and Claude Opus 5.5 were used for implementing the experiments, writing templated prompts, and API calls.

Source: Hacker News · Summarized by HeadlinesBriefing