HeadlinesBriefing favicon HeadlinesBriefing.com

Reverse-Engineered Jev-Like Model: Starter for Option Scoring

Hacker News •
×

Train a small model that chooses among a changing list of text options. A Jev-like model takes a piece of text and a list of N text options. It returns one probability for each option in one pass instead of writing an answer word by word. Jev is Type Safe's commercial model for this task, but Type Safe has not published its design. This repository is an independent starter model with the same input and output shape.

The same option-attention head can score controller buttons from image patches. A ten-second film joins two selected five-second windows: live deadly_corridor combat on the seven Doom buttons, then a chess controller walking to and playing moves with five keys. The Doom window came from the supplied joint checkpoint, which averaged 0.60 kills and -97.50 reward across its ten recorded episodes. The chess window came from the stronger chess-only checkpoint, which scored 4 wins, 46 draws and 0 losses in 50 sampled games against a random mover, but 0 wins, 2 draws and 48 losses against Stockfish level 0. The windows were selected for activity and are not typical-play or competence claims.

Install the game extras and record a fresh 640 by 480 Doom trace from the released joint checkpoint: uv pip install -e '.[games]' python examples/doom/play.py examples/checkpoints/joint-imitation.pt --episodes 10 --game-seconds 35.3 --device cpu --capture-resolution 640x480 --output runs/doom.mp4 --trace runs/doom-trace.json Render the trace in the same visual layout. This writes a silent film because the author-owned soundtrack source is not part of the repository. (cd examples/film && npm install && npx playwright install chromium) examples/film/make-film.sh runs/doom-trace.json runs/doom-film.mp4 10

The release includes the Doom example, the chess example, the single-game checkpoints and the shared 12-option checkpoint. Both games import the visual scorer from jevlike.vision; there is no second model copy in either example. Architecture Each option becomes a query vector, a short list of numbers representing its text. The query assigns attention weights to the context tokens. Those weights make one context vector for that option. A shared dot product turns each option and context pair into one score. A softmax runs across the options. The default encoder learns byte embeddings from scratch. The optional Hugging Face path uses a frozen pretrained encoder. Data format Use one JSON object per line: {"context":"The customer needs a refund.","options":["refund","sales","technical support"],"label":0} label is the zero-based index of the correct option. Each row may have a different number of options, with a minimum of two. Quickstart Run these commands from the repository root: uv venv source .venv/bin/activate uv pip install -e '.[dev]' jevlike-data synthetic --output data/synthetic jevlike-train data/synthetic/train.jsonl --validation data/synthetic/validation.jsonl --output runs/synthetic.pt jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl jevlike-predict runs/synthetic.pt --context "Choose the exact badge amber badger. Badge: amber badger." --option "azure crane" --option "amber badger" --option "gold heron" The evaluation prints top-1 accuracy, top-3 accuracy, expected calibration error, and a shuffled-context control. Use your own data: Export train, validation and test JSONL files in the format above.