Ask a chatbot for a random number between 1 and 100 and you will rarely get an even spread. Language models have a clear habit of returning the same few values, and most write-ups only count how often each number appears. This analysis asks the security question instead: if a model picks a number that protects something, how many guesses would an attacker need?
The author reads the probability the model assigns to every answer from 1 to 100 directly from its logits, avoiding sampling noise. From those 100 probabilities the analysis computes bits of randomness and the expected number of guesses needed to crack the pick. It compares base models with their instruction-tuned versions to see whether the habit comes from pretraining or from post-training. In one test, a 1.5-billion-parameter instruction-tuned model answered 47 about three times in four with a plain-text prompt, and still picked 47 more than a third of the time after the prompt was reworded.
Prior research has documented the same pattern. A large sampling study across six models and seven languages found repeated favorites such as 37, 47 and 73, and linked the bias to human tendencies. A separate study found that GPT-4 and Llama 3 exaggerate nearly every human bias tested. An earlier informal comparison of base and tuned OpenAI models suggested that tuning can sharpen a favorite number, though that post was later corrected.
The author adds three things: open checkpoints that anyone can rerun, tests of whether a favorite survives a change of prompt, and a risk-focused measure of how guessable a secret is.
Source: Towards Data Science · Summarized by HeadlinesBriefing