Ask a chatbot for a random number between 1 and 100 and the answer is rarely uniform. Language models repeatedly favor a few values, and that habit has real implications when a model's pick is meant to protect something.
This analysis takes a security-minded view of the question: if a model chooses the number that guards a secret, how many guesses does an attacker need? Instead of counting answers across many sampled runs, the author reads the probability assigned to every value from 1 to 100 directly from the model's logits. From those 100 probabilities, the analysis computes bits of randomness and the expected number of guesses required to crack the pick. Base models and their instruction-tuned versions are compared side by side to see whether the habit comes from pretraining or from post-training.
The headline finding is stark. With a plain-text prompt, a 1.5-billion-parameter instruction-tuned model tested picks 47 about three times in four. With a reworded prompt, it still selects 47 more than a third of the time.
Earlier studies documented the same pattern. One sampling study across six models and seven languages reported favorites including 37, 47 and 73, and linked the bias to human tendencies. An informal 2022 comparison of OpenAI models found a more uniform base model and a tuned model with a sharp favorite. Research on fine-tuning stages has also found that reinforcement learning from human feedback significantly reduces output diversity.
Because open checkpoints now allow anyone to repeat the readout, the author can trace the pick through each post-training stage. For anyone relying on a model-generated 'random' value, the takeaway is clear: a predictable favorite can be cracked quickly.
Source: Towards Data Science · Summarized by HeadlinesBriefing