Google researchers have developed a framework to evaluate how closely large language models align with human behavioral dispositions in practical scenarios. The study tested 25 models using established psychological questionnaires like IRI and ERQ, adapted into Situational Judgment Tests that present realistic scenarios with two possible courses of action.

Smaller models under 25 billion parameters showed significantly lower alignment with human preferences, often performing at near-chance levels when consensus existed among human annotators. Larger models over 120 billion parameters achieved near-perfect alignment in unanimous scenarios but plateaued in the low-to-mid 80s when consensus dropped below 90%. The research revealed that models frequently prioritize emotional openness over professional composure and harmony over standing one's ground in conflicts.

The study found that models systematically overconfident even when human opinions were divided, failing to represent the full spectrum of human viewpoints. Different training approaches produced distinct behavioral patterns, suggesting that alignment procedures significantly shape how models navigate social dynamics. These findings highlight the need for better behavioral alignment to ensure models appropriately handle the nuances of human interaction.