Google AI introduces ConvApparel, a new dataset addressing the realism gap in user simulators for conversational AI systems. Current LLM-based simulators often behave unrealistically, exhibiting excessive patience or unrealistic knowledge that doesn't match genuine human interactions. This gap undermines AI development, as agents trained on unrealistic simulations perform poorly when facing real users with actual expectations and limitations.

ConvApparel contains over 4,000 human-AI conversations from a unique dual-agent protocol where participants interacted with either a helpful Good agent or an intentionally unhelpful Bad assistant. The dataset captures the full spectrum of human emotional responses through fine-grained annotations, providing researchers with unprecedented insight into authentic user behavior across different interaction quality levels.

Google's three-pillar evaluation framework—population-level statistics, human-likeness scoring, and counterfactual validation—reveals that even advanced simulators struggle to adapt to novel, frustrating agent behaviors. The supervised fine-tuned simulator performed best but still failed to fully replicate authentic human frustration patterns, revealing fundamental challenges in developing truly realistic user simulators for conversational AI development.