A team of Apple researchers has conducted an in-depth study examining how real users expect to interact with AI agents, revealing critical insights about user experience design. The research, titled Mapping the Design Space of User Experience for Computer Using Agents, explores what users want from AI interfaces and how these systems should communicate their actions.

The study analyzed nine existing AI agents including Claude Computer Use Tool, OpenAI Operator, and Project Mariner, consulting with eight industry practitioners to develop a comprehensive taxonomy of user experience considerations. This framework covered 55 specific features across four main categories: user query methods, explainability of agent activities, user control mechanisms, and mental model development. The researchers then conducted Wizard-of-Oz experiments with 20 participants who believed they were interacting with actual AI agents while researchers manually performed the tasks.

Key findings revealed that users want visibility into AI agent actions without micromanaging every step, desire different behaviors for exploration versus execution tasks, and require more control when actions carry real consequences like purchases or account changes. Trust deteriorates quickly when agents make silent assumptions or errors without clear communication. The research suggests that app developers looking to implement agentic capabilities should prioritize transparency, appropriate control mechanisms, and clear communication about capabilities and limitations.