Despite advancements in LLM-based agents for tasks like coding, their ability to interact with real-world software interfaces remains a significant bottleneck. Current benchmarks like OSWorld-V2 show completion rates of only around 20.6%, far below user expectations. Many agents bypass intended UI interactions by directly calling APIs or scripting solutions, a method that is often more complex and less reliable than simply using the interface.

This bypass strategy is a workaround for the core issue: current agents lack human-level ability to perceive and manipulate graphical user interfaces. They dedicate excessive computational resources to visual grounding and low-level manipulation rather than task reasoning. This "trillion-scale reasoner working around clicks" approach is deemed a dead end by Steelman Labs.

Steelman Labs proposes a new direction, emphasizing that agents must be able to use any interface a person can. They advocate for separating planning from execution and introducing a "System 1" manipulator for motor control, adopting a vision-first approach with human-level reaction times. This aims to enable agents to spend more effort on solving tasks rather than overcoming interface limitations.