As AI assistants evolve from simple text interfaces to multimodal companions, we are seeing a shift toward more proactive, situated assistance. Recent innovations like Project Astra and Gemini 3.1 Flash Live already allow users to discuss their physical surroundings in real time, often utilizing visual bounding box overlays to identify objects in a camera feed. While these overlays are highly effective for 2D screens, the transition to immersive platforms like Android XR presents a unique challenge: how do we move beyond flat UI to create a truly embodied, spatially aware dialogue?

To bridge this gap, we introduce Agent Hands, published at CHI 2026, a research prototype that brings the power of co-speech gestures to the 3D world. In human communication, our hands do more than just point; they describe shapes, mimic actions, and emphasize points, all synchronized with our voice. By leveraging the spatial understanding capabilities of Extended Reality (XR), Agent Hands replicates this natural synergy.

Following up our prior research in Human I/O and Sensible Agent, Agent Hands further equips AI agents with expressive, synchronized hand gestures that transform abstract verbal instructions into intuitive, physical demonstrations. The core innovation is its ability to map the high-level reasoning of LLMs into precise, real-time physical motions that match the agent's "voice" and the user's XR environment. The workflow includes environment awareness, a hand gesture event library, gesture-embedded reasoning, and synchronized XR execution.

By integrating these modules, Agent Hands creates a seamless bridge between linguistic intent and physical action. The system transforms a standard LLM output into a rich, multimodal performance where the agent's generated responses are manifested through both speech and spatially accurate movement.