Google researchers developed a two-stage method for understanding user intent from mobile and web interactions using small multimodal LLMs. The approach first summarizes individual screens before extracting a final intent, outperforming larger models in evaluations. This technique enables on-device intent understanding, reducing latency, cost, and privacy risks associated with server-side processing.

The decomposed workflow tackles a key limitation of large LLMs for real-time UI analysis. By separating screen summarization from intent extraction, the method makes the task tractable for smaller models like Gemini 1.5 Flash 8B. The team fine-tuned the second stage and used a Bi-Fact evaluation metric to track atomic facts, achieving results comparable to the much larger Gemini 1.5 Pro model.

This work demonstrates that architectural decomposition can unlock powerful capabilities from smaller, more efficient models. For developers, it opens the door to building responsive, privacy-preserving assistive features directly on mobile devices. The technique could influence how future on-device AI systems anticipate user actions without relying on constant cloud connectivity.