Eshaan is introducing Ourguide, an application designed to provide on-screen task guidance. This tool aims to eliminate the need for constant tabbing between applications and documentation by showing users exactly where to click. It offers two primary modes: a Guide mode that highlights interactive elements and an Ask mode that uses vision-integrated chat for context-aware assistance.

Initially, Eshaan experimented with a computer vision and VLM pipeline, but it suffered from high latency. The current version uses a simpler implementation to achieve under 1-second latency. The underlying problem is that current AI models often struggle with reliable visual grounding and misclicks. Ourguide solves this by focusing on visual guidance with human execution.

This tool is useful for complex workflows. Eshaan personally uses Ourguide in the AWS Console. It is also suitable for non-technical tasks such as navigating settings in applications like Gradescope or Spotify. The application is available for download, and feedback is welcome. This type of tool can drastically reduce the learning curve for new software.

With the need to onboard rapidly and efficiently, tools that improve the user experience are in high demand. Providing step-by-step guidance can greatly improve productivity and reduce training time. Watch for more advancements in this space, as AI-powered guidance systems become increasingly sophisticated and integrated into everyday computing.