Machine learning models rely on human-labeled data, but real-world applications often involve subjective concepts like 'healthy food' or 'inspiring artwork.' Traditional training workflows force users to provide rigid labels even when their definitions are still evolving. To address this, Google AI has introduced Agile Deliberation, a framework that treats the creation of subjective AI classifiers as an iterative conversation. Instead of acting as static data labelers, users collaborate with the AI system to sharpen their own definitions. By resolving human ambiguity first, the framework builds more effective, robust, and aligned AI systems. The code and algorithms have been open-sourced on GitHub. The framework was evaluated across 18 live, 1.5-hour sessions with human participants, showing a drastic reduction in user fatigue and anxiety. It replaces manual prompt engineering with direct feedback on borderline images, resulting in more consistent, higher-performing AI models that outperform standard baselines.
Agile Deliberation splits the task into two stages: concept scoping and concept iteration. In the first stage, users break down broad concepts into structured hierarchies of sub-concepts and exclusions. For example, 'Healthy Food' might be decomposed into 'Salads,' 'Grain Bowls,' and 'Lean Proteins,' while excluding 'Processed Foods' or 'Raw, Unprepared Ingredients.' The second stage moves from static planning to active testing, drawing inspiration from active learning to collaboratively refine definitions based on real-world examples.
Source: Google AI Blog · Summarized by HeadlinesBriefing