A software engineer has published a detailed critique of the current discourse around generative AI, arguing that most claims about its usefulness lack scientific rigor. The author, who has been a generative model skeptic since the beginning, observes that discussions typically devolve into vague assertions about productivity gains without addressing specific use cases or measurable outcomes.

The piece introduces a three-factor model for evaluating generative model utility: the cost of encoding tasks into prompts versus direct creation, the cost of verifying generated outputs, and the degree to which tasks depend on process versus artifact. This framework provides a concrete way to assess whether these tools actually deliver value for specific applications rather than relying on subjective feelings of productivity.

According to the analysis, generative models are most likely to be useful when creating artifacts is difficult for users but verification is trivial, such as tasks requiring cross-referencing specific information. The model predicts these tools become less useful as task complexity increases, since probabilistic outputs are less likely to meet complex requirements. The author emphasizes that domain expertise remains essential when using generative models, as users must verify outputs meet requirements. The framework offers a path forward for more rigorous evaluation of AI tools beyond marketing claims and anecdotal evidence.