METR has acknowledged critical flaws in its developer productivity experiment measuring AI tool impact. The research team found their data from August 2025 onward provides unreliable signals due to significant selection bias among participants. Developers increasingly refuse to work without AI assistance, systematically excluding those who expect the highest productivity gains from the study.

Their initial 2025 study showed AI tools caused a 19% slowdown in task completion among experienced open-source developers. However, follow-up experiments revealed selection effects from both developer preferences and a reduced pay rate from $150/hour to $50/hour. Some participants reported difficulty tracking time when using multiple AI agents concurrently, while others noted quality differences between AI-allowed and AI-disallowed conditions.

Due to these severe methodological issues, METR is redesigning their study approach. The team plans to explore more intensive experiments with higher compliance rates and observational data from software development platforms. They believe current AI tools likely provide greater productivity benefits than their flawed experiments suggest, particularly for developers who have fully integrated agentic tools like Claude Code into their workflows.