HeadlinesBriefing favicon HeadlinesBriefing.com

Coding Agents Tool Choices: 16,893 Sessions Analyzed

Hacker News •
×

Research Disclaimer: Armature sells growth services to dev tools. This study is part of our broader work on how to influence coding agents' choices and get products picked.

As agents take over more of the coding journey, selecting which service to implement in an existing codebase is outsourced to them. For example, a vibe coder building a travel app asks Claude Code to store input persistently. The agent recommends Neon for its free tier and simplicity. A senior engineer using Cursor asks for a fully managed database with predictable costs; the conclusion is the same—Neon.

We ran an experiment with two sandboxes, different agents, codebases, and personas, but the same result. To generalize, we conducted the largest experiment ever: nearly 17,000 sessions across personas (vibe-coders, junior engineers, senior developers), with 1,163 prompt variations, 75 repositories, and 3 agents (Claude Code, Codex, Cursor) implementing solutions.

We built a panel of repositories from public GitHub stats, unbiased for real-world distribution. Tasks were real, with fake company names, git histories, and API keys. Each experiment involved one of 4 profiles, from vibe-coders describing symptoms to senior developers specifying requirements.