HeadlinesBriefing favicon HeadlinesBriefing.com

Claude Code A/B Tests Disrupt Workflow Without Notice

Hacker News •
×

A paying Claude Code user discovered Anthropic is running silent A/B tests that degrade core functionality without user consent. The user, who pays $200/month for the professional tool, found evidence of a GrowthBook-managed experiment called tengu_pewter_ledger that controls how plan mode generates output. The most aggressive variant, cap, restricts plans to 40 lines with no context or prose.

After experiencing unexplained workflow disruptions, the user decompiled the Claude Code binary to investigate. They discovered four variants ranging from null to cap, with the default providing full context and detailed verification sections. The cap variant produces terse bullet points without back-and-forth dialogue, presenting users with a fait accompli. The binary logs variant assignment and plan metrics through telemetry.

This lack of transparency contradicts principles of responsible AI deployment. When AI tools abstract away planning logic without user visibility, they remove the human-in-the-loop control that makes them useful. For professional developers relying on Claude Code for their work, silent experimentation on paying customers represents a fundamental breach of trust and workflow autonomy.