The default 50/50 traffic split is not cost-optimal when treatment and control arms have different costs per user. For example, if treatment costs twice as much as control, allocating half the traffic to treatment does not minimize cost for a given precision. Cost-optimal allocation balances the precision gained per subject against the subject's cost, which varies by arm.

This approach can save 5% to 15% of experimental budgets—thousands to millions in large-scale tests—by adjusting sample sizes based on cost and variance. The method is especially relevant in industries using LLMs, discounts, vouchers, or cash-backs, where treatment imposes real financial costs. While tech often assumes experimentation is free, integrating paid features like agentic AI changes this, making cost-aware designs necessary.

However, business incentives favoring speed and convention often override statistical optimality, creating a tension between velocity and efficiency. The article explores when cost-optimal splitting becomes valuable and why it remains underused despite clear mathematical and financial benefits.