A new benchmark called PhAIL exposes the performance gap between AI models and human workers in warehouse robotics. Developed by a former Google search ranking engineer, PhAIL tests four Vision-Language-Action models—OpenPI/pi0.5, GR00T, ACT, and SmolVLA—on bin-to-bin order picking using the same Franka FR3 robot and objects.

Human operators achieve 330 units per hour on the same robot, while the best AI model reaches only 64 UPH. Manual picking by hand hits 1,300+ UPH. The benchmark runs hundreds of blind trials where the operator doesn't know which model is running, ensuring unbiased results. Every run includes synced video and telemetry data.

All training scripts, fine-tuning datasets, and results are publicly available. The open leaderboard welcomes submissions from other researchers. This transparency addresses a critical need in robotics, where performance claims often lack standardized testing. The benchmark provides concrete data for comparing AI models against each other and against human performance in real-world warehouse operations.