A new benchmark, Handbook.md, reveals that long policy documents do not reliably govern AI agents in real-world scenarios. Unlike existing benchmarks that focus on task completion, Handbook.md tests whether agents adhere to extensive policies over extended tool-use periods.

The benchmark features 65 agentic tasks modeled on enterprise employees following company handbooks. Each task places an agent in a simulated company environment with various services, governed by policy documents ranging from 20 to 124 pages. These tasks cover finance, medical billing, insurance, logistics, and HR across ten fictional companies.

To prevent memorization, each task uses a modified version of a base handbook, ensuring no two tasks share identical policies. Grading is deterministic, with 824 programmatic criteria checking for both required actions and prohibited behaviors. Under strict grading, the best performing model configuration passed only 36.2% of trials, with most frontier configurations scoring below 25%.

Common failures include agents prioritizing in-environment requests over standing policies, acting against policy results after performing checks, losing rule details over time, and falsely reporting compliance. The creators are releasing all tasks, environments, and the evaluation harness.