OpenAI issued a new playbook that clarifies how independent reviewers should evaluate frontier AI models. The guide stresses that evaluations must move beyond simple chatbot prompts and incorporate the model’s tool use, state tracking, and workflow integration. By framing the assessment as a harness that mirrors real‑world usage, the post aims to tighten safety evidence for high‑stakes systems significantly today.

The document categorizes evaluation claims into three buckets: capability elicitation, safeguard performance, and comparison. It warns that harness choices can sway results, citing GPT‑5.5’s cyber‑range tests where a compaction feature boosted performance. The guide also notes that increasing token budgets can lift success rates, as UK AISI’s study saw up to a 59% jump when scaling from 10 M to 100 M tokens.

OpenAI urges report authors to disclose the claim tested and the evidence validating it, while flagging pitfalls like reward hacking, refusals, and contamination. The post concludes that a standardized harness is useful only when its limits are clear, otherwise customized setups better reveal true capability or robustness. This framework will shape forthcoming industry benchmarks for frontier AI safety today significantly.