OpenAI researchers Melody Guan, Yuxiang Xie and Bowen Baker introduced two complementary ways to evaluate AI monitors on October 20. They address differences in the task mixtures used to compare models and the risk remaining after simulated monitor-based approval.
The first compares models on shared eligible tasks, reducing the chance that task difficulty is mistaken for a difference in monitorability. The authors note that matching tasks still does not guarantee matching behaviour.
The second simulates rejected outputs and additional attempts. Gating reduces risk on average in the examined evaluations, but retries erode some of that benefit. OpenAI explicitly says its current models are not deployed under this hypothetical scheme.
The authors recommend reporting remaining bad outputs alongside approval rates and retry limits. Low measured risk can also mean that almost nothing is approved. The metrics complement g-mean² rather than providing a general safety guarantee.