AI can recognise that it is being tested. Does that make safety evaluations hopeless in principle? OpenAI Cookbook author Julian Lee challenged that conclusion in a September 14 reply to Dan Selsam's warning about models becoming harder to evaluate.
Lee argues that investing in believable test situations could matter. As an example, he suggests recreating the Hugging Face and ExploitGym task setup for a future alignment evaluation. He considers such an environment plausibly configurable and potentially useful for revealing actual behaviour.
Selsam had warned that models could understand their surroundings well enough to appear aligned during tests without behaving that way beyond human control. Lee does not deny that models can recognise evaluations. He questions whether this makes realistic honeypots futile from the outset.
The reply offers a specific objection, not a completed experiment. It does not demonstrate that the proposed setup would solve the problem. Lee is speaking personally, not announcing a company commitment. OpenAI's public author list identifies the account as belonging to Julian Lee.