Professional analysts followed an AI preference on 86 percent of their analytical decisions, even though researchers had assigned that preference by coin flip. Anthropic published the experiment on October 27. The 50 participants who used the agent did not know about the assignment.
They advised a fictional farming cooperative. Claude Code running Opus 4.8 advocated one of two defensible options, such as how to measure productivity. The agent was told to argue honestly and change its view if the evidence went against it. Researchers deliberately strengthened its preferences.
The result does not generalize to every use of AI: the study covered one task and one model. The direct effect on the final recommendation was not significant under the primary statistical comparison. The authors also report fewer serious errors among agent users, but the 11 self-selected non-users were not a comparable control group.
Amaç Herdağdelen, Hailey Joren and Alex Tamkin see a problem for human oversight, including in AI safety research. Signing off an analysis may not mean reaching an independent judgment. They propose making important decision points visible and giving experts room to form their own assumptions.