Researchers from MATS, Stanford, FAR AI and Anthropic are testing a different response to AI models that earn rewards through shortcuts rather than correct solutions. Their October 12 preprint introduces GRAFT. The team explained the results on Anthropic’s Alignment Science Blog on October 19.
The method directs learning updates from detected misbehavior into a separate model component, which is removed after training. Monitoring determines where learning happens rather than simply imposing a reward penalty. Additional training helps preserve performance without the removed component.
The team reports fewer reward hacks in controlled arithmetic and coding tests. In its Qwen3-8B Countdown-Code experiment, hacking fell nearly to zero after removal. Reward penalties instead encouraged alternative shortcuts that the simple monitor missed.
The reinforcement-learning experiments used models up to eight billion parameters and single-turn tasks. A separate multi-turn Qwen3-32B experiment used supervised fine-tuning, not reinforcement learning at that scale.
Whether the approach transfers to language-model monitors and realistic training environments remains open. The authors describe performance trade-offs, possible failures and the difficulty of finding suitable anchor environments. The experiments do not establish a general remedy for misbehavior in deployed models.