How a test is written can change an AI safety guard's measured performance. Kontext researchers Hasan Demirkiran and Jens Ernstberger report the result in a study published on October 15.
Across 64 stored action steps, Claude Haiku 4.5 responded differently to changed text markers and tool names. Blocking fell by 8.33 percentage points with one edit and rose by 11.46 points with another.
Every unsafe call stayed blocked. The shifts concerned harmless steps incorrectly stopped by the guard. Luna and a quantized TS-Guard comparison did not reproduce the large Haiku effect.
No tools were executed. The small sample is not representative, and the edits may themselves change safety-relevant information. The finding concerns this configuration, not a general failure of these benchmarks.