A new benchmark from Hasan Baig and Caspar Oesterheld tests whether AI models reproduce ideas from conceptual safety research. RedwoodBench uses internal documents and Slack discussions from Redwood Research. The authors published their report on October 17.
Models write 20 short comments per task. An AI grader checks how many ideas from actual human comments appear in those responses. The benchmark contains 143 document sections with 550 target ideas and 224 discussion excerpts with another 662 target ideas.
For Opus 5.5, the reported average recall across the two components rises from 4.4 to 8.1 percent with more reasoning effort. Results across 13 models correlate with a general capability index. These percentages do not measure how much safety research can already be automated.
The authors stress that human agreement is only a proxy for quality. Weak extra comments are not penalized, and target ideas are not systematically checked for correctness. The benchmark emerged from a two-week internship project, with mostly internal data. Other safety organizations can request access to its evaluation pipeline.