AI researcher Leo Gao is calling for stronger evidence that researchers genuinely understand a language model's neurons. In an October 17 post, he welcomes Goodfire researcher Nick Cammarata's new observations but rejects the conclusion that interpretability is solved or has become vastly easier.
Gao proposes simulating neurons from an explanation and measuring how much this changes the model's output. Similar-looking activation patterns could mislead. Weak activations might also carry important information; he says more concrete results on that point are still forthcoming.
Cammarata's original essay describes his approach as a working hypothesis. He sees new opportunities to study neurons directly and calls for experimental tests, while arguing that some previous simulation metrics are unsuitable. A detailed technical companion paper is still to come.
Google DeepMind researcher Neel Nanda recommended Gao's post on October 17 as a counterpoint to exaggerated success claims. Gao and Cammarata had already worked together on OpenAI's 2023 research into automated neuron explanations.