Alignment Research Center researcher Jacob Hilton warns against confusing understandable descriptions of individual neurons with a complete understanding of large language models. In an October 16 thread, he challenges an optimistic interpretation of new work by Goodfire researcher Nick Cammarata.
Hilton accepts that many neurons may have recognizable dominant functions. But descriptions may omit secondary roles, and their completeness remains unclear. A comprehensive explanation would also need to show how neurons connect into circuits.
His scaling argument is that larger models solve more tasks, while understandable examples do not automatically explain their entire increase in capability. Hilton still considers small, well-understood parts useful. That is different from reliably aligning AI by understanding everything.
Google DeepMind researcher Neel Nanda recommended the thread as a contribution to the interpretability debate. Cammarata's original essay calls its approach a working hypothesis and asks for experimental tests. Hilton also corrected a label in his thread: the graph metric he discussed measures completeness, not directly mechanistic faithfulness.