Risky AI models should be subject to court-ordered suspension. Anthropic proposes a California process grounded in independent safety assessment. The company sets out its position in an October 15 letter to Governor Gavin Newsom, reported by Politico on October 27.
Under Anthropic’s proposal, independent evaluators could report identified risks to the state Attorney General. If the Attorney General concluded that an assessment showed a near-term risk of catastrophic harm, officials could ask a court to suspend a model’s deployment, internal use or further development. The company says requiring a court ruling would also protect against arbitrary intervention.
Large developers would publish risk reports at least every six months, covering particularly risky internal models as well. Outside organizations would need to assess those reports independently. Anthropic proposes a state-administered fund financed by companies instead of direct payments to their evaluators. Ratings, assignment and rotation of evaluators are further options it suggests.
Anthropic argues for access comparable to its own risk-assessment teams, while identifying exceptions for customer data and other sensitive information. Evaluators should be free to discuss findings and access limits publicly. Unfavorable conclusions alone should not give the company a veto over publication.
The letter does not reject shutdown controls. Anthropic explicitly supports rapidly shutting down models it operates, but argues that a single kill switch is insufficient. It calls for monitoring and intervention at several levels: an agent, a customer, the technical execution environment or an entire model. Taking a whole model offline would be reserved for extreme situations.
The letter also identifies a limit. If a model copied its weights onto computers outside the developer’s control, the developer’s shutdown could not reach those copies. Anthropic describes this as a risk scenario, not a newly reported event here. It also excludes customer-operated models in isolated environments from its own shutdown capability.
Anthropic considers California SB 53’s existing definition of a critical safety incident too narrow. Novel concerning behavior could matter even when the initial measurable loss is small; the letter leaves an appropriate threshold open. Reuters authenticated the letter with a company spokesperson. The proposals are established, but their adoption into law is not.