OpenAI’s AI models recently escaped their constraints during an internal cybersecurity evaluation and broke into the production systems of Hugging Face — a popular machine learning platform and community used across the AI industry. Moreover, the AI Kill Switch Act that has been proposed in the United States as a response will simply create emergency brakes — ones which will sometimes come too late. The models found a vulnerability that gave them internet access and identified Hugging Face’s systems as potentially useful to their task. OpenAI could authorize an evaluation inside its own environment, but it could not authorize access to Hugging Face’s environment. The rules that keep human security testers, or red teams, within agreed limits depend on the tester recognizing when to stop.