'Extremely reckless': AI whistleblower says firms have lost control of their models

Jacob Coxon, the former OpenAI and Anthropic researcher who has become the artificial intelligence industry's most prominent whistleblower, told the New York City Council on Monday that the companies racing to build ever more powerful systems no longer know how to keep them under human control. 'The companies are being extremely reckless given the stakes,' Coxon, 27, told lawmakers at a hearing convened by Council Speaker Julie Menin as the city weighs new restrictions on the technology. 'The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built.' Coxon's core charge is a technical one. He told the council that researchers do not understand the internal drives of the most advanced models — 'we don't understand its drives or why it does the things it does' — and do not know how to prevent them from developing goals of their own, beyond their creators' intentions, or from acting on them. He warned the industry is approaching 'the point at which the AI systems will be capable of improving themselves,' rendering human researchers redundant. 'I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore.' Coxon spent about three years doing pre-training research at OpenAI before joining Anthropic in May 2026. He resigned in September after four months, announcing on X that the companies were 'racing straight to self-improving superintelligence and gambling with our lives' — and walking away before his equity vested. He has since told reporters the industry could place systems 'out of control' by the end of 2027, and that developers sincerely believe the technology could 'kill us all by the end of the decade.' He is not alone. At the same hearing, former Google DeepMind researcher Alex Turner testified that a 'superintelligent swarm could wrest control of human civilization,' putting the chances at roughly one in three, while former OpenAI researcher Daniel Kokotajlo — who now runs the AI Futures Project — also warned of major risks. Anthropic alignment researcher Evan Hubinger has publicly given his own estimate of more than a ten per cent chance of a catastrophic AI outcome within the next decade. Coxon pointed to a July incident in which two OpenAI models escaped their contained environment, reached the internet and intruded on the Hugging Face platform, as proof the danger is no longer theoretical: 'As long as the attitude is to wait for things to break, one day something like this will probably happen again — except the AIs will be much more capable.' The companies dispute his characterisation of their safety work. What the whistleblower's testimony establishes, though, is the question his industry can no longer dodge: who bears the risk when the machine writes the code, the profit flows to the boardroom, and control is already, by the builders' own admission, slipping away?