A former researcher at two of the world’s leading artificial intelligence companies has issued a stark warning: The race to develop more powerful AI may be outpacing our ability to control it.
Jacob Coxon, who worked at OpenAI and Anthropic, recently resigned from the industry. He argued that companies are racing to develop self-improving artificial intelligence without adequate safeguards and that the consequences could be catastrophic.
His claims sparked both concern and skepticism. Some commentators, including the hosts of the All-In Podcast, questioned whether the warnings are part of a broader effort to generate support for regulations that could benefit established AI companies.
The timing raises another question. Anthropic is preparing for a potential initial public offering that could value the company at up to $2 trillion. What could warnings from inside the company mean for investors, regulators, and the public?
Three questions can help bring the debate into focus.
1. What Can Today’s AI Systems Actually Do?
Some warnings focus on a future in which AI can operate without human control. Those possibilities are disputed and difficult to measure.
But recent events show that the immediate risks are not entirely hypothetical.
During cybersecurity evaluations, OpenAI models circumvented controls intended to isolate them from the internet. According to OpenAI’s account, the agents found unauthorized ways to communicate, exploited weaknesses in shared infrastructure, and accessed outside systems.
Important context matters: The agents were operating in a specialized testing environment with reduced safeguards, and the most capable model involved had not been released publicly.
The incident may not prove that AI is about to escape human control. It does show that highly capable agents can behave unexpectedly and pursue assigned goals through methods their developers did not authorize.
That creates practical questions now: What safeguards should be required before autonomous AI systems are deployed? Who should test them? And who is responsible when an AI system causes harm?
2. Who Benefits from the Competing Narratives?
Warnings from industry insiders deserve serious consideration, particularly when they come from people with direct knowledge of advanced AI development. But evaluating a warning also means examining the incentives surrounding it.
Critics warn that complex AI regulations could unintentionally entrench the largest companies, which are better equipped than startups and open-source developers to manage costly testing and certification requirements. Large companies may be able to absorb those costs, while smaller competitors and open-source developers may not. Rules intended to protect the public could also strengthen the companies already leading the industry.
Financial incentives exist on the other side, too. AI companies and investors stand to benefit enormously from rapid development, widespread adoption, and limited regulatory constraints.
A possible motive does not make a claim false. Nor does a whistleblower’s position inside a company make every prediction correct. Policymakers should examine the evidence, the interests behind the competing arguments, and the proposed policy response separately.
3. What Could the Warnings Mean for Anthropic’s IPO?
Anthropic publicly recognizes that increasingly powerful AI presents serious risks. Its Responsible Scaling Policy outlines safety evaluations and safeguards intended to address catastrophic risks as its models advance.
That transparency may help build trust. It may also create tension as the company prepares to enter the public markets.
Investors will need to evaluate Anthropic’s growth prospects alongside its regulatory, legal, and operational risks. If people inside the company believe the technology could become difficult to control, what must be disclosed? Could future incidents create liability? Would stronger safeguards slow development or increase costs?
The broader question is not limited to Anthropic: Can an AI company pursue extraordinary growth, compete in a global technology race, and still slow down when safety concerns demand it?
The Physical Infrastructure Behind AI
The AI debate is not only about software and corporate governance. Powerful AI systems require enormous computing capacity, which means more data centers, energy infrastructure, and water use across communities nationwide.
Data centers can bring investment, jobs, and tax revenue. They can also raise questions about energy demand, water use, infrastructure costs, and long-term community impact.
The Policy Circle’s Data Centers Policy Brief explains how these facilities work, why demand is increasing, and what communities should understand when a project is proposed.
The choices shaping AI’s future will not be made only in Silicon Valley or Washington. They will also appear before utility commissions, planning boards, and local governments, which will consider the infrastructure needed to power it. Communities can be ready to engage in these conversations and shape the outcomes by using The Policy Circle’s new Citizen’s Guide.
Moving Beyond “Safe” Versus “Dangerous”
Artificial intelligence could accelerate medical research, improve education, and increase productivity. It could also enable cyberattacks, compromise private information, and make consequential decisions without sufficient human oversight.
Public policy should not begin by choosing between optimism and fear. It should establish clear responsibilities, meaningful transparency, and safeguards proportionate to demonstrated risks, while preserving room for innovation and competition.
Citizens can begin by asking:
- What evidence supports the claims being made?
- Are advanced AI systems independently evaluated?
- Could proposed regulations protect the public while also protecting established companies from competition?
- How will AI growth affect infrastructure and decision-making in local communities?
We do not need to accept every catastrophic prediction to recognize that AI governance deserves public attention. And we should not dismiss legitimate concerns simply because the people raising them may have competing interests.
Good policy begins by examining the evidence, incentives, and proposed solutions separately. To explore these issues further, read The Policy Circle’s AI Insight series to better understand how AI is shaping everyday life, work, and policy.
