For most Americans, warnings about AI safety have long sounded abstract, but that is quickly changing. Months of reports about autonomous swarms of AI agents breaking out of their assigned work and reaching systems around the world have caught the attention of people who never gave the subject much thought. And industry leaders themselves now warn that if we cannot constrain increasingly capable systems, the potential consequences will reach far beyond a compromised website to include potential human extinction. These are no longer forecasts about what AI might do someday; they are reports of what it is already doing.
This week, Washington offered two contrasting approaches to this growing problem. President Trump gathered leading technology executives to sign what he called a “morally binding” accord: their companies would monitor their own systems, work with outside evaluators, and send findings to committees of their own boards. Representative Ro Khanna, on the other hand, has outlined legislation that would create a federal AI safety agency and a number of other far-reaching technical safeguards, as well as regulatory accountability. We have not yet seen the bill text, but the choice between a voluntary agreement and enforceable public oversight is already clear.
I don’t believe the majority of Americans are willing to put their faith in some of the world’s richest men, whose future fortunes depend on outcompeting one another to build the most capable AI system. They may sincerely want their systems to be safe, but that does not give the public access to the evidence or anyone outside those companies the authority to act when safeguards fail. The difference between a morally binding promise and a legally binding duty is worth examining.
The Public Has Already Decided
The public is already telling us that confidence has not kept pace with adoption. Pew found this summer that 52% of Americans are more concerned than excited about AI in daily life, up from 37% in 2021. A March Quinnipiac poll found that 55% believe AI will do more harm than good in their daily lives. People are using these systems, often finding them useful, and still wondering who is responsible for what happens when they fail.
That concern becomes especially visible at the county planning board. Gallup found that 71% of Americans oppose an AI data center in their area, compared with 53% who oppose a local nuclear plant. At least 75 data center projects worth roughly $130 billion were blocked or delayed in the first quarter of 2026. Some objections concern land, water, power, and cost, and they deserve to be heard on their own terms. I also see a broader problem of agency: people have little say in how frontier models are trained or released, but they can speak at a zoning hearing. That is where a larger frustration becomes concrete.
The same public is willing to distinguish oversight from a ban on innovation. In a June AI Policy Institute survey, 68% of likely voters favored formal government review of the most advanced models before wide release; 20% preferred company testing followed by intervention if problems emerged. Those numbers do not settle the design of a law, but they should put to rest the assumption that Americans are content to let developers make the release decision alone.
What We Need to Constrain AI
Other industries with the power to cause widespread harm have taught us the value of enforceable oversight, and recent events give us reason to take seriously where AI systems are heading. Effective AI safety legislation should be enacted before a catastrophe forces the issue. Trump’s pledge calls for internal controls, internal review, an external evaluator, and a committee of each company’s board to receive reports. Those steps may be useful, but the companies still control the process; the public needs duties established in law and an independent institution with the power to enforce them. Here is what that legislation should contain, much of what is contained Khanna’s proposed bill:
- Create an independent AI safety agency with authority to set standards, inspect frontier labs, and intervene when a system poses an unacceptable risk. Trump has floated a 10-person committee to oversee the industry, but has announced no powers for it to require evidence, enforce standards, or stop a dangerous deployment. An agency charged with protecting the public needs those powers in law.
- Require licenses at the points where risk changes: training a frontier model, giving it tools and outside access, and deploying it widely. Developers should have to demonstrate what a system can do, where its boundaries are, and how those boundaries were tested. The agency must be able to condition, suspend, or deny approval when the evidence does not support release.
- Put independent auditors inside frontier labs and have them report to the agency, not the companies they inspect. The accord asks each company to partner with an external evaluator and send findings to a committee of its own board, but does not give an independent public authority access to those findings. An evaluator whose access and reporting line depend on the company cannot carry the public’s trust alone.
- Set enforceable technical standards for containment, isolated testing, network access, shutdown controls, and continuous monitoring. The Hugging Face incident showed agents finding ways to communicate, reach the internet, and compromise systems beyond their assigned evaluation. The law should require developers to test controls against behavior they did not intend, not merely document the behavior they expected.
- Restrict systems that can autonomously alter their objectives, capabilities, containment, or shutdown controls until safeguards can be demonstrated and independently approved. “Recursive self-improvement” needs a definition that covers consequential changes made through tools or other agents without treating every automated improvement as equally dangerous. A narrow rule we can test and enforce is stronger than a sweeping phrase with an easy workaround.
- Impose liability and require insurance so the organizations creating these risks bear the costs when safeguards fail. Reserve criminal penalties for knowing acts such as disabling required logs, containment, or shutdown mechanisms. “The AI did it” must never become a way for the people who trained and deployed it to avoid responsibility.
- Require prompt, mandatory incident reporting to the affected organization and the regulator when there is credible evidence of unauthorized access or significant harm. An OpenAI agent accessed nonpublic files in an Australian government portal in June, but the affected agency was not notified until September. The accord sets no comparable duty to warn, while legislation can require an early notice with a technical contact and preserved evidence, followed by a fuller report.
In addition to these legislative actions, we must pursue international agreements on minimum safety practices, incident communication, and verification, alongside appropriate controls on dangerous capabilities. Trump says cooperation with President Xi on AI governance could hinder American companies, but agreeing on safety limits does not require sharing our best models or surrendering our lead. The United States and the Soviet Union managed nuclear risks while competing elsewhere, and AI deserves at least that level of seriousness.
The AI accord signed this week suggests these measures might become law “over time,” but as we are weeks from a midterm election, it is difficult to see a comprehensive AI safety law taking shape before the next Congress convenes in January. That political reality does not slow an agent down or give an affected organization back the time it loses while waiting to hear that its systems were accessed. The incidents are accumulating now, and so is the evidence of what our controls must be able to do. Congress should use the months ahead to put real legislative language on the table, test it in public, and be ready to act when it returns. We have a chance to write the rules before a catastrophe forces us to.
