.png)
2 hours ago
2
Hello and welcome to Eye on AI. In this edition:
- AI’s X-risk breaks into the mainstream
- Anthropic CEO Dario Amodei calls for a coordinated industry safety effort
- Anthropic details attempts to misuse its AI models
- China’s top spy warns AI could pose a risk to the Communist Party
- OpenAI is violating California’s new AI safety law, watch dog group says.
- Half of companies aren’t following their own AI governance policies, E&Y survey says.
In the past few days, I’ve heard a lot of people repeating that old saw—often wrongly attributed to Vladimir Lenin—about there being “weeks when decades happen.” It certainly seemed to be one of those weeks in AI. Concern about existential risk has been a strain of AI discourse for decades. But, despite occasionally making headlines when someone like Elon Musk, Sam Altman, or Geoffrey Hinton would express their fears about AI posing a grave risk to the species, it never really cemented itself in the general public’s consciousness in the way, say, climate change, or the risk of nuclear war, has. If politicians debated AI regulation at all, the discussions centered around data center construction and utility bills, jobs, education, mental health, algorithmic discrimination, and civil liberties, not the risk of rogue AI killing people—maybe even all the people. Until now, that is.
The drumbeat of dire warnings from employees resigning from—or in some cases still working for—Anthropic, OpenAI, and Google DeepMind, all saying that the leading AI companies are developing the technology recklessly and risking human extinction has dominated the global news cycle for an entire week (which is really saying something in this day and age.) AI company CEOs and politicians have been stirred to respond. After years in which both domestic AI regulation and efforts at some kind of international AI governance regime had mostly stalled, suddenly the air is electric with possibility.
My Fortune colleague Nick Lichtenberg had a good story on why the resignation jeremiad of former Anthropic and OpenAI safety researcher Jacob Coxon had such impact when previous warnings, often from much higher-profile individuals, did not. The short answer is that coverage of the Hugging Face incident and other “rogue AI” episodes as well as people’s own experiences using AI agents seems to have opened the Overton window on discussing “loss of control” dangers. The timing, with Anthropic on the verge of an IPO and OpenAI edging closer to one too, also no doubt played a role.
The question now is what happens next? Fortune editor-in-chief Alyson Shontell sat down with Altman on Friday to ask him those questions for her “Fortune 500: Titans & Disruptors of Industry” vodcast (we just call it “Titans” for short.) Altman said the company was in favor of coordinating an industry-wide slowdown in the pace of AI development with bitter rivals, including Anthropic CEO Dario Amodei and SpaceX CEO Elon Musk—two men with whom he has had acrimonious and, in the case of Musk, litigious disputes—as well as Google DeepMind, Meta, and perhaps others. He hinted that such discussions were already underway and that a coordinated slowdown might be announced soon. He also said that, if necessary, he would have no problem telling investors that OpenAI had taken actions to prioritize safety that had cost them financially—and that OpenAI’s investors were warned of this possibility going in. He also definitively said OpenAI would not go public this year, in part due to the current concerns about the safety of the latest AI models, but also, he hinted, because OpenAI’s business isn’t yet in the right place. You can check out the full vodcast episode here. It’s well worth your time to watch.
A coordinated slowdown?
After Alyson’s interview, Amodei put out a blog post also calling for a coordinated slowdown or pause among frontier labs in democratic countries. He said that in some cases, though, coordinating with other AI labs would require an antitrust exemption from the government. He also said that Anthropic would appoint independent evaluators to be permanently on-site at its offices to review its safety work. (He mentioned the nonprofit AI evaluation company METR as his preferred partner for this.) He also said that the U.S. and other democracies should try to strike some kind of international AI governance agreement with China and authoritarian states, if possible. Altman quickly came out and endorsed most of what Amodei said—in particular saying that OpenAI would also embed outside evaluators alongside its research teams—although he was careful to note that “pacing does not mean stopping.”
In the wake of Coxon’s warnings and Amodei’s call to action, a number of U.S. lawmakers introduced legislation or renewed efforts to push forward existing bills. Some, such as a bill introduced by Vermont independent Sen. Bernie Sanders, call for an outright ban on the development of “artificial superintelligence” and mandate that U.S. AI companies pause current research until safety techniques improve. Others, such as a bipartisan bill from Republican Sen. Ted Cruz, Senate Majority Leader John Thune, and Democratic Sen. Amy Klobuchar, would impose a duty on AI companies to prevent catastrophic harms. There were also calls for Congressional oversight hearings on AI’s catastrophic risks. Former President Barack Obama urged Democrats to put AI governance at the center of their legislative and campaign agenda. Meanwhile, a group of 70 U.K. parliamentarians signed an open letter calling for the British government to ban the creation of artificial superintelligence and work on an international AI treaty.
Trump pushes back
But there was strong pushback from some of the politicians that matter the most. President Trump posted to his Truth Social platform that the only guardrails AI needed “is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that in spades!” He criticized Amodei by name, criticizing him for “pretending to be a ‘perfect little angel’” and said his administration had already stopped Anthropic from “doing bad, or potentially bad, ‘things.’” He said the U.S. already had regulatory power and criminal laws that applied to AI companies and that there was “a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China.” He made similar comments in a phone call to Nvidia CEO Jensen Huang that Huang, with Trump’s permission, broadcast to a live audience at an “All in Podcast” summit. This was followed up by the Republican Speaker of the House, Rep. Mike Johnson, saying that fear of AI was drummed up by the media and that “we’re not going to take stupid, knee-jerk reaction prescriptions on this.” Not to be outdone, Chinese state media also criticized Amodei’s proposals, saying they were “self-serving” and “Cold War tactics” designed to hobble China’s technological and economic rise.
With all of that, it seems the prospects for some kind of executive order mandating improved AI safety are poor. The same goes for any actual legislation, such as a bipartisan bill from Republican Sen. Ted Cruz, Senate Majority Leader John Thune, and Democratic Sen. Amy Klobuchar, that would impose a duty on AI companies to prevent catastrophic harms—at least until after the November midterms. Three points though that have come up in the discussion that are worth addressing.
Are antitrust concerns legit?
One is the debate about whether AI companies need an antitrust waiver to talk to one another about slowing development. Some, such as former Trump administration AI and crypto czar David Sacks, have said the AI companies don’t need such a waiver to coordinate a slowdown. And I agree that we should not grant a broad waiver to these tech giants. But I do think that there are legitimate concerns from the AI companies that any discussion of a pause—or of a coordinated decision not to undertake certain product innovations—could create antitrust issues.
Currently, each new generation of AI models tends to drive down the cost of existing, older models. So limiting the rollout of newer models would potentially keep prices higher for longer for consumers, which would seem to open the AI firms up to antitrust claims. (Matt Levine at Bloomberg had a good column on this.) Also, some of the specific innovations that worry AI safety experts, such as greater use of looped Transformers, also happen to have the benefit of using fewer tokens than forcing a model to spit out its complete reasoning trace in its “chain of thought.” This too has the effect of potentially lowering costs for consumers. So prohibiting this technique on safety grounds would also tend to result in higher prices for customers. Again, that looks problematic from an antitrust perspective. For what it’s worth, Chris Lehane, OpenAI’s chief global affairs officer, has come out and said OpenAI doesn’t think it needs an antitrust waiver to discuss shared safety standards with the other AI companies. He also said that there have already been discussions with Anthropic and Google DeepMind on safety standards. But the issue may be that these standards are voluntary, with no mechanism to compel compliance if one company cheats on its commitments. Enforcing the standards would presumably require government action.
Is product liability law enough?
In an example of the strange bedfellows this issue has created, Sacks and former Biden administration FTC head Lina Khan have both said that existing product liability laws could be used to prevent AI companies from releasing unsafe products. Sacks in particular has said that these laws are the reason no new government agency is needed to police AI companies. But there are two problems here. One is that product liability laws generally only apply to products that are sold to customers. Some of the biggest concerns with AI risks lately—as was the case in the Hugging Face incident—have involved unreleased, internal models that were undergoing development or were only deployed inside the AI companies themselves. Product liability law would not cover these internal models.
What’s more, while the fear of liability lawsuits might deter unsafe behavior by AI companies, it might not—and if it doesn’t, suing the companies after the fact is not ideal. This is especially true if the risks are actually existential ones, such as engineering a bioweapon. Suing won’t help us if we’re dead. But even if the risks are merely bad—like hacking into a single financial institution or hospital, manipulating the stock market, or taking out an electrical grid—suing a company after the fact won’t really provide the outcome society wants. Better to prevent these things from happening in the first place. That’s why we do have agencies that police systemically important financial institutions, regulate air travel, ensure power that grids adhere to certain standards, etc.
What about ‘regulatory capture?’
Finally, Sacks and others, including some on the more libertarian left as well as some of the CEOs of AI companies that are slightly behind the frontier, have attacked the proposal for a coordinated pause and agreement on safety standards as an attempt at “regulatory capture.” The claim is that these companies will write the rules in such a way that their leadership position at the front of the AI race gets locked in. I am not denying that this could happen. But it also seems that there are ways to prevent this from happening. Accelerationists like Sacks act as if all regulation results in regulatory capture. But, as I mentioned in a previous newsletter, UC Berkeley AI researcher Stuart Russell likes to quip that there are more mandatory requirements for sandwich shops in San Francisco than there are on OpenAI or Anthropic. And you don’t see too many restaurateurs complaining about regulatory capture. It is simply not the case that mandatory safety rules always result in regulatory capture.
I would also argue that a certain degree of regulatory friction that happens to privilege incumbent players is a price worth paying for a safe industry in cases where failure poses significant risks to human life or physical and financial health. In fact, the industries that pose the greatest potential risks of mass casualty events tend to have fewer players in them, and yes, the burden of regulatory compliance is one of the reasons. But I think this is a tradeoff the public actually thinks is worth the fact that it may also mean they pay slightly more for certain things. There are only a handful of companies around the world that design and build nuclear power plants, for example; only a handful that make commercial aircraft, too. But these also happen to be some of the safest industries out there in terms of their actual operational records. Would there be more players in these industries if there were fewer government safety rules and inspection regimes? Almost certainly. But is the public screaming about regulatory capture and asking for safety standards on nuclear power plants and aircraft to be relaxed?
With that, here’s more AI news.
Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn
Before we get to the news, just a reminder to check out this week’s episode of our new vodcast Fortune AI Weekly. This week, Bea Nolan and I talk to Substack cofounder and CEO Chris Best about his decision to add an AI writing detection feature to the platform. We also talk about AI doomerism going mainstream and the controversy over OpenAI’s Navier-Stokes mathematical breakthrough. You can check out the vod here on YouTube.
Also, come join me at the Fortune AIQ Summit at the New York Stock Exchange on October 1! We’ll join C-suite leaders Bank of America, Booking Holdings, Citi, Ecolab, Elevance Health, United Healthcare, S&P Global, and more to hear about how they are using AI to deliver the growth, innovation, and transformation that is putting them at the top of their respective industries. It promises to be an afternoon of eye-opening insights and inspiration. You can register to attend here.
This story was originally featured on Fortune.com
English (US) ·