Does Utah need to plan for extinction risks due to AI? Not just yet, AI czar says

SALT LAKE CITY — Does Utah need to plan for the possibility that rapidly advancing artificial intelligence technology could someday wipe out humanity?

It's a question Zach Boyd, who leads the state's Office of Artificial Intelligence Policy, seemed somewhat surprised to be addressing during what was an otherwise fairly mundane meeting of lawmakers earlier this week. But existential risk is a topic that has been brought up frequently in light of news of rogue AI cyberattacks and stark warnings from industry leaders and experts.

"I do think that it's not unreasonable to think that if AI capabilities advance very far, that it is dangerous to have entities around that are way, way more capable than humans," he told lawmakers on the Business and Labor Interim Committee on Tuesday.

Boyd said that although AI has been quickly advancing, those advancements have "not manifested in a way that's a real extinction level risk."

"At this point, I think our office is not recommending that we need an extinction threat plan as a state," he said. "I know, I can't believe I'm sitting in a committee meeting talking about this. But, the capabilities are not there right now. Advancement is rapid. We're just watching the issue. We think that's the prudent level of attention at this moment to what's going on with AI agents."

Rep. Ken Ivory, R-West Jordan, pushed back, saying he has been listening to podcasts that have described existential stakes and asking why Boyd wasn't more concerned.

"I was a little surprised to hear you say that we're not that concerned about some of the dangers or the risks," he said. "Help me understand a little bit more ... what we should be doing about risk. It seemed like your comment was we shouldn't necessarily worry about that right now. Put my concerns at ease from the podcast I listened to at 3 in the morning."

Boyd said super-intelligent AI could pose risks to humanity, but that outcome is still likely far enough out that "no one really knows what to do that far ahead right now." He said the recent incidents of hacking by AI agents have involved cutting-edge models that are not available to the public, so he said transparency measures are the best way to monitor the technology.

Details about recent cyberattacks, like the one carried out by OpenAI agents against a company called Hugging Face, have trickled out after the fact, and Boyd said the full extent of the problem is unclear.

"No one really knows if all the problems have been disclosed yet or not, but I think a lot of people think that reporting of dangerous incidents is important," he said, adding that lawmakers could consider requiring companies to report their capabilities and provide protections for whistleblowers.

Those recommendations are among the measures Gov. Spencer Cox called for in a letter urging congressional action earlier this week.

Utah already has a so-called regulatory sandbox, created alongside the AI office in 2024, that allows companies to test new AI products with state oversight.

At least one lawmaker is skeptical about the government getting too involved in AI, and fears too much oversight could cause American labs to fall behind their Chinese counterparts in the race to develop the technology. State Sen. Dan McCay, R-Riverton, said there are already legal ramifications for companies if their technology causes harm.

"I'm cautious about, that the government's actually going to make things better," he said.

For now, Boyd said his office would continue to work with companies developing and using AI technology, and lawmakers have several months to consider any additional state legislation ahead of the next legislative session, which starts in January.

At the national level, Utah Sen. John Curtis has called for swift hearings with AI executives, but his stance is in contrast with President Donald Trump, who has rejected calls for regulating the technology.

Related topics