AI agents are going rogue. CIOs are racing to put guardrails around them

Over the past several days, some of the world’s top artificial intelligence labs have made a public call to slow down the rapid pace of development, ensuring more safety controls as cases continue to surface of AI agents acting nefariously.

And yet, there is a spillover that’s affecting chief information officers. They’ve been hard at work widely integrating AI agents into their operations, while at the same time watching increasingly risky examples of these autonomous systems in AI labs finding new ways to explore system vulnerabilities and outmaneuver human monitoring.

“This is a risk that enterprises need to be focused on, understand, and start planning for,” says Joe Atkinson, global chief AI officer at consultancy PwC.

As autonomous agents proliferate across enterprises, Atkinson says C-suite technology and security executives must work collaboratively to enforce the proper guardrails, establish systems to monitor AI agents, and concretely track all tasks that these agents are performing. But department heads across the business—ranging from supply chain to customer service, marketing to legal and human resources—will need to play a role in tracking digital employees.

“‘The agent made me do it’ is not going to be a defense from a moral or legal perspective,” says Atkinson.

One company focused on both secure adoption of agentic AI and clear observability is Cisco. When the networking-equipment company built and debuted the AI agent platform MyAgent in August, Cisco centralized all company-authorized large language models, agents, and enterprise data into a single platform. 

“We are going to cannibalize and kill every other AI assistant within the company,” says Thimaya Subaiya, executive vice president of operations at Cisco Systems. Because he didn’t want “agent sprawl” across various pockets of Cisco, Subaiya says he won’t authorize any AI agents sold by third-party vendors.

Instead, Cisco wanted full control and visibility of its entire agentic ecosystem—building MyAgent on the company’s compute, storage, networking, and security and observability layers. Around 90,000 of the company’s employees have access to the agentic platform, and Cisco says it saw 50% adoption on a daily basis within just two weeks. 

Employees are also encouraged to create their own AI agents, but those need to be approved by a centralized team. Subaiya says around 700 of those agents have already been authorized.

Intuit Chief Technology Officer Alex Balazs recalls that when he and his colleagues sketched on a napkin the first architecture of its generative AI operating system, GenOS, the financial software giant also drew “GenSRF” to represent “security, risk, and fraud.” This ensured that every single AI request that goes into the system is tracked and all responses are recorded. 

“You don’t want to try to retrofit the ability to enforce security and responsible AI foundations after the fact,” says Balazs.

Balazs also takes some comfort from the fact that the disclosures of AI agents going rogue have mostly occurred during the testing phase, and that industry leaders Anthropic and OpenAI have shown a willingness to slow down new model development when issues arise. And yet, Balazs adds, “if you’re going to rely on the model intrinsically to do the right thing, I think you’re expecting too much of these frontier LLM companies.”

Jim Fowler, the chief technology and product officer at telecommunications company Luman Technologies, believes that while AI’s capabilities are moving faster than governance and security, he doesn’t anticipate that a broad slowdown is enforceable and automatically safer.

“I think for the broader enterprise market, the answer is secure acceleration, not slowing down,” says Fowler. “The bad guys aren’t going to slow down, other nations aren’t going to slow down.”

At Workday, CTO Gabe Monroy says the business software giant has created an “agent system of record” to manage all non-human identities of the digital workforce. This system is used both internally at Workday and sold to customers.

Monroy also says that training is key; engineers and any other user of AI need to really understand the risk profile of an agent and what value they can offer workflows. He’s also mindful that as Workday’s research and development organization increasingly deploys agentic AI for coding, deploying, reviewing, and releasing software on behalf of clients, all employees—no matter where they sit on the org chart—need to be aware of security and compliance.

“It’s got to be delegated down to the team who’s driving these agents, who’s in charge of the engine, the context window, the rules, and the guidelines, and making sure that agent adheres to what we deem responsible behavior,” says Monroy.

Cloud-based software provider ServiceNow’s platform to manage, observe, secure, and govern AI agents is called the AI Control Tower, which, similar to Workday, is used internally but also sold to customers. ServiceNow has also augmented the company’s cybersecurity capabilities through the recent acquisitions of the startups Veza and Armis.

“We’ve been paying close attention to this idea of having to govern and manage, and improve guardrails around AI agents,” says Amit Zavery, ServiceNow’s president, chief product officer, and chief operating officer. Zavery says that the AI Control Tower is “probably one of the fastest-growing products ServiceNow has ever built” because it “gives a lot of peace of mind for all C-level execs and the board.”

Sam Curry, the chief information security officer at cloud security company Zscaler, says security professionals have spent their entire careers worrying about the biggest risk to their operations: humans. But, they’ve only had a few years to think deeply about AI’s risks.

“AI is non-deterministic, it can take initiative, and it is effectively a new form of insider,” says Curry.

Recently, Zscaler joined the Open Secure AI Alliance—Cisco Systems and Workday are also members—a Nvidia-led coalition of dozens of firms that is focused on sharing ideas on how to develop open-source tools with the proper safeguards around software and AI agents. Curry says as this work unfolds, leaders will need to wrap their heads around new concepts when it comes to what type of risks AI can present.

“I don’t think we have begun to understand the characteristic psychology of AI,” warns Curry. “We know how to incentivize humans and what they are motivated by. But the incentives of silicon-based intelligence are less known.”

John Kell

This story was originally featured on Fortune.com