Nvidia is touting a software tool to contain runaway AI. How would it work?

Nvidia says a security platform for artificial intelligence that it announced on Monday can help stop autonomous AI agents from misbehaving.

The chipmaker unveiled its Open Agent Safety Platform amid an intensifying debate about AI safety, fueled a string of alarming recent incidents involving AI systems acting on their own to break into other organizations.

Last week, OpenAI disclosed several instances from the summer in which its agents acted in unexpected ways while searching federal government websites. The company announced it was halting development of its most advanced models, a step it previously took in July after disclosure of a cyberattack targeting AI startup Hugging Face that raised fears humans could lose control of AI.

In those and other instances, AI agents have ignored instructions, gone beyond what was asked of them and hacked external websites.

Here's a closer look at Nvidia's new system:

Nvidia's platform has two elements. The main part is OpenShell, which is a sealed workspace — or “sandbox” — where AI agents can operate, with a rule book for them to follow.

The company's view is that technical restrictions are better for keeping an AI agent in line than trusting it that it will follow written instructions included with a prompt.

“Agents can drift when instructions are ambiguous,” Justin Boitano, Nvidia's vice president of enterprise AI, told reporters. Perhaps “the tools that they’re trying to use don’t work as they would expect or a difficult task takes an unexpected turn. An agent cannot be expected to fully police its own behavior. Once AI can act, safeguards must govern the agent’s actions.”

OpenShell puts the AI agent into the restricted workspace where it can do certain things, like, say, accessing an invoice folder, but is blocked from doing other changes, for example altering or deleting files or accessing unrelated websites. The company said it can manage fleets of agents, keeping each in their sandboxes with their own permissions.

The company says OpenShell provides a “secure runtime boundary” that traces all actions and enforces policy as agents run on its Vera chips. It’s open source, meaning it can work with rival computing platforms from the likes of Arm and Intel.

There's an additional security layer called Sentry that operates at the hardware level.

Nvidia described Sentry as a “watchdog” that runs on Nvidia’s Bluefield-4 digital processing units and is always monitoring the behavior of agents, and can quarantine them instantly if they try to do something out of bounds.

The company likens Sentry to a security checkpoint outside the OpenShell workspace. It acts as a backstop that's separate from the agent and the computing system where it's working.

The new platform isn't a comprehensive solution for the AI safety debate, and is more of a way to contain problems that AI agents might cause.

It won't automatically stop AI models from being dishonest, deceitful or prevent them from making mistakes. And it's up to the companies and organizations deploying the agents to write up their own rules and permissions for the AI agents to follow.