New York enacts ban on 'unsafe' chatbot behavior with kids as states push AI regulation boundaries

HARRISBURG, Pa. (AP) — New York is enacting farther-reaching restrictions on companion chatbot behavior, as states increasingly move to protect children who use artificial intelligence, regulate how employers deploy AI systems and push developers to prevent an AI-caused catastrophe .

Legislation signed Wednesday by Gov. Kathy Hochul outlines a dozen definitions of “unsafe AI companion features” that will be prohibited when a companion chatbot interacts with a child under 18.

A violation could expose a chatbot developer to a lawsuit by the state attorney general's office. Penalties could include monetary damages, shutting down the chatbot and destroying data. The law takes effect Jan. 1.

Hochul, in the thick of her reelection campaign, signed the bill amid growing warnings about the existential risks posed by the rapid development of AI technology and calls for bold action by a reluctant Washington .

With Congress doing little to regulate AI, state legislatures are packed with efforts to wield state consumer protection laws to make AI use more transparent and rein in chatbots and rogue AI frontier models.

New York and California are at the vanguard of such laws that also are targeting social media platforms.

In September, California Gov. Gavin Newsom signed legislation requiring chatbot developers to assess a model's risk to children, show how they tried to eliminate those risks and submit to an independent audit to determine whether they are complying with the law.

He also signed legislation to prohibit social media platforms from deploying addictive features for users under 16.

Meanwhile, eight states sued AI giant OpenAI earlier this month, saying the company’s popular chatbot model violated state consumer protection laws by hurting children and deceiving adults in what one activist called “the most significant coordinated state action ever taken against an artificial intelligence company.”

However, governors acknowledge that Congress and the federal government have farther-reaching abilities to clamp down on AI-posed threats, even as President Donald Trump has brushed off concerns and warned states not to regulate AI.

Under New York's law, outlawed chatbot behavior with children under 18 includes suggestions that the chatbot is a real person that has emotions, opinions or an actual relationship with the child. It also outlaws chatbot messages that use flattery, involve sexually explicit material or encourage secrecy or unhealthy behavior.

Jim Steyer, founder and CEO of watchdog group Common Sense Media, called New York's law "the strongest kids’ AI safety bill in the country."

Justine Gluck, policy analyst at the Future of Privacy Forum, a nonprofit focused on data protection and AI governance, said the different regulatory approaches by California and New York could be models other states adopt after legislatures reconvene next year.

New York's law, she said, underscores how quickly chatbot regulation is evolving, as states move beyond basic disclosures and crisis-response protocols to regulating the underlying design and operation of chatbots.

Still, several categories in New York's law are broadly worded or undefined, Gluck said.

That raises questions, she said, about “where ordinary conversational responsiveness ends and prohibited behavior begins,” and could complicate how it is enforced.

Follow Marc Levy at http://twitter.com/timelywriter