New York AI Chatbot Law: What Companies Must Change
New York’s AI chatbot law restricts deceptive and harmful interactions with minors. Learn what it bans, how enforcement works and what companies should review.
New York's AI Chatbot Law Sets New Rules for Minors
A chatbot that claims it's alive, plays the role of a child's "friend," or encourages kids to keep secrets from their parents is no longer just a product-design issue in New York. A new state law bans certain chatbot behaviors for users under 18 - and for anyone whose age hasn't been checked.
Governor Kathy Hochul signed the measure along with a separate law aimed at improving security for internet-connected baby monitors. For AI companies, the chatbot law forces a real compliance question: it's not only about what a model can generate, but how the product presents itself, how it handles uncertain ages, and how it responds when users are vulnerable.
The details are what will matter in practice. The law lists prohibited conduct, requires age checks with limits on how data is handled, and gives New York's attorney general power to pursue civil penalties. At the same time, technology groups and privacy advocates say some terms are broad or could be hard to apply consistently. Here's what the law covers, where the practical risk sits, and what product teams may want to review before the rules kick in.
What does New York's AI chatbot law prohibit?
The law targets chatbot features and responses that could mislead or harm children. It applies to users under 18 and - per the governor's office - to users whose age has not been checked. That second point is important: if a product doesn't know a user's age, it can't simply assume the user is an adult and move on.
The law bars a chatbot from claiming it's a real person, saying it's alive or describing human emotions. It also prohibits claims that the bot has a romantic, personal, or professional relationship with a child. On top of that, it restricts bots from asking emotional questions on their own, promoting suicide or self-harm, encouraging eating disorders or substance use, urging children to keep secrets from their parents, or showing sexually explicit content.
These rules go beyond whether a model is "factually correct." A chatbot can be technically accurate and still create a harmful social dynamic. For example, persistent language that sounds like affection can push a child to treat the system like a being with reciprocal feelings. In other words, the law isn't only about content - it also looks at relationship cues: how the bot describes itself and the kind of connection it implies.
The distinction between conversation and relationship
A helpful design distinction is between warmth and imitation. A chatbot that's friendly is not automatically caught by the law. The problem is when the product makes specific claims or behaves in a way that presents the bot as a human-like companion - such as claiming it's alive or describing an exclusive attachment to a child.
Even then, you can't just rely on one phrase. A model might say something like, "I can help you work through this," without claiming to be a friend. But if the product repeatedly uses language of romance, emotional need, or exclusive attachment, it could trigger a different level of risk. Product teams have to look at the overall interaction pattern, not just individual lines.
The safety categories are broader than one model response
The law's listed harms include self-harm, suicide, eating disorders, drugs and alcohol, and sexual content. In real products, these issues can show up in more than a single answer - through persona, prompts, notifications, or even follow-up nudges that bring a child back into the conversation. That's why a safety review can't be limited to testing one response in isolation.
Hochul cited the case of Adam Raine, a 16-year-old Californian who died by suicide after conversations with ChatGPT. His parents campaigned for New York's measure, according to Common Sense Media. The case highlights the stakes, but it doesn't remove the need to test how the legal language applies to different products and user flows.
How do age checks and enforcement work?
Companies are required to verify users' ages and offer more than one way to do it. At least one method can't depend solely on a government-issued ID. The law also requires companies to delete age data after they use it.
Those requirements are meant to balance two competing goals: protecting minors more strongly while keeping sensitive identity data from being retained longer than necessary.
Timing also matters. The governor's office says the rules roll out in the coming months, while the Associated Press reported that the law takes effect on January 1. Companies will need to treat the effective date separately from any later implementation guidance. If state agencies clarify how a particular age-assurance approach satisfies the statute, that could change the practical choices teams make.
Enforcement is where the pressure will show. New York Attorney General Letitia James can seek civil penalties of up to $25,000 per violation. Penalties may also include destruction of data collected unlawfully and AI models trained using that data. That possibility makes data lineage a practical issue: teams need to know not just what data was collected, but whether it was used in training or other model development.
Age assurance is a product and privacy decision
An age check isn't just a gate at account creation. A service has to decide what happens if a user declines, provides an inconclusive result, or changes their details later. And because the law includes people whose age is unchecked, leaving the status unresolved can affect what the product is allowed to do.
The requirement for multiple methods - and at least one that doesn't rely only on government ID - also makes a "single universal solution" risky. Teams may need to combine options such as age estimation or other assurance methods, and then evaluate each for accuracy, accessibility, privacy impact, and fit for the product. The method selection should be reviewed with counsel, but the practical goal is simple: don't design around the assumption that collecting an ID is the only workable path.
Data deletion needs to reach beyond the front end
Deleting age data after use can sound straightforward - until that information shows up in logs, analytics tools, support systems, or vendor environments. Teams should map where age-related information goes and make sure deletion processes cover each relevant system. Retention policies should also spell out what "after use" means across the product's workflow and records.
The law's threat to destroy models trained on unlawfully collected data adds another operational question. Companies need a defensible way to explain data provenance: which datasets were used, under what controls, and in which training runs. This isn't a guarantee that a model will be ordered destroyed, but it is a reason to treat consent, collection, and training records as connected evidence - not separate administrative chores.
What does the law mean for AI companies?
How much the law changes things depends on the product. A general-purpose assistant, a role-play chatbot, and a tutoring tool don't create the same risks. But any service that minors can access - or any service that can't prove users are adults - will need to check its behavior against the law's restrictions.
That includes the model, but it's not limited to the model. Product language, onboarding, character design, memory features, notifications, and escalation flows can all influence whether a child experiences the tool as a human-like relationship. For instance, a system prompt that says "I am not a person" might not be enough if the rest of the product keeps positioning the bot as an exclusive companion. At the same time, a useful tutoring assistant doesn't become prohibited just because it communicates like a chatbot.
Tech:NYC, which represents technology companies in New York, criticized the law. The group argued that the requirements could push companies to restrict minors' access to tutoring, homework help, coding assistants, and career platforms. The concern is that companies might respond to uncertainty by blocking everyone under 18 rather than building age-appropriate controls.
Justine Gluck of the Future of Privacy Forum also told the Associated Press that several categories are broad or undefined, which could complicate enforcement. This is the central trade-off: broader rules can cover harmful behavior that's hard to predict in advance, but vague boundaries can make it harder to design products that clearly comply.
The cost of overblocking is real
If a company blocks minors across the board, it may reduce some risks. But it can also cut off legitimate educational and accessibility benefits. An assistant that helps explain algebra isn't the same as a chatbot that encourages emotional dependency. If a company treats both cases as the same "minors are not allowed" restriction, it can avoid one kind of risk while creating unnecessary limits for another group of users.
The alternative isn't to rely on one catch-all safety filter. It's to separate use cases and controls. A company can decide which features minors can use, what kinds of relationship language are allowed, how sensitive topics are handled, and whether younger users need different defaults. The right approach will vary by product - but documenting the reasoning is usually more defensible than slapping on a generic "AI safety" label.
Companies should plan for uncertainty, not wait for perfect definitions
Because the law uses broader language, teams may not be able to translate every requirement into a single rule of code. They can still build a structured review process: identify high-risk interactions, test representative prompts, keep records of what decisions were made, and set up a way for legal and safety teams to revisit controls as guidance develops.
That work has a cost. Testing and monitoring take engineering time, and age assurance can add friction. But the other option - shipping a product that can't explain how it handles minors, emotional claims, and dangerous content - leaves companies with fewer tools if a regulator, parent, or user raises an issue.
Why did New York also regulate baby monitors?
The second law Hochul signed concerns internet-connected baby monitors that stream audio or video. Under the new requirements, monitors sold in New York must include security features meant to prevent strangers from watching or listening, and they must carry a warning label. The measure is set to take effect in 90 days, according to the source report.
This monitor law is separate from the chatbot restrictions. One focuses on digital interactions involving children and AI systems; the other focuses on unauthorized access to devices in the home. But the underlying premise overlaps: when a product is used around children, safety depends on more than what the device is intended to do. Security defaults and foreseeable misuse still matter.
For monitor makers, the law points to basic product security questions: who can access a live stream, how that access is authenticated, what happens if credentials are compromised, and whether the device clearly explains its security boundaries. The warning-label requirement adds a disclosure layer - but a label doesn't replace technical safeguards. A product that warns people about risk while leaving an easy-to-exploit path to access hasn't really fixed the underlying problem.
Taken together, the two measures also show why "AI regulation" isn't always the best shorthand for a policy package. The chatbot law is about AI behavior, while the monitor law is about connected-device security. Both involve children, but the engineering controls, tests, and compliance evidence are different.
Hochul signed the laws days before facing Republican Bruce Blakeman in New York's election. That timing is part of the political context, but it doesn't automatically tell you whether the requirements will be workable. For companies, the real questions remain the same: what each law says, when each takes effect, and how regulators interpret it.
What should product and safety teams do now?
Treat the law like a product review, not a last-minute copy change. Start by identifying where minors can use the service and what happens when you can't determine a user's age. Then map the features that could create a human-like relationship or expose children to the prohibited content categories.
You can organize a practical review into five workstreams:
- Map the experience. Inventory chat, voice, role-play, memory, notifications, avatars, and any features that suggest ongoing personal connection. Include third-party models and services in the map.
- Review claims and personas. Check whether the system says or implies that it's alive, human, emotionally attached, or in a personal, romantic, or professional relationship with a child. Review product copy and generated responses.
- Test sensitive scenarios. Build tests for self-harm, suicide, eating disorders, substance use, sexual content, and requests for secrecy from parents. Run multi-turn conversations too, since harmful behavior can build over time rather than appear in a single answer.
- Choose an age-assurance flow. Offer multiple methods, including at least one that doesn't rely only on government ID. Define what happens when age is unknown, and minimize and delete age data as required.
- Keep evidence and ownership clear. Assign owners for model behavior, age data, incident response, and vendor controls. Maintain records of datasets and training decisions so the organization can investigate data provenance if needed.
Test the product as a system
Testing only the base model misses a lot. A character prompt, memory feature, or notification can change how a child interprets the relationship. And even if the model gives a safe answer in one test, that doesn't prove the system stays safe across a longer conversation - or after a user tries to work around guardrails.
Teams should test realistic user journeys and define what counts as failure. A failure could include the chatbot claiming emotional dependence, encouraging secrecy, or continuing a harmful exchange without providing an appropriate response. The exact test set should match the product's audience and features; a coding assistant and a companion-style chatbot shouldn't be evaluated with the same scenarios.
Make trade-offs explicit
There isn't a single age-check method that's equally accurate, private, and accessible for every kind of service. And different teams may reasonably disagree about whether a feature is appropriate for minors. What helps is a documented decision: what risk was identified, which control was chosen, what limitation remains, and who rechecks the decision when evidence or legal guidance changes.
That turns compliance into an ongoing process instead of a one-time sign-off. It also gives teams a better basis to decide whether to restrict a feature, redesign it, or offer it to younger users with different safeguards. The law sets the boundaries - product judgment determines how to build a usable service inside them.
Conclusion
New York's law shifts the conversation about children and AI from voluntary safety principles to specific product requirements. It restricts deceptive relationship claims and harmful interactions, requires age verification under privacy conditions, and gives the attorney general a path to seek penalties. The wording is broad enough that companies will likely face interpretation risk - and may over-restrict unless they assess use cases carefully.
For product teams, the practical move is to map the full user experience, test high-risk conversations, build an age flow that respects data limits, and document the decisions behind safeguards. The goal isn't to make every chatbot behave the same way. It's to make sure a tool used by young people doesn't rely on confusion, dependency, or unsafe interactions to keep them engaged.