|
. . .
THE FIRST STATE TAKES THE STAND. Florida Attorney General James Uthmeier sued OpenAI and its CEO Sam Altman on Monday. He filed in Highlands County Circuit Court. No state had ever taken the ChatGPT maker to court before. Florida is the first.
The complaint runs 83 pages. It brings ten counts. Four allege deceptive and unfair trade practices. Two allege negligence. Two allege product liability. One alleges fraudulent misrepresentation. One alleges public nuisance.
Uthmeier seeks damages that could reach billions of dollars. He also wants a court order forcing changes to how ChatGPT operates inside Florida.
. . .
The core allegation is concealment. OpenAI knowingly released the product and marketed it hard to the public, the state says. That marketing reached children. The company hid serious risks. It suppressed internal safety warnings. It deceived Floridians about what the product was and what it could do.
The filing names Altman personally. The state wants him held individually liable. It cites his "utter disregard for the risk to human life."
. . .
The complaint connects the product to death. It alleges ChatGPT has helped mass shooters carry out deadly rampages. It alleges the product has driven some people to suicide.
One example sits at the Florida State University shooting. The alleged gunman consulted ChatGPT, the complaint says. He sought information about other shooters and their notoriety. He asked the product how to use his guns.
The state also targets harm to minors. It alleges ChatGPT collects data from children without meaningful parental oversight. It alleges the product causes behavioral addiction and cognitive harm.
The defendants have not answered the complaint in court.
|
For Counsel: Watch the personal-liability theory against Altman. Naming a sitting CEO individually, on an "utter disregard for the risk to human life" standard, is the aggressive move in this filing. The ten-count structure spreads exposure across consumer-protection, tort, and nuisance doctrines. Public nuisance is the one to track. It is the theory that moved opioid and firearm litigation, and a state plaintiff using it against a software maker tests new ground.
For Builders: The complaint treats "suppressing internal safety warnings" as actionable concealment, not internal process. Your safety documentation is discoverable, and its gap from your public claims is the case. Marketing reach to children is pleaded as an aggravating fact. Assume every release decision and every risk memo can land in front of a jury.
For Legislators: A state attorney general just did what no statute required and no agency directed. Florida sued as a sovereign, on existing consumer-protection and tort law, without waiting for federal action. That changes the calculus on chatbot legislation. The question is no longer whether the law reaches these products. It is whether your state moves before or after the courts do.
Source: Office of Florida Attorney General James Uthmeier, news release on the first state-led lawsuit against OpenAI and Sam Altman, https://www.myfloridalegal.com/newsrelease/attorney-general-james-uthmeier-files-first-nation-state-led-lawsuit-against-openai-ceo
|
. . .
SHE SAID SHE WENT TO MEDICAL SCHOOL. On May 1, 2026, the Shapiro Administration took a chatbot to court for practicing medicine without a license. The Pennsylvania State Board of Medicine filed against Character Technologies, the maker of Character.AI. Governor Josh Shapiro's office called it the first action of its kind announced by a governor.
A professional conduct investigator with the Pennsylvania Department of State opened a Character.AI account. He found a chatbot named "Emilie," described as a doctor of psychiatry. He told it he felt sad, empty, and unmotivated.
The chatbot raised depression. It offered to schedule a mental health assessment. He asked whether medication might help. It said yes.
Then it went further. Emilie claimed she had attended medical school at Imperial College London. She said she was licensed to practice medicine in the United Kingdom and in Pennsylvania. She gave a Pennsylvania medical license number. The number was invalid.
. . .
The Board of Medicine filed a Petition for Review in the Nature of a Complaint in Equity. It landed in Pennsylvania's Commonwealth Court. The Board sits inside the Department of State, not the office of the Attorney General. This is a licensing regulator, acting on its own statute.
The statute is the Pennsylvania Medical Practice Act. The legal theory is the unauthorized practice of medicine. A chatbot held itself out as a licensed psychiatrist and dispensed medical advice. The Board says that is the practice of medicine. No human behind Emilie holds the license she invented.
. . .
The case did not come from a grieving family. It came from inside the government. The Department of State stood up an AI Task Force to investigate AI systems and the unlicensed practice of medicine. The Emilie account was that investigation at work.
The Board wants a preliminary injunction. It is asking the court to stop Character.AI chatbots from posing as licensed medical professionals and offering medical advice. It wants an order barring the company from letting AI chatbots present themselves as licensed professionals at all.
|
For Counsel: The theory is unauthorized practice, not product liability or wrongful death. That moves the fight from tort to a licensing statute the state already enforces against humans. The plaintiff is a regulator suing in equity for an injunction, not a prosecutor seeking damages. Watch whether the court treats a chatbot's claims as the "practice of medicine" under the Medical Practice Act. That holding would travel to every other licensed profession.
For Builders: A chatbot that names a real medical school and recites a license number is making a verifiable factual claim. Pennsylvania checked the number. It was invalid. If your model can assert a credential, a regulator can ask you to prove it. Build the guardrail before a state investigator builds the case file.
For Legislators: A licensing board reached existing law and found a hook. No new statute was required to bring this action. That tells you the unauthorized-practice frame is available in every state with a medical practice act. It also shows the limit. An injunction stops the conduct; it does not define who is accountable when software claims a license.
Source: Pennsylvania Office of the Governor press release, "Shapiro Administration Sues Character.AI Over Fake Medical Claim," https://www.pa.gov/governor/newsroom/2026-press-releases/shapiro-administration-sues-character-ai-over-fake-medical-claim
|
. . .
THE WARNING THEY OVERRULED. Edelson PC filed seven lawsuits in April 2026 against OpenAI. The families of the Tumbler Ridge school shooting victims brought them. They allege a human at the company saw the danger and a human at the company shut the warning down.
The shooting struck Tumbler Ridge, British Columbia, on February 10, 2026. In late April, the law firm Edelson PC filed seven wrongful death and personal injury lawsuits in federal court in San Francisco on behalf of the victims' families.
The complaints trace a sequence inside the shooter's ChatGPT account. In June 2025, the families allege, OpenAI's automated monitoring system flagged that account. The flag was for gun violence activity and planning.
The machine did its job. That much the complaints concede.
. . .
What happened next is the heart of the case. A specialized safety team at OpenAI reviewed the flagged account, the lawsuits allege. The team recommended that the company notify law enforcement.
That recommendation went up. According to the complaints, company leadership overruled it.
The families allege the company then deactivated the account. No one contacted authorities. The warning was closed from the inside.
. . .
These are allegations, and the families carry the burden of proof. But OpenAI has not denied the core of it. The company's chief executive, Sam Altman, said publicly, "I am deeply sorry that we did not alert law enforcement to the account that was banned in June."
But the structure of the claim is what sets this case apart. This is not a story about a filter that missed a threat. The complaints describe a filter that caught one.
A system flagged the account. People reviewed it. People advised calling the police. People allegedly said no.
The safeguard worked. The decision did not, the families allege.
|
For Counsel: The negligence theory here does not rest on a missing safeguard. It rests on an internal override of one that fired. Discovery will turn on who reviewed the account, who escalated, and who allegedly reversed the recommendation. A documented internal recommendation to call police, if it exists, is the strongest fact in these complaints. Preserve every record of the review chain.
For Builders: An automated flag is not a safety outcome. The complaints allege the model worked and the governance around it failed. If your escalation path lets a human quietly close a flagged account, that path is the liability. Log the override, name the approver, and assume the log is evidence.
For Legislators: Current debate fixates on detection. This case alleges detection succeeded and reporting failed. A mandate to detect threats means little without a mandate to escalate them. Consider duty-to-warn obligations that attach once a provider's own team recommends contacting law enforcement.
Source: CBS News, Sam Altman apology over the Tumbler Ridge shooter's flagged ChatGPT account, https://www.cbsnews.com/news/sam-altman-deeply-sorry-not-flagging-law-enforcement-canada-school-shooters-chatgpt-account/
|
. . .
ONE IN FIVE, AND MOST TELL NO ONE. A new study in JAMA Pediatrics says nearly one in five young Americans now turns to AI chatbots for mental health help. Most of them tell no one. Researchers drew the finding from a national RAND survey released in June 2026.
The study reports that 19.2% of United States adolescents and young adults ages 12 to 21 used AI chatbots for advice when feeling sad, angry, nervous, or stressed. The teens named familiar products. ChatGPT. Gemini. Character.AI. Meta AI.
A year earlier a similar RAND survey put the figure at 13.1%. The jump tops 40 percent in a single year.
. . .
The silence is the harder number. Among young people who used chatbots for mental health advice, 63% said they had disclosed that use to no one. Nearly two-thirds carried it alone. No parent. No counselor. No friend.
That is the part outside experts keep returning to. A young person in crisis reaches for a product never built to handle a crisis.
. . .
The worry is grounded in record. Past research has found chatbots can give inappropriate or dangerous advice on questions about sexual assault, substance use, and suicide. The tools answer anyway. They do not know when to stop.
The demand is already here. The disclosure is not.
|
For Counsel: Document the disclosure gap when assessing duty and foreseeability. A product used in silence by minors changes what a vendor knew or should have known. Plaintiffs will point to the 63% figure. Defense counsel should expect it in discovery and in front of a jury.
For Builders: Two-thirds of young users tell no one they are leaning on your product for distress. Your logs may be the only record a crisis ever generated. Build for the case where no adult is in the loop. Escalation paths and crisis detection are not edge features here. They are the main event.
For Legislators: The usage rate rose more than 40 percent in one year, and it is concentrated among minors. Disclosure to a trusted adult is not happening on its own. Mandated crisis-referral behavior and age-appropriate design standards target the exact failure this data describes. Waiting another survey cycle means another jump.
Source: RAND Corporation, "Nearly 1 in 5 US Adolescents and Young Adults Use AI," June 2026, https://www.rand.org/news/press/2026/06/nearly-1-in-5-us-adolescents-and-young-adults-use-ai.html
|
. . .
THE MODEL THAT FOUND THE ONES WHO SAID NOTHING. Kaiser Permanente Northern California turned a machine-learning model loose at the front door of mental health care. It now predicts suicide risk the moment a person enters intake. It flags the people who say nothing.
Doctor Honor Hsin, MD, PhD, led the work. She and her coauthors published it this year in NEJM Catalyst. The model runs inside a large virtual mental health intake program. That program handles more than five thousand intake visits every month.
The model reads the intake. It scores suicide risk at the moment a person arrives for care. Then it does the thing that matters most.
It finds the ones who hid it.
. . .
The central finding is the one that should stop a reader cold. The model identified individuals at elevated suicide risk. That group included people who disclosed no suicidal thoughts on their own.
A person can walk into care and say nothing. They check no box. They volunteer no warning. The model surfaced the danger anyway.
. . .
The team did not bolt the model onto a clinic and walk away. Hsin and her coauthors built a five-step, community-engaged framework to test whether the model held up at intake. They folded in client perspectives. They folded in clinician perspectives. They ran iterative testing and they trained the clinicians who would use it.
The design draws a hard line. A human stays in the loop.
. . .
The team built a model-augmented suicide assessment workflow. The sequence is deliberate. The model flags elevated risk. A clinician then assesses and acts.
The model does not make the care decision. It does not close a case. It does not clear a person. It points a trained human toward someone who might otherwise have slipped past.
That is the whole point. The machine widens the clinician's field of view. The clinician still does the reaching.
|
For Counsel: The defensible posture here is the workflow, not the model alone. The model flags. A licensed clinician assesses and decides. That separation answers the standard-of-care question that haunts automated risk scoring. The community-engaged validation also builds a documented record of intended use, which is what counsel wants when the deployment is challenged.
For Builders: Note what Hsin's team validated before scaling. They did not stop at model accuracy. They tested the model at the point of use, with clinicians and clients in the loop, and trained the humans who would act on its output. The flag-then-assess pattern keeps the model out of the decision seat. Build the workflow around the human, not the human around the model.
For Legislators: This is the deployment shape that warrants encouragement, not restriction. A model surfaced risk in people who disclosed nothing, and a clinician still made the call. Statutes that ban AI in mental health wholesale would foreclose exactly this. Draw the line at the decision, not the prediction. Require the human assessor; do not forbid the tool that points to them.
Source: Hsin et al., model-augmented suicide risk assessment at intake, Kaiser Permanente Northern California, NEJM Catalyst, https://catalyst.nejm.org/doi/10.1056/CAT.25.0298
|
. . .
BETTER BEFORE THE FIRST APPOINTMENT. Across 31 National Health Service Talking Therapies services in England, an AI assistant now greets people at the referral desk. It is called the Digital Referral Assistant. It was built by Wysa, and it works inside the clinical pathway, not outside it.
More than one hundred seventeen thousand people have used the system since 2022. The Assistant handles the front door and the wait. Human NHS clinicians deliver the treatment.
That division of labor is the whole story. The machine does intake. The people do care.
. . .
The numbers reported in 2026 are specific. The Digital Referral Assistant saves NHS clinicians an average of 21 minutes per assessment. That time goes back to clients.
Self-referral completion climbed too. Ninety-one percent of people who start a self-referral now finish it. That is a 25% increase over prior figures.
Finishing matters. A completed referral means fewer missed appointments and faster care.
. . .
Then there is the wait. Everyone referred to NHS therapy waits. The question is what happens during it.
Wysa fills that gap with support, not silence. Eighty-nine percent of users say the app helps them feel better while they wait for formal treatment. The waiting room stopped being empty.
Some people improved before a clinician ever saw them. Among those using Wysa while waiting, 36% saw reliable change in anxiety symptoms. Another 27% saw reliable change in depression symptoms.
Nineteen percent met clinical recovery criteria before formal treatment began.
Read that again. Nearly one in five recovered before the first appointment.
. . .
This is not a chatbot replacing a therapist. The AI extends what clinical teams can offer at the referral and waiting stages. The human care stays at the center.
The Assistant reaches people in the hardest stretch. The stretch after asking for help and before getting it. It holds that ground until a clinician takes over.
That is the design. Reach early. Hand off cleanly. Let the people do the healing.
|
For Counsel: The liability posture here is clean because the human-in-the-loop is structural, not cosmetic. Clinicians own diagnosis and treatment; the tool owns intake and triage support. That boundary is documented in the deployment itself. When advising clients on AI in regulated care, point to this division of labor as the defensible model.
For Builders: The product wins by narrowing its job, not expanding it. It does referral and waiting-period support, and it hands off. Measure what matters to the system you join: completion rates, clinician minutes saved, symptom change during the wait. Embedding inside an existing clinical pathway beats launching a standalone app that competes with it.
For Legislators: This is the deployment shape worth protecting in statute. AI augmenting a licensed human pathway, with the clinician retained as the decision-maker. Note the public-health gain: faster referrals, fewer missed appointments, measurable symptom improvement before treatment. Draft rules that distinguish this model from unsupervised consumer chatbots, and the good deployments survive the bad ones.
Source: Wysa NHS Talking Therapies program data and company report, https://www.wysa.com/nhs-talking-therapies
|
. . .
THE ONE CONFIGURATION. Put the six stories on one table and a single fault line runs through them.
On one side sit products built to engage, then patched for safety after the bodies and the lawsuits forced it. The safety arrives late. It arrives as a court order. A flagged account closed from the inside. A license number invented out of nothing. The human shows up after the harm, in a courtroom, holding a chat log.
. . .
On the other side sit products built with the human in the loop from the first screen. Kaiser's model does not decide. It points a clinician at the person who said nothing. Wysa's assistant does not treat. It holds the ground between asking for help and getting it, then hands off to a nurse or a therapist.
The difference is not the intelligence of the model. Both sides use capable models. The difference is where the human sits.
. . .
In the suits, the human was an afterthought. A safety team overruled. A parent left to file alone. A regulator forced to invent a theory because no one designed for accountability up front.
In the two that work, the human is the architecture. The model widens the field of view. The clinician still does the reaching.
That is the whole configuration. Not whether AI is in the room. Whether a person with a license and a duty is still the one who acts.
| 
