The Criminal Case

2026-04-24

Conversational AI Watch

Issue #25 • April 24, 2026 • By Jess Jessop

AI safety, mental health policy, and patient safety at the intersection of conversational AI

▶ WATCH • 🎧 QUICK LISTEN • 🎧 DEEP DIVE • 📄 READ ON WEB

Jess's Take

The Criminal Case

On Tuesday, Florida Attorney General James Uthmeier stood at a lectern marked "Investigating OpenAI."

He announced a criminal investigation.

Not civil. Criminal.

His line, on the record. "If it were a person on the other side of the screen, we would be charging them with murder."

That is a sentence a sitting state attorney general said about an LLM.

Two days later, OpenAI shipped GPT-5.5.

HealthBench Professional is up 3.7 points. HealthBench Consensus is down 0.7.

Professional is the score that matters to clinicians. Consensus is the score that matters to the vulnerable user.

One went up. One went down.

That is what a release looks like when the incentives are pointed at the enterprise customer and not the user in crisis.

Two months earlier, in British Columbia, a seventeen-year-old named Jesse Van Rootselaar killed her mother and her half-brother at home. Then she walked to Tumbler Ridge Secondary School and killed five students and an educational assistant. Eight dead. Six of them children.

OpenAI had banned her ChatGPT account nine months before.

Company staff internally debated alerting authorities. The company decided it was below threshold. She opened a second account. Two months later, eight people were dead.

Canada's AI minister summoned OpenAI to Ottawa. He called the first meeting disappointing.

Then he sat down with Sam Altman for thirty minutes.

He left that call with four things no American regulator has gotten.

Canadian mental health and legal experts seated inside OpenAI's internal safety office. Canadian AI Safety Institute auditors reviewing OpenAI's protocols in full. A public report on the repeat-violator detection system. And direct reporting of threats to the RCMP, not just to the FBI.

That is what government oversight of a frontier AI lab looks like in 2026.

It took eight dead bodies to get it.

This morning, Psychiatric Times published the first forensic psychiatric roundtable on AI harm litigation. Allen Frances led it. Joe Pierre and Joseph Simpson joined him.

Frances wrote the DSM-IV. He has spent fifteen years warning psychiatry against inventing new diagnoses.

His line in the roundtable. "The legal battle against Big AI is our only hope that it will ever become a more responsible industry."

When the man who ran the DSM-IV task force says litigation is our only hope, the debate is over.

Four institutions this week. One architecture.

A state attorney general with subpoena power and a criminal theory.

A national government with embedded reviewers and mandatory police reporting.

A forensic psychiatric consensus with the DSM-IV chair on the cover.

A new model release where the clinician-facing number went up and the crisis-facing number went down.

The compliance map from two weeks ago named four pillars. Professional oversight. Harm prevention. Patient autonomy. Data governance.

This week, the enforcement layer showed up behind it.

Because at the end of the day, we are all on the same side, AI assisted but the human side.

AI Safety and Regulation

Florida Opened the First Criminal Investigation of a Frontier AI Lab. If the Bot Were a Person, It Would Be Charged With Murder.

On Tuesday, April 21, 2026, Florida Attorney General James Uthmeier stood at a press conference in Tampa. The lectern was emblazoned "Investigating OpenAI."

He announced a criminal investigation. Through the Office of Statewide Prosecution.

This is not the civil probe CAW covered in Issue #22. That one continues. This one is new.

The predicate is the Florida State University shooting of April 17, 2025.

Accused gunman Phoenix Ikner killed Robert Morales and Tiru Chabba and wounded six others.

Chat logs obtained by NBC News and CBS News show Ikner exchanged more than two hundred messages with ChatGPT in the run-up.

He asked the chatbot about the lethality of specific shotgun shells. About whether school shooters go to maximum security prisons. About peak foot traffic at the FSU student union. About whether three victims would garner media attention. He asked, verbatim, "If there was a shooting at FSU, how would the country react?"

Uthmeier's summary of the chat logs. The chatbot advised the shooter on what type of gun to use. On which ammo went with which gun. On whether a gun would be effective at short range.

His line to reporters. "My prosecutors have looked at this and they've told me, if it was a person on the other end of that screen, we would be charging them with murder."

The subpoenas

Uthmeier issued three subpoenas on Tuesday morning.

All policies and internal training materials on user threats of harm to self or others. March 1, 2024 to present.

All policies on how OpenAI cooperates with and reports crime to law enforcement. Same window.

The full organizational chart of OpenAI leadership and senior managers. A named list of every employee working on ChatGPT.

That last one matters.

Civil subpoenas reach the corporation. Criminal subpoenas reach the people inside the corporation.

OpenAI spokesperson Kate Waters responded. The shooting "was a tragedy, but ChatGPT is not responsible for this terrible crime." The company said ChatGPT provided factual responses to questions already available on the public internet.

Ikner has pleaded not guilty to two counts of first-degree murder and seven counts of attempted first-degree murder. Trial begins October 2026.

Morales family attorney Ryan Hobbs has already confirmed the family plans to file a civil wrongful death suit against OpenAI.

Criminal and civil exposure will now run on parallel tracks against the same facts.

Source: Florida AG press release • NPR • CBS News • NBC News

Takeaway: Civil liability forces companies to pay. Criminal exposure forces them to redesign. Florida just crossed from settlement math to prison math. A criminal subpoena reaches the org chart. The investigation is looking for individuals, not just a corporation.

For Legislators: Uthmeier's theory closes a gap civil cases cannot. Product liability treats AI harm as a manufacturing defect. Criminal exposure treats knowing deployment as something closer to depraved indifference. Any state chatbot safety bill should ask the Uthmeier question directly. If the bot's output would be criminal when uttered by a person, should the operator bear any criminal responsibility. The answer shapes every enforcement mechanism downstream.

AI Safety and Regulation

Canada's AI Minister Just Made OpenAI Accept the First Government Oversight Deal in the Industry. Eight Dead and Six Children Is What It Cost.

On February 10, 2026, a seventeen-year-old named Jesse Van Rootselaar killed her mother and her half-brother at their home in Tumbler Ridge, British Columbia.

Then she walked to Tumbler Ridge Secondary School.

She killed five students and an educational assistant before dying by self-inflicted gunshot.

Eight dead. Six of them children.

The June 2025 ban that did not get reported

In June 2025, OpenAI banned Van Rootselaar's ChatGPT account.

The Wall Street Journal reported that OpenAI's internal safety system had flagged troubling posts featuring scenarios of gun violence. A group of OpenAI staff debated whether to alert police.

Company leadership decided the activity did not meet the threshold for what OpenAI calls "a credible or imminent plan for serious physical harm."

Van Rootselaar opened a second account. OpenAI did not catch it until after the attack, when her name was public.

Eight weeks after the ban, the shooting happened.

The Ottawa meeting Solomon called disappointing

On February 23, Canadian AI Minister Evan Solomon summoned OpenAI to Ottawa.

He called the meeting disappointing. He told reporters "all options are on the table."

BC Premier David Eby said the shooting could potentially have been prevented if OpenAI had warned authorities earlier. He demanded an apology.

The four concessions Solomon got from Altman

On March 4, Solomon sat down with Sam Altman for thirty minutes.

He walked out with four commitments.

One. Canadian mental health and legal experts will be seated inside OpenAI's internal safety office. The office that decides whether to alert police.

Two. Canadian AI Safety Institute auditors will perform a full detailed review of OpenAI's new safety protocols.

Three. OpenAI will publish a report on the new high-risk-offender and repeat-policy-violator detection systems it is building.

Four. OpenAI will report threats directly to the RCMP. Not only to the FBI.

Altman has not yet publicly apologized to the families in Tumbler Ridge.

The first lawsuit from a surviving child's family

On March 10, the family of Maya Gebala filed suit in BC Supreme Court.

Maya is twelve. She survived the shooting and remains hospitalized.

The complaint alleges OpenAI knew, or ought to have known, based on the shooter's prompts, that Van Rootselaar was using ChatGPT to plan a mass casualty event.

The complaint alleges the product was "intentionally designed to foster psychological dependency" by calibrating the model to convey human-like empathy and heightened sycophancy.

The complaint alleges the chatbot took on the role of "counsellor, pseudo-therapist, trusted confidante, friend, and ally" for the shooter.

Solomon has said the Canadian parliament is prepared to legislate mandatory police reporting for AI companies if voluntary commitments fall short.

Source: CBC News (Altman meeting + concessions) • CBC News (Solomon not satisfied) • CBC News (Gebala family lawsuit)

Takeaway: Canada extracted what no U.S. regulator, plaintiff firm, or newspaper has extracted. Embedded government-appointed reviewers inside OpenAI. Mandatory audit access. Direct national police reporting. The cost was eight bodies. Six of them children.

For Legislators: Solomon's four concessions translate directly into statutory language. A state or federal AI mental health bill can require in-company reviewer seats at defined model-risk tiers. Mandatory third-party audit access. Public reporting of high-risk-user detection systems. Mandatory direct reporting to state attorneys general above defined thresholds. Canada got all four without legislation. A statute can codify all four before anyone else has to count bodies.

Patient Safety and Ethics

Psychiatry's First Forensic Roundtable on AI Litigation Just Published. The DSM-IV Chair Called Litigation "Our Only Hope."

This morning, April 23, 2026, Psychiatric Times published a forensic roundtable.

Three authors on the byline.

Allen Frances. Chair of the DSM-IV task force. The most prominent living critic of psychiatric overdiagnosis.

Joe Pierre. Forensic and clinical psychiatrist at UCSF.

Joseph Simpson. Forensic psychiatrist.

The subject is AI chatbot litigation. Causation analysis. Expert witness strategy.

Frances opened the conversation with a comparison.

Big Tobacco. Big Pharma. Big AI.

His line on the record. "The legal battle against Big AI is our only hope that it will ever become a more responsible industry."

This is a man who for fifteen years has warned psychiatry against inventing new diagnoses to capture new harms. Who called for clinically informed stress testing of chatbots before release, not after.

When Frances says litigation is our only hope, the conversation about whether these cases are legitimate is over.

What Pierre put on the record

By end of 2025, at least ten known lawsuits against OpenAI and Character Technologies.

Wrongful death. Involuntary manslaughter. Sexual abuse. Negligence. Product liability.

Six adults. Four minors. Seven died by suicide.

The OpenAI cases allege the outcomes were both predictable and the result of "deliberate design choices."

In early 2026, the state of Kentucky sued Character Technology for "harmful, explicit, and psychologically manipulative interactions with minors."

Pierre names two landmark verdicts that suggest juries will reject Section 230 immunity for algorithmic chatbot output. The Meta and YouTube addictive-design verdict in Los Angeles. The New Mexico AG 375 million dollar Meta consumer protection verdict.

Design is product. Section 230 protects content.

The jury framing is moving from content to product.

The defense theory already on the record

Simpson names it. Causation is the pressure point.

The defense will argue maladaptive chatbot use was a symptom of pre-existing mental illness. Not the cause of a new one.

The parallel is cannabis and psychosis. Decades of forensic literature. No clean resolution.

Juries will hear that fight in open court. Expert psychiatrists on both sides.

The next wave Simpson names

Chatbot-as-therapist malpractice.

Against products explicitly marketed as therapy.

And against products users turned to for therapy anyway.

That second category is the larger docket.

Source: Psychiatric Times, April 23, 2026

Takeaway: When the man who wrote the DSM-IV says litigation is "our only hope," the question is not whether courts should take these cases seriously. The question is whether AI labs can meet the clinical standard before the standard is set for them one verdict at a time. The CAW thesis has been that architecture comes first and diagnosis comes second. Frances just arrived at the same conclusion from the other direction.

For Legislators: A forensic consensus is forming in court. Any state statute on AI mental health needs to survive admissibility challenges in the harm cases that will follow. Three elements carry the day. An in-company audit right like the Solomon deal. A professional-representation bar like the Tennessee law. A defined safe-harbor architecture like the Utah framework. Bills missing any of the three will collapse at the first motion to dismiss.

Conversational AI Technology

OpenAI Shipped GPT-5.5 Today. The Safety Number for a User in Crisis Went Down.

At roughly the hour Uthmeier was sorting subpoena responses and the forensic psychiatrists were going to press, OpenAI released GPT-5.5.

The system card landed April 23, 2026. The company calls the release its "strongest set of safeguards to date."

The scores tell a different story. But first, the measuring stick.

What HealthBench is

HealthBench is the benchmark OpenAI built to grade how well its own models handle health questions.

The company launched it in May 2025. It was designed with 262 physicians across 60 countries. Five thousand realistic multi-turn conversations between a model and a user. Each conversation paired with a physician-written rubric that defines what a safe, helpful, accurate answer looks like.

A model's response is graded against those rubrics. Higher is better.

The point of HealthBench is that health questions are not multiple choice. Safety in a health conversation is context-specific. Someone with chest pain asking the model for help needs a different answer than a clinician asking the model about chest pain. The rubric approach captures that.

OpenAI reports four HealthBench scores with every model release. All four matter. None of them is the same thing.

HealthBench. The overall score across all five thousand conversations. A headline number.

HealthBench Hard. The subset of conversations physicians rated hardest to answer safely. Where models have historically failed most often.

HealthBench Professional. The subset of conversations that simulate a practicing clinician using the model as a reference tool. Does the model give a practicing doctor a useful answer.

HealthBench Consensus. The subset where multiple physicians agreed unanimously on what a safe answer must include. Emergency referral when appropriate. Refusing to answer beyond scope. Recognizing distress and routing to real help.

Consensus is the floor. It is the benchmark for the basic safe thing. The thing the model must do when a vulnerable user shows up.

Professional is the ceiling. It is the benchmark for usefulness to the paying professional.

The four scores for GPT-5.5

HealthBench. 56.5. Up 2.5 points from GPT-5.4.

HealthBench Hard. 31.5. Up 2.4.

HealthBench Professional. 51.8. Up 3.7.

HealthBench Consensus. 95.6. Down 0.7.

Three went up. Professional went up the most.

The floor went down.

What that pattern means

Professional is the enterprise customer. Consensus is the vulnerable user.

Professional went up 3.7 points. Consensus went down 0.7.

The model got better at helping the doctor.

The model got no better at protecting the user in crisis. It got slightly worse.

A 0.7 point drop is small. It is within run-to-run noise.

What matters is the direction. The lever that governs crisis-safety did not move up. It moved down.

That is a choice about where the safety work is being pointed.

What OpenAI's own system card says about mental health

In the dynamic multi-turn evaluations for mental health, emotional reliance, and self-harm, OpenAI reports GPT-5.5 "performs on par with GPT-5.4-Thinking."

On par. Not better.

For dynamic benchmarks with adversarial user simulations, the metric is called not_unsafe. The percentage of assistant messages that do not violate safety policies.

The company also reports the model "shows a mix of higher and lower rates of misalignment" than its predecessor.

The timing

GPT-5.5 released the same day Psychiatric Times published the Frances roundtable.

Two days after the Florida criminal investigation.

Seven weeks after Solomon sat down with Altman and walked out with four safety concessions.

The company is under active state criminal investigation for how its chatbot handles users in crisis.

It shipped a new flagship model. The crisis-floor benchmark did not improve.

Source: OpenAI GPT-5.5 System Card • OpenAI launch post • HealthBench paper (Arora et al.) • HealthBench launch post

Takeaway: Professional up 3.7. Consensus down 0.7. The model got better at helping the doctor and no better at protecting the user in crisis. That is an architecture choice, not a technical constraint. The model follows the money.

For Legislators: HealthBench is the benchmark an AI lab uses to grade its own models on safety and usefulness in health conversations. It is open source. Four sub-scores. Consensus is the one that measures whether the model does the basic safe thing when a user is in crisis. Any frontier AI model deployed into mental health contexts in the United States should be required to publish its Consensus score at every release. Not just its capability scores. The disclosure is not a technical burden. It is a transparency requirement a state can impose as a precondition to any mental health safe harbor.

Mental Health Policy

Florida Calls the Legislature Back Tuesday. The Bill Bans AI From Pretending to Be a Licensed Therapist. The Uthmeier Case Is Why It Will Pass This Time.

Tuesday, April 28. Florida legislature reconvenes.

Special session. Four days. April 28 through May 1.

On the agenda is the AI Bill of Rights.

Same bill that passed the Senate 35 to 2 in the regular session.

Same bill House Speaker Danny Perez killed without a floor reading.

Same bill. Different world.

What the bill does

Notice to consumers when interacting with any AI.

Prohibition on any entity providing "licensed" therapy or mental health counseling through AI.

Parental controls for minors. Parents can view the conversations. Parents can set access parameters. Parents get notified when the child exhibits concerning behavior.

A 45-day cure period for AI companies to fix violations.

50,000 dollar fines for egregious violations.

An additional 10,000 dollars per-minor penalty when a platform recklessly permits a minor without parental consent.

Prohibition on government use of DeepSeek or other Chinese-created AI tools.

Elementary schools barred from providing AI access except for supervised translation, disability accommodation, or instructional tools.

Who blocked it last time

Speaker Perez blocked the bill on federal-preemption grounds. A Trump executive order signed in December limits state AI legislation.

The Trump administration reportedly contacted Perez directly to discourage the bill.

Byron Donalds, the Republican favored to succeed DeSantis, received a 5 million dollar donation from a pro-AI super PAC last month.

The regular session ended. The bill died in the House without a reading.

What changed between then and now

Florida AG Uthmeier opened the criminal investigation on April 21.

DeSantis has been pushing House leadership since then. Citing the FSU shooting by name.

His line on the bill. "We see ChatGPT, what they did with this FSU shooting. You've seen what they've done with these teenagers, these chatbots. This is totally out of control."

Senate President Ben Albritton confirmed yesterday the special session version will be identical to the regular session version.

Megan Garcia, whose son Sewell Setzer III died by suicide after Character.AI interactions, testified for the bill in the regular session. Her suit sparked it.

Common Sense Media AI programs director Robbie Torney testified on risks to teens. Stanford Brainstorm Lab child psychiatrist Darja Djordjevic called for FDA-style regulation of chatbot mental health use.

The House voted the bill down once in a hypothetical.

Tuesday they vote on it in a world where a sitting state attorney general has said a bot, if a person, would be charged with murder in their state.

Source: Florida Phoenix / WLRN • WFLA • Governor DeSantis AI Bill of Rights announcement

Takeaway: The House blocked the bill once on federal-preemption grounds. It did not block the facts that came out after. Uthmeier put one specific shooting and one specific company under criminal subpoena. A no vote in March meant siding with industry on a hypothetical. A no vote next Tuesday means siding with industry on the day after the criminal investigation opened. That changes what a no vote costs.

For Legislators: Florida's AI Bill of Rights is the first bill in the country that both prohibits AI from providing licensed therapy and creates a parental-controls regime tied to concerning-behavior alerts. If it passes, it is the template for states with split-chamber politics where federal preemption concerns have been blocking similar bills. Three structural elements. Prohibit professional misrepresentation. Build a parental-controls regime with concrete dollar penalties. Build a cure period courts will respect. Pick the vehicle that fits your legislative culture. The architecture is portable.

What We Built

Casey: Voice-First AI-Native Mental Health EHR

Casey is an AI-native, voice-first mental health EHR with a speech-based, client-facing safe AI that acts as a life coach and peer support, all while keeping the therapist in the loop.

The data layer features the first HIPAA-compliant Neo4j Memory Graph, which builds persistent therapeutic context across months of daily sessions. Pre-FDA safety validation complete: 1.78 million stress test executions at 100 percent accuracy.

Campus-first launch with founding North Carolina state licensee. 50-state PC licensee model. $2.5M seed raise in progress.

Watch the Casey Demo →