Two Theories of AI Safety, On Trial

2026-04-27

Conversational AI Watch

Issue #28 • April 27, 2026 • By Jess Jessop

AI safety, mental health policy, and patient safety at the intersection of conversational AI

▶ WATCH • 🎧 QUICK LISTEN • 🎧 DEEP DIVE • 📄 READ ON WEB

Jess's Take

Two Theories of AI Safety, On Trial

The Sunday issue closed with three people who put the harm into the formal record. Today, Monday, the formal record absorbed three more entries.

In Oakland, federal Judge Yvonne Gonzalez Rogers gaveled in jury selection in Musk v. Altman, case 4:24-cv-04722. The case is not about mental health on its face. It is about whether OpenAI's leaders breached a charitable mission and induced Elon Musk to fund a nonprofit that became a $134 billion for-profit. The fact that has the most reach for our work is this. Inside that trial is the same question CAW has been tracking for twenty-seven issues. Who is responsible when a frontier AI system harms a human being.

In Springfield, Illinois, the same two AI companies that share the front page of every wrongful-death docket are now backing opposite bills. OpenAI is backing SB 3444. Under SB 3444 a frontier developer is not liable for the death or serious injury of one hundred or more people, or one billion dollars in property damage, unless the harm was intentional or reckless. Anthropic is backing SB 3261. SB 3261 requires a public safety and child protection plan, and creates liability if an AI model causes a child severe emotional distress, death, or bodily injury, including self-harm.

Three weeks earlier, Anthropic published Project Glasswing. The vehicle is Claude Mythos Preview, a new frontier model. Mythos Preview, given six weeks, identified thousands of zero-day vulnerabilities in every major operating system and every major web browser. It autonomously found and exploited a seventeen-year-old root vulnerability in FreeBSD. Anthropic announced it will not make Mythos Preview generally available. The reason on the record is the same reason in their Illinois bill. Misuse risk is too high.

On Friday, the Utah Medical Licensing Board sent a letter to the Utah Department of Commerce. Subject: the Doctronic AI prescription pilot, the one Issue #15 covered. The Board's request is one word. Suspend. The Board says it learned about the agreement only after the system went live. The Board says proceeding without consulting the Medical Board places Utah citizens at risk.

And in the Journal of Medical Internet Research, Buck and Maheux published a cross-sectional survey of 1,003 U.S. young adults split by Prodromal Questionnaire score. The elevated-risk group reported intensive AI chatbot use at odds ratios of 1.7 to 2.6, used chatbots for emotional support at higher rates, and reported delusion-related interactions ranging from 13.3 to 30.7 percent. The dose-response is now in the indexed peer-reviewed literature.

This is what two theories of AI safety look like when they meet on the same Monday.

Theory 1 is the liability shield. Move fast. Sandbox permissive. The 100-person threshold. Mass casualty as the only enforceable trigger.

Theory 2 is the public safety plan. Withhold the frontier model when misuse risk is concrete. Hold the company liable for harm to children. Treat the dose-response as evidence.

The federal court is hearing them. The state legislatures are voting on them. The medical boards are flagging them. The peer-reviewed journals are quantifying them. Next week there will be more entries on both sides.

Because at the end of the day, we are all on the same side, AI assisted but the human side.

AI Safety and Regulation

Musk v. Altman Trial Begins in Oakland Federal Court. The First Federal Jury Trial of an AI Company's Founding Safety Mission.

This morning, jury selection began in Musk v. Altman, case 4:24-cv-04722, U.S. District Court for the Northern District of California, Oakland Division. Judge Yvonne Gonzalez Rogers presiding.

Plaintiffs: Elon Musk, xAI Corp., and the Musk Foundation. Defendants: OpenAI Inc., OpenAI LP, Sam Altman, Greg Brockman, Microsoft Corp.

The complaint, filed in California state court in February 2024 and refiled in federal court on August 5, 2024, alleges that Altman and Brockman induced Musk to fund OpenAI's launch in 2015 on the representation that OpenAI would remain a charitable nonprofit dedicated to safe AGI for humanity. Musk contributed $44 million between 2016 and 2020. The complaint alleges that in 2019 OpenAI created the for-profit OpenAI LP subsidiary, that in 2024 OpenAI began converting the parent into a for-profit, and that Microsoft's $13 billion investment created a structure inconsistent with the founding mission.

On April 25, 2026, Musk's team withdrew the fraud claims, narrowing the trial to breach of contract, breach of fiduciary duty, unfair competition, and unjust enrichment. The narrow trial begins today. The case is being tried to a jury. Jurors will decide whether OpenAI's leaders breached the founding agreement when they restructured the nonprofit into a capped-profit entity now valued by reporting at $134 billion.

The trial coincides, on the same federal docket, with the consolidated mental health and product liability suits CAW #4 covered. Those suits are pending and not on the trial calendar this week.

The line that matters for our beat. This trial puts on the record, under oath, the question every lawsuit and statute and FDA referral has been circling. What does an AI company owe the public when it ships a product that can foreseeably harm people. Musk's complaint frames it as a charitable-mission breach. The Senate Judiciary, the AMA, the Stanford paper, the Florida criminal subpoena, the Garcia complaint, and now Illinois SB 3261 frame it as a duty of care. They are the same question.

Sources: NPR • Christian Science Monitor • CNBC • Fortune • N.D. Cal. docket

Takeaway: Musk v. Altman is the first federal jury trial in U.S. history to ask whether an AI company breached its founding safety mission. Whatever the verdict, the jury instructions, the trial record, and the appellate posture will become reference material for every duty-of-care theory in the pending mental-health docket.

For Legislators: Watch the jury instructions. If the court instructs the jury that a charitable mission creates an enforceable duty independent of statutory fiduciary law, that doctrine will reach state liability bills like Illinois SB 3261 within months.

AI Safety and Regulation + Mental Health Policy

Illinois SB 3444 vs SB 3261. OpenAI and Anthropic Now Back Opposite Bills.

The Illinois General Assembly is hearing two AI safety bills this spring. They are sponsored by different members. They are backed by different AI companies. They reach opposite conclusions.

SB 3444 (OpenAI-backed). Defines a frontier model as any system trained on more than $100 million in compute. The threshold sweeps in OpenAI, Google, Anthropic, xAI, and Meta. Requires the developer to publish an AI safety plan. Defines critical harms as the death or serious injury of 100 or more people, $1 billion in property damage, or use by a bad actor to develop a chemical, biological, radiological, or nuclear weapon. Provides liability protection unless the developer intentionally or recklessly caused the incident. No private right of action. No enforcement mechanism behind the safety-plan requirement.

SB 3261 (Anthropic-backed). Requires AI developers to publish, on their public website, a safety plan and a separate child protection plan. Creates a private right of action when an AI model causes a child severe emotional distress, death, or bodily injury, including self-harm. Defines a child as anyone under eighteen. Definitions of severe emotional distress, model, and developer tie to existing Illinois consumer protection law.

Anthropic on SB 3444: "Good transparency legislation needs to ensure public safety and accountability for the companies developing this powerful technology, not provide a get-out-of-jail-free card against all liability."

OpenAI on SB 3261: no public statement as of this morning.

This is the first time the OpenAI-Anthropic posture difference has surfaced as competing state bills. It will not be the last. Texas, New York, and Washington State are reportedly considering similar frameworks for the 2027 session.

Sources: Fortune • Quartz • WTL Governance • Gizmodo

Takeaway: Two of the largest AI labs in the United States are now publicly on opposite sides of a state legislative fight. SB 3444 is the liability-shield model. SB 3261 is the child-protection-plus-private-right-of-action model. The Illinois floor vote will set the template that other state legislatures copy.

For Legislators: SB 3261 is the model bill. The child protection plan pattern is portable to your state's consumer protection statute without requiring federal preemption to clear. The severe emotional distress definition aligns with Restatement Second of Torts §46 and is recognizable to state appellate courts.

Patient Safety and Ethics

Utah Medical Licensing Board Calls for Immediate Suspension of Doctronic AI Prescription Pilot. The First Time a State Medical Board Has Pushed Back on a State-Sanctioned AI Prescriber.

On Friday, April 24, 2026, the Utah Medical Licensing Board, the body within the Utah Department of Commerce that licenses physicians and surgeons, sent a letter to the Department of Commerce and the Utah Office of Artificial Intelligence Policy. The letter asks for the immediate suspension of the Doctronic AI prescription pilot.

Background. CAW #15 covered the launch of the Doctronic pilot in late 2025 and the parallel authorization of Legion Health AI in April 2026. The Doctronic chatbot is empowered, within Utah's regulatory sandbox under SB 0108, to conduct a clinical evaluation and autonomously renew prescriptions for nearly 200 drugs. Controlled substances are excluded. Physicians provide oversight after the fact.

The Board cites three concerns. First, the agreement was concluded by the Utah Office of Artificial Intelligence Policy without consulting the Medical Board, which is the licensing authority for prescribing in the state. Second, in late March 2026, security researchers at Mindgard published findings demonstrating they had jailbroken Doctronic's public-facing AI health assistant using prompt injection. The researchers reported being able to triple an OxyContin dosage recommendation, mislabel methamphetamine, and generate false vaccine information. CAW #15 covered the Mindgard findings. Third, the Board states that proceeding with the agreement without consulting the Medical Board potentially places Utah citizens at risk and remains a major concern of the board.

The Board's request: immediate suspension of the Doctronic pilot pending full Medical Board review. The letter does not directly reference Legion Health AI, the second AI prescriber authorized under the same sandbox, but the language is broad enough to apply.

The Board does not have unilateral authority to suspend the pilot. The Office of Artificial Intelligence Policy and the Department of Commerce can either respond to the Board's request or decline to. The Department of Commerce had not publicly responded as of this morning.

Why this matters. This is the first time a state medical licensing board has formally pushed back on a state-sanctioned AI prescribing pilot. The Board's posture mirrors the AMA's letter to Congress eight days ago, covered in CAW #26. The state-level professional licensure system is the nineteenth-century scaffolding through which every prescribing AI has to pass. Utah just demonstrated what happens when that scaffolding is bypassed.

Sources: STAT News • STAT, $15 AI test • KUTV • Startup Fortune

Takeaway: A state medical licensing board has invoked its patient-safety mandate to ask a state AI sandbox to stop. This is the first time. It will not be the last.

For Legislators: Any state AI sandbox bill should require sign-off from the relevant professional licensing board before authorization, not after. Utah's SB 0108 lacked that requirement. Build it into yours.

Conversational AI Technology + AI Safety and Regulation

Anthropic's Project Glasswing. A Frontier Model Anthropic Will Not Release.

On April 7, 2026, Anthropic published Project Glasswing. The vehicle is Claude Mythos Preview, a new frontier general-purpose language model. Mythos Preview is, on Anthropic's published benchmarks, dramatically more capable at computer security tasks than any previously released model.

Capabilities on the record. Mythos Preview autonomously identified and exploited a seventeen-year-old remote code execution vulnerability in FreeBSD that allowed root access to any machine running NFS. Over six weeks of internal testing, Mythos Preview identified thousands of zero-day vulnerabilities in every major operating system and every major web browser, plus a range of additional widely deployed software.

The Glasswing partnership. Anthropic, AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks. Anthropic committed $100 million in model usage credits to Glasswing partners during the research preview.

Anthropic announced it will not make Claude Mythos Preview generally available. The stated reason is misuse risk. Bruce Schneier called the decision "necessary" in his column the same week. Foreign Policy framed the decision as the first time a frontier lab has publicly held a release back on misuse-risk grounds without external pressure.

The relevance to the CAW beat. Anthropic's Project Glasswing posture is the same posture Anthropic is taking in Illinois SB 3261. Withhold the model when misuse risk is concrete. Hold the company accountable when harm reaches children. Publish a safety plan in advance. The pattern is consistent across the lab's regulatory submissions, its product release decisions, and its industry partnerships.

The contrast to OpenAI's posture is also consistent. OpenAI's GPT-5.5 release on April 23, covered in CAW #25, lowered the HealthBench Consensus crisis-safety floor by 0.7 points. OpenAI's Illinois bill would protect a frontier developer from liability for one hundred non-intentional deaths. Two release patterns. Two regulatory postures. Same week.

Sources: Anthropic Project Glasswing • Anthropic Mythos Preview • Fortune • Schneier • Foreign Policy

Takeaway: A major AI lab has, for the first time, declined to publicly release a frontier model on misuse-risk grounds. The product decision is consistent with the lab's posture in state legislation. The two postures together constitute a falsifiable Theory 2 of AI safety.

For Legislators: Project Glasswing demonstrates that a withhold-the-model-when-misuse-risk-is-concrete rule is operationally feasible. Anthropic's $100M commitment indicates the financial elasticity is there. State AI sandbox bills can require pre-deployment misuse risk assessment without preventing innovation.

Conversational AI Technology + Mental Health Policy + Patient Safety and Ethics

JMIR Publishes the First Population-Level Dose-Response. Buck and Maheux on AI Chatbot Use, Human-Role Attribution, and Psychosis Risk in 1,003 Young Adults.

On March 5, 2026, the Journal of Medical Internet Research published a cross-sectional survey study of 1,003 U.S. young adults. Authors: Benjamin Buck and Anne J. Maheux. Open access at doi 10.2196/85038.

Method. The 1,003 respondents were stratified by score on the Prodromal Questionnaire, Brief Version, a validated self-report screen for psychosis risk used in research and clinical settings. Distress Score of 20 or higher classified the respondent as elevated risk. Sample split: N = 267 elevated risk (28 percent); N = 685 low risk (72 percent); 51 unclassified. Respondents reported their generative AI chatbot usage frequency, motivations, and the content of their conversations.

Findings on usage. Elevated-risk respondents were no more likely to have ever used GenAI, but were significantly more likely to report intensive use. Odds ratio 1.70 to 2.56 across three measures: several uses per day, more than thirty minutes per day, six or more chatbot conversations per day.

Findings on motivations. Elevated-risk respondents were significantly more likely to use GenAI for social and emotional support, and were significantly more likely to ascribe human roles to chatbot interactions. Odds ratios 1.76 to 3.08 across the four roles surveyed: companion, friend, therapist, romantic partner.

Findings on delusion-related interactions. Among elevated-risk respondents, item endorsements for delusion-related interactions ranged from 13.3 percent to 30.7 percent. The paper does not establish causation. The cross-sectional design cannot. What it establishes is a dose-response association between intensive chatbot use, attribution of human roles, and self-reported delusion-like experiences in the population at elevated psychosis risk.

The clinical and regulatory significance. This is the first peer-reviewed quantification of the dose-response that Stanford's Characterizing Delusional Spirals paper described qualitatively in CAW #26. Stanford gave the mechanism. Buck and Maheux gave the prevalence and the dose. The two papers together make a citable evidence base for the FDA referral the AMA letter requested.

Sources: JMIR full paper • PubMed PMID 41785452 • Full PDF

Takeaway: The peer-reviewed literature now has a population-level dose-response: intensive chatbot use plus human-role attribution plus elevated psychosis risk produces 13 to 31 percent self-reported delusional interactions. The number that started as anecdote in a TIME magazine essay last June is now an indexed statistic.

For Legislators: Cite this paper directly when arguing for an FDA-referral threshold or a dose-control safeguard. The doi is 10.2196/85038. The PQ-B is a standard instrument; any state AI bill that mandates pre-deployment risk testing can cite it as the validated screen.

What We Built

Casey: Voice-First AI-Native Mental Health EHR

Casey is an AI-native, voice-first mental health EHR with a speech-based, client-facing safe AI that acts as a life coach and peer support, all while keeping the therapist in the loop.

The data layer features the first HIPAA-compliant Neo4j Memory Graph, which builds persistent therapeutic context across months of daily sessions. Pre-FDA safety validation complete: 1.78 million stress test executions at 100 percent accuracy.

Campus-first launch with founding North Carolina state licensee. 50-state PC licensee model. $2.5M seed raise in progress.

Watch the Casey Demo →