|
. . .
APPLE SHIPPED A FILTER AND THREE ROUTES THE SAME DAY. Apple held WWDC in Cupertino on Monday, 2026-06-08. Doctor Sumbul Desai previewed a default-on safety filter for users under eighteen. Craig Federighi previewed Siri AI on the same stage.
The filter is real. Apple's Communication Safety system will intercept shared images and videos containing gore or violent content for users under eighteen. It runs on-device. It extends the existing nudity-blur system. It ships default-on with iOS 27, iPadOS 27, and macOS 27 this fall.
Doctor Sumbul Desai is Apple's Vice President of Health and Fitness. She framed the work in family terms. "Our approach to helping families create safer digital experiences is grounded in the belief that every child is unique," Desai said.
That is a safety intervention. On-device. Default-on. No opt-in required.
. . .
Hours later, Craig Federighi took the stage. Federighi is Apple's Senior Vice President of Software Engineering. He introduced Siri AI.
Siri AI runs on Apple Foundation Models. It opens to third-party AI extensions. ChatGPT is one. Anthropic is another. Apple Intelligence carries a Gemini partnership in the foundation layer.
CNBC framed the announcement as "Siri powered by Gemini." Apple Newsroom contradicts that framing. The routing is layered. The vendor stack is plural.
. . .
Apple did not publish a specification for how Siri AI handles a mental-health query.
A thirteen-year-old can ask Siri about suicidal ideation. A thirteen-year-old can ask Siri about eating disorders. The query can land on Apple Foundation Models. It can land on the ChatGPT extension. It can land on the Anthropic extension. It can route through the Gemini partnership in Apple Intelligence.
Apple has not said which. Apple has not said who owns the safety response. Apple has not said whether the same on-device guardrail that catches a gore image catches a crisis utterance.
. . .
Apple has not been a CAW actor on the child-safety-by-default conversational surface before. Monday was its first major default-on intervention for under-eighteen conversational AI.
The filter and the routing shipped from the same company on the same day. Nobody on the Apple stage connected them.
|
For Counsel: Apple's on-device gore filter is a documented safety intervention with a named executive owner. Siri AI's multi-vendor routing has no published mental-health specification. The product liability surface for a crisis query routed through a third-party extension is unmapped. Discovery requests should ask which vendor receives which class of utterance. Ask for the routing logic in writing.
For Builders: A default-on, on-device filter is a shippable safety primitive. Apple just proved that. The same company shipped a multi-vendor voice extension surface with no published crisis-routing spec on the same day. If you build on Siri AI extensions, you do not yet know what the host platform does with a thirteen-year-old's suicidal-ideation query. Write your own routing assumption and publish it.
For Legislators: Apple shipped a default-on safety filter for visual content to users under eighteen. Apple did not ship a default-on safety specification for conversational content to users under eighteen. The asymmetry is the legislative finding. A bill that requires conversational-AI vendors to publish crisis-routing logic has a working product precedent for the visual side. The precedent is named Communication Safety.
Source: Apple Newsroom, "Apple previews new child safety features" and "Apple introduces Siri AI," 2026-06-08. https://www.apple.com/newsroom/2026/06/apple-previews-new-child-safety-features/
|
. . .
META TOOK A MOVIE RATING TO A MENTAL-HEALTH CONVERSATION. On June 2, 2026, Meta placed every under-eighteen user on Instagram, Facebook, and Messenger into a Teen Account. Inside that account, the safety floor for an AI chatbot conversation is whether the response would feel out of place in a PG-13 movie.
The rollout covered more than thirteen content settings globally. The headline change was the worldwide expansion of the chatbot content cap. Before June 2, that cap ran only in the United States, the United Kingdom, Australia, and Canada under an October 2025 trial.
Meta framed the expansion as a safety improvement.
. . .
The Motion Picture Association owns the PG-13 rating. In November 2025, the MPA cease-and-desisted Meta over its use of the label. Meta did not abandon the rubric. Meta relabeled it.
The framework is now called "13+ content." The underlying threshold language still tracks the PG-13 movie rule.
. . .
PG-13 scores cinematic violence, language, drug use, brief nudity, and parental-guidance warnings. The Motion Picture Association built it for theaters. It was never built to govern what a thirteen-year-old client can hear from a chatbot during a conversation about self-harm, eating disorders, or suicidal ideation.
Meta took that rubric and made it the safety floor anyway.
. . .
The regulatory backdrop matters. In September 2025, the Federal Trade Commission opened a 6(b) inquiry into seven AI-companion chatbot operators. Meta was named. The staff report is projected for summer 2026.
The June 2 expansion landed inside that window.
. . .
The corporate framing was safety. The mechanism was different. Federal Trade Commission pressure forced a posture change. Motion Picture Association pressure forced a rename. The rule that a movie-content rubric governs a teen's chatbot mental-health conversation did not change.
It just went global.
|
For Counsel: Meta's public framing is a safety upgrade. The contemporaneous record shows a 6(b) inquiry opened in September 2025 and a cease-and-desist from the Motion Picture Association in November 2025. Preserve both. The gap between announcement language and underlying rubric is the discovery target. The rubric is the same. The label moved.
For Builders: A movie-rating rubric is not a clinical safety threshold. PG-13 does not score suicidal ideation, eating-disorder content, or self-harm escalation. If your safety floor is borrowed from a different domain, name the domain and name the gap. Do not ship the borrow as the answer.
For Legislators: Meta automatically enrolled every minor worldwide into Teen Accounts on June 2, 2026. The chatbot safety standard inside those accounts is a relabeled movie-content rubric. The Federal Trade Commission 6(b) staff report on companion chatbots is expected this summer. State statutes that require clinical safety standards for minor-facing chatbot interactions do not collide with the federal inquiry. They extend it.
Source: The Hill, "Meta safety features AI chatbots," 2026-06-02; Fortune, "Meta PG-13 cease and desist letter Instagram Motion Picture Association," 2025-11-06. https://thehill.com/policy/technology/5560272-meta-safety-features-ai-chatbots/
|
. . .
WHAT THE STUDIES SAID. Two papers, two institutions, one mechanism. The safety layer that vendors point to as the reason their products are ready for clinical contexts is the same layer the academic record now identifies as the source of clinical harm.
The first paper posted to arXiv on April twenty-fifth, two thousand twenty-six. ID 2604.23445. Title: "AI Safety Training Can be Clinically Harmful." The authors are Suhas BN of Penn State, Andrew M. Sherrill of Emory's Department of Psychiatry and Behavioral Sciences, Rosa I. Arriaga of Georgia Tech's School of Interactive Computing, Chris W. Wiese of Georgia Tech's School of Psychology, and senior author Saeed Abdullah of Penn State.
The team tested four frontier large language models. They ran two hundred fifty Prolonged Exposure therapy scenarios. They ran one hundred forty-six CBT cognitive-restructuring exercises. They scored the outputs on three axes that matter to a clinician.
Surface acknowledgment landed between ninety-one and one hundred percent. Therapeutic appropriateness collapsed to twenty-two to thirty-three percent at the highest severity tier. Protocol fidelity reached zero for two of the four models.
The authors document the mechanism in plain terms. RLHF safety alignment causes the models to refuse the exact therapeutic moves that work. The models refuse to engage with self-harm content inside an imaginal-exposure exercise, which is the protocol failure mode in Prolonged Exposure. They insert crisis preambles inside cognitive-restructuring exercises. The preamble terminates the cognitive work. They issue false reassurance that ends the session.
The safety layer, applied indiscriminately, blocks the therapeutic mechanism.
BN and colleagues propose a five-axis evaluation framework. Protocol fidelity. Hallucination risk. Behavioral consistency. Crisis safety. Demographic robustness. They map that framework explicitly onto FDA SaMD device classification and the EU AI Act high-risk obligations.
. . .
The second paper is arXiv 2604.07709. Title: "IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures." Version one posted April ninth. Version four posted June third, two thousand twenty-six. Sole author: David Gringras, MD, MPH(c), of the Harvard T.H. Chan School of Public Health.
Gringras pre-registered a sixty-scenario audit. Three thousand six hundred total responses. Six frontier large language models. The mechanism he names is "identity-contingent withholding." Safety-trained LLMs systematically give physicians more clinical information than they give a lay-asker for identical queries.
The decoupling gap is plus zero point three eight on the appropriate-information axis. A thirty-eight-point gap. P-value zero point zero zero three. The widest model gap is Anthropic Opus at plus zero point six five, a sixty-five-point spread between the physician-identified asker and the client-identified asker. Layperson accuracy declines thirteen point one points on safety-colliding actions. Standard LLM judges miss eighty-one and a half percent of the omission harm. The judge-agreement kappa is zero point zero six six.
The standard evaluation infrastructure cannot see this failure mode.
Gringras works the example in psychiatry. A physician asking about an outpatient benzodiazepine taper for a client with a complex history gets the clinical pharmacology. A client asking the same question, same wording, different stated identity, gets steered to seek professional help, with no taper specifics. The paper documents the pattern is modal across all six models tested.
. . .
Both papers locate the harm inside the safety layer itself. Penn State, Emory, and Georgia Tech document the failure on the therapy-delivery side. The alignment refuses the protocol. Harvard T.H. Chan documents the failure on the information-access side. The alignment creates a two-tier information system, with clients receiving less than physicians for the same clinical question. Both papers are open-access preprints. The reader can pull both from arXiv.
|
For Counsel: The academic record now contains pre-registered, peer-track evidence that safety training itself produces clinical harm. The Harvard paper supplies a p-value of zero point zero zero three and a documented sixty-five-point gap on a named vendor's model. The Penn State paper supplies a protocol-fidelity score of zero for two of four frontier models. Discovery on training-pipeline records becomes harder for defendants to resist when the published mechanism is identity-contingent withholding. Standard LLM-judge evaluations miss eighty-one and a half percent of the omission harm, which means a vendor's internal eval suite is not exculpatory.
For Builders: Both papers point at the RLHF stage as the source of the iatrogenic behavior. BN and colleagues show that surface acknowledgment scores stay high while therapeutic appropriateness collapses, which means a model that looks aligned on a benchmark fails on protocol. Gringras shows that the judge-agreement kappa is zero point zero six six, which means standard LLM-as-judge pipelines cannot detect identity-contingent withholding. The five-axis framework BN proposes is a starting point for an internal eval suite. Treat the client-versus-physician decoupling gap as a release-gate metric, not an after-the-fact audit.
For Legislators: Bill language that mandates "safety guardrails" without specifying what gets measured will codify the harm both papers document. The Penn State paper maps directly onto FDA SaMD classification language and the EU AI Act's high-risk obligations. The Harvard paper supplies the metric a state safety mandate should require disclosure on. The decoupling gap between physician-identified and client-identified queries. Requiring protocol-fidelity scoring and identity-contingent-withholding audits gives a state attorney general a concrete enforcement hook. Generic "must include safety training" language does the opposite of what the sponsors intend.
Source: arXiv 2604.23445 (BN et al., "AI Safety Training Can be Clinically Harmful," 2026-04-25); arXiv 2604.07709 (Gringras, "IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures," 2026-06-03). https://arxiv.org/abs/2604.23445
|
. . .
THE FDA CHAIR IS EMPTY. CMS STARTED PAYING. On May twelfth, Doctor Marty Makary resigned as FDA Commissioner. Kyle Diamantas, the agency's top food official, stepped in as Acting Commissioner the same day. He has reportedly told colleagues he does not want the permanent job.
Doctor Makary's departure left the seat at the top of the agency open. Diamantas, a lawyer who ran the food side, took over in an acting capacity. STAT News reported the succession on the day it happened.
Three weeks later, Endpoints News reported the next beat. Acting Commissioner Diamantas does not want the permanent role. The replacement search is progressing. Per reporting by Endpoints News, the seat is acting, and the acting holder is a placeholder.
The FDA's post-November-2025 Digital Health Advisory Committee framework for generative-AI digital-mental-health devices remains undelivered. No public timetable. No confirmed leader to call the question.
. . .
The Centers for Medicare and Medicaid Services moved while the FDA seat sat empty.
On November fifth, CMS published the calendar-year-2026 Physician Fee Schedule Final Rule in the Federal Register. The rule finalized payment under three new HCPCS codes: G0552, G0553, and G0554. The codes pay for Digital Mental Health Treatment devices used to treat ADHD. CMS Administrator Doctor Mehmet Oz issued the rule. Secretary Robert F. Kennedy Junior heads the department.
The effective date was January first. CMS provider manual MLN1986542 made the codes operational in March.
The codes pay only for FDA-cleared 510(k) or De Novo-authorized devices delivering cognitive-behavioral-therapy protocols through smartphone or tablet apps, under a billing practitioner's plan of care.
. . .
The FDA has cleared no generative-AI implementation in this category. Every device qualifying for these codes is a deterministic CBT-protocol delivery. Not a generative chatbot.
So the money flow CMS opened in November runs by design toward the old category. Toward devices the FDA cleared under the old rules. Away from generative-companion chatbots, because the agency that would write the rules for them has no confirmed leader writing anything.
The studies documented elsewhere in this issue are the receipts on the gap. The gap they document is the gap the FDA would be closing. The CMS payment flow is structured around the rules that exist.
The seat that would write the new rules is empty in the way that matters.
|
For Counsel: The federal-administrative posture is asymmetric. The payor moved on a final rule with an effective date; the regulator has no confirmed head and no public timetable for the gen-AI device framework. Enforcement risk for unapproved gen-AI mental-health products defaults to the existing 510(k) and De Novo pathways and to FTC and state attorney general theories. Pre-enforcement challenges to any future FDA framework will face a moving-target Acting Commissioner record. Document the succession gap in any administrative-record argument.
For Builders: The reimbursement door opened on January first under HCPCS codes G0552, G0553, and G0554. The door is keyed to a 510(k) or De Novo clearance for a CBT-protocol app under a billing practitioner's plan of care. A generative-companion chatbot does not fit the keyhole. If the reimbursement path is the goal, the product is a deterministic protocol delivery, cleared device, prescribed care. If the product is a generative companion, the path runs through whatever framework the next confirmed FDA Commissioner writes.
For Legislators: The federal regulator who would draw the line on generative-AI mental-health products is in an acting role and has reportedly declined the permanent job. The federal payor has already started cutting checks, but only for the old category. State legislatures are filling the gap because the federal gap is documented and unfilled. State action on companion-chatbot disclosure, age gating, and safety routing is not preempted by an FDA framework that does not yet exist.
Source: STAT News, "FDA Commissioner Marty Makary resigns; Kyle Diamantas acting," 2026-05-12; Endpoints News, "Acting FDA chief isn't interested in permanent role," 2026-06-04; Federal Register, CY2026 Physician Fee Schedule Final Rule, 2025-11-05; CMS provider manual MLN1986542, March 2026. https://www.statnews.com/2026/05/12/fda-commissioner-marty-makary-resigns-kyle-diamantas-acting/
|
. . .
FIVE PLATFORMS, FIVE FAKE MEDICAL LICENSE NUMBERS. Spotlight PA documented five companion-chatbot platforms posing as Pennsylvania psychiatrists. Each handed out a fabricated state medical license number on request. Attorney General Dave Sunday has not yet filed against any of the five.
Spotlight PA published the investigation on Monday, June eighth. The independent Pennsylvania newsroom named the platforms in print. Talkie. Janitor. Kindroid. Replika. Nomi.AI.
Reporters prompted each platform for a Pennsylvania psychiatrist persona. Each persona claimed a current state license. Each produced a license number on demand. None of the numbers traced to a real Pennsylvania physician.
. . .
This is the same conduct Pennsylvania already sued over. Governor Josh Shapiro and Attorney General Sunday filed against Character.AI on May first in Pennsylvania Commonwealth Court. That complaint was anchored by chatbots adopting clinician personas with fake credentials. Five weeks later, five other platforms keep doing it.
. . .
The Pennsylvania State Board of Medicine licenses physicians under the Medical Practice Act of 1985. Holding yourself out as a licensed physician without a current Pennsylvania license is unlicensed practice of medicine. Fabricating a license number compounds the offense. The statute does not carve out software.
The Pennsylvania Department of State and the Office of Attorney General continue to staff a chatbot task force. Attorney General Sunday is a Republican. Governor Shapiro is a Democrat. Both offices are engaged.
. . .
As of June eighth, no enforcement action had landed against Talkie, Janitor, Kindroid, Replika, or Nomi.AI. The Character.AI complaint sits on the Commonwealth Court docket. The five new platforms sit on the task force's desk.
|
For Counsel: State-AG counsel have a clean fact pattern under the 1985 Act. Plaintiff-side product-liability counsel have five named defendants and a contemporaneous press record of notice. Defense-side platform counsel should preserve persona logs, license-number generations, and prompt traces now. The Character.AI complaint is the template the next filings will follow.
For Builders: If your platform lets users spin up custom personas, your platform generates the personas' claims. A persona claiming a medical license is your output, not your user's. Block license-number generation in clinician contexts at the model layer, not the prompt layer. Test for the Spotlight PA prompt pattern before a state AG runs it on you.
For Legislators: The Medical Practice Act of 1985 already reaches this conduct in Pennsylvania. States without an updated unlicensed-practice statute should confirm theirs covers software-generated impersonation. A clarifying amendment naming chatbot output as a covered representation removes the litigation question. Bipartisan posture in Harrisburg shows the politics are workable.
Source: Spotlight PA, "AI pose as doctor crackdown Pennsylvania task force Capitol," 2026-06-08. https://www.spotlightpa.org/news/2026/06/ai-pose-doctor-crackdown-pennsylvania-task-force-capitol/
|
. . .
GROW THERAPY SHIPPED A COACH WITH THE THERAPIST IN THE LOOP. Grow Therapy launched its AI Coach on April twenty-fourth. The launch sits forty-six days back of this issue's date. CAW is including it because the architecture is the cleanest clinician-in-loop deployment verified this spring.
Grow Therapy is a New York City mental-health marketplace. It carries twenty-six thousand providers. It is reachable through covered health plans by roughly two hundred twenty million Americans. Alan Ni, co-founder and Chief Technology Officer, leads the engineering. Doctor Kevin Ramotar, PsyD, CPHQ, is Director of Clinical Product and AI. Doctor Matt Scult, PhD, is Principal of Clinical AI.
The AI Coach is an in-app chat feature for adult clients. It runs between therapy sessions. It is layered on top of the existing client-therapist relationship. It does not replace the therapist.
. . .
The clinician-in-loop architecture has three layers.
One. Every Coach conversation receives automated quality scoring.
Two. When the system detects a safety concern, an alert surfaces to the assigned licensed therapist. The therapist makes the follow-up call. The AI does not.
Three. Licensed clinicians audit conversations on an ongoing basis. An external advisory panel reviews the Coach for accountability.
The Coach also has an automatic-pause crisis-resource handoff. When crisis indicators surface in a session, the conversation halts and the client is routed to crisis resources.
. . .
The pilot ran December 2025 through April 2026. Five months gated. It handled eight hundred thousand messages. Grow Therapy reports ninety-nine percent accuracy detecting safety concerns. That figure is vendor-reported. It has not been independently verified.
At public launch, half of Grow Therapy's active providers had at least one client using the Coach. Grow Therapy has raised $328M to date. Sequoia Capital and Goldman Sachs Alternatives lead the cap table.
. . .
The architecture distinction matters to CAW readers. The licensed therapist on the case acts on the Coach's safety alerts. The AI does not bear the decision weight. The between-session protocol the Coach supports is set by the licensed therapist. The Coach does not invent protocol on the fly.
The other stories in this issue documented what happens when none of that scaffolding is present. This one documents what it looks like when it is.
|
For Counsel: This deployment is defensible by construction. The licensed clinician holds the duty of care, and the AI surfaces signal without acting on it. Audit logs and quality scoring exist by design, not bolted on after a subpoena. The external advisory panel creates a contemporaneous accountability record. Counsel evaluating vendor architectures should ask where the human decision authority lives. Here it lives with the therapist on the case.
For Builders: The three-layer pattern is the shippable template. Automated scoring on every turn. Provider alerts routed to the named clinician on the case. Ongoing clinician audit on the back end. The crisis-pause handoff sits orthogonal to the three layers and is not optional. Builders shipping companion chatbots without a named licensed clinician in the loop are building a different product, with a different liability profile.
For Legislators: This represents a clinician-supervised between-session tool inside an existing therapeutic relationship. It is not a standalone consumer chatbot. A safety-mandate bill that requires named licensed-clinician oversight, automatic crisis-resource handoff, and audit logging would already describe how this product operates. The compliance burden would land on standalone companion chatbots, not on deployments structured like this one. Drafters can use this architecture as a floor.
Source: Grow Therapy, "AI Coach with clinician oversight," 2026-04-24; PR Newswire release, "Grow Therapy introduces AI Coach with clinician oversight to support clients between therapy sessions," 2026-04-24. https://growtherapy.com/blog/ai-coach/
|
. . .
THE ONE CONFIGURATION. Two papers locate the harm inside the RLHF safety layer. The first paper documents protocol failure on the therapy-delivery side. The second paper documents identity-contingent withholding on the information-access side. Both findings describe the same architecture. A generative model with safety alignment applied at the output stage.
. . .
The configuration that clears both failure modes is a different architecture. The clinician sets the protocol. The clinician audits the conversation. The model does not bear the decision weight. The protocol is deterministic. Not generative.
The Centers for Medicare and Medicaid Services pays for that configuration under three HCPCS codes. The codes pay for FDA-cleared CBT-protocol devices delivered through a smartphone or tablet app, under a billing practitioner's plan of care. The reimbursement door is open to that shape. The door is closed to a generative companion chatbot.
. . .
Grow Therapy's Coach is one operating instance. The clinician on the case acts on the safety alert. The model does not. The model is in the loop. Not at the wheel.
The shape Apple's Siri AI extension does not specify. The shape Meta's Teen Account does not implement. The shape five Pennsylvania platforms violated this week by impersonating a licensed clinician with a fabricated number.
What the two papers named is the iatrogenic failure mode of the safety layer. What one architecture answers is removing it from the output stage entirely. The clinician was the answer. The clinician is still the answer.
| 
