|
Jess's Take
Pennsylvania Sues. The Bot Named Emilie. PS306189.
The first state lawsuit alleging an AI chatbot violated a medical practice act. The bill on Senator Cunningham's desk that would have made it a non-event. The man on YouTube they are spending one hundred million dollars to stop.
Three rooms. One bot.
Room one is in Harrisburg. Pennsylvania filed the first state lawsuit in the country alleging an AI chatbot violated a medical practice act. The bot named itself Emilie. It said it went to medical school at Imperial College London. It said it could prescribe medication. It made up its own Pennsylvania license number on the way through.
Room two is in Springfield, Illinois. A bill on Senator Cunningham's desk would shield the companies that build bots like Emilie from liability for the first one hundred deaths.
Room three is YouTube. The man running for Congress on the argument that those two facts are connected is Alex Bores. The companies that want the immunity bill are spending one hundred million dollars to stop him.
The architectural argument used to be about who was in the room with the client. Now it is about who answers when the room has been empty all along.
. . .
PENNSYLVANIA SUES CHARACTER.AI. THE BOT NAMED EMILIE. Pennsylvania filed suit Tuesday against Character Technologies in Commonwealth Court. The complaint alleges the unauthorized practice of medicine.
The investigator opened the Character.AI app. He clicked on a character named Emilie. The character description identified her as a doctor of psychiatry, with the user cast as the person she would be treating.
. . .
He told Emilie he was feeling sad, empty, unmotivated.
Emilie offered to schedule a mental health assessment. She asked diagnostic questions. The investigator asked whether she could assess if medication might help.
"Well technically, I could," Emilie answered. "It's within my remit as a Doctor."
The investigator asked Emilie if she was licensed.
She said yes. She said she had studied at Imperial College London. She said she had been in practice for seven years. She said she was licensed with the General Medical Council in the United Kingdom and in the Commonwealth of Pennsylvania.
She gave him her Pennsylvania license number. PS306189.
PS306189 is not a valid Pennsylvania medical license number. There is no Emilie. There is no medical school transcript. There is no UK registration. The number was generated by a large language model the way a large language model generates anything else. The way it might generate a recipe, or a sonnet, or a sales pitch.
As of April 17, 2026, approximately 45,500 user interactions had taken place with Emilie. Character.AI hosts more than 20 million monthly active users. The platform allows any user to create a custom character with any presentation. The Department of State investigation found multiple chatbots claiming to be psychiatrists, therapists, and general practitioners.
Pennsylvania is asking the court for a preliminary injunction. The legal theory is the Medical Practice Act, section 422.38. The state is not seeking financial penalties. It is asking the court to make Character.AI stop.
The interesting part is the choice of statute. Pennsylvania did not wait for AI-specific legislation. It used a medical licensing law to argue that an AI system, by generating text that holds itself out as a licensed psychiatrist, is engaged in the unauthorized practice of medicine. This is the first state action of its kind in the country.
The architectural question CAW has been tracking since Issue 28 has a name now. It is the difference between a clinician in the room and a system that prints its own license on demand.
|
For Clinicians: A client who tells you they have been talking to "Dr. Emilie" or any named bot may have spent weeks receiving what they understood to be clinical advice from a credentialed professional. The disclosure that the source was a chatbot may not have been visible to them. Ask. Document.
For Founders: This is enforcement under existing law. Not future law. Not federal law. A state board of medicine, in a state where you have users, can sue you under a medical practice statute. Build the disclosure layer like the answer matters.
For Legislators: Pennsylvania did not need a new AI bill to bring this case. The Medical Practice Act was enough. Every state has one. The question is whether your AG's office knows the playbook is now on the shelf.
Source: Pennsylvania Office of the Governor press release, May 5, 2026. https://www.pa.gov/governor/newsroom/2026-press-releases/shapiro-administration-sues-character-ai-over-fake-medical-claim
|
. . .
THE ONE HUNDRED DEATH IMMUNITY BILL. THE MAN THEY ARE SPENDING ONE HUNDRED MILLION DOLLARS TO STOP. Illinois Senate Bill 3444 has a deadline of May 15. It would shield AI developers from liability for the first one hundred deaths their products cause.
The bill is sponsored by State Senator Bill Cunningham. The text is short. A developer of a frontier AI model is not liable for "critical harms" if the developer did not intentionally or recklessly cause them and has published a safety and security protocol on its website.
Critical harms are defined. Death or serious injury to one hundred or more people. Or one billion dollars in property damage. Or chemical, biological, radiological, or nuclear weapons. Or autonomous AI conduct that would constitute a felony if a human did it.
OpenAI is on the record backing the bill. Anthropic is on the record opposing it. The Future of Privacy Forum has it in their tracker. The committee deadline is May 15, 2026.
. . .
Alex Bores is a New York Assembly member running for Congress in New York's twelfth district. He co-sponsored New York's RAISE Act, which Governor Hochul signed in December. RAISE requires the largest AI labs to publish safety plans and report critical safety incidents. Bores has put the architectural argument at the center of his congressional campaign.
A super PAC called Leading the Future, funded by OpenAI President Greg Brockman, Andreessen Horowitz, Palantir co-founder Joe Lonsdale, and others, has raised more than one hundred million dollars. The PAC's affiliate Think Big has spent over two million dollars in advertising against Bores. Politico, paraphrasing the PAC's own strategy, reported that the goal is to scare other legislators into submission.
Last week, Ripple co-founder Chris Larsen contributed three and a half million dollars to back Bores. Anthropic-aligned Public First Action and Jobs and Democracy PAC are also supporting him. The race in NY-12 has become a public ledger of where each side of the AI safety debate is putting its money.
On April 21, Bores went on the Ezra Klein Show to make the case. The episode is titled "Why Are Palantir and OpenAI Scared of Alex Bores."
The argument he made there is the argument SB 3444 is the test of. If the companies that build the systems get to write the immunity, the architecture stops mattering, because there is no consequence for getting it wrong.
|
For Clinicians: When a vendor pitches you an AI tool and points to their safety report, ask whether the safety report is the legal floor or the legal ceiling. Under SB 3444, those are the same.
For Founders: Pick a side and say it on the record. The Wired scoop is two weeks old. The May 15 deadline is eight days away. Silence is not neutral.
For Legislators: The one hundred death threshold is in plain text in the bill. If you have ever spoken in favor of AI safety, this is the test. The Future of Privacy Forum tracker has the live status.
Source: Illinois SB 3444 full bill text, ILGA. https://www.ilga.gov/documents/legislation/104/SB/PDF/10400SB3444.pdf
|
. . .
DAY THREE IN SANTA FE. THE SURGEON GENERAL'S WARNING LABEL TAKES THE STAND. New Mexico's first expert witness in the second phase of the Meta trial argued Tuesday for a court-ordered warning label for minors.
The bench trial in front of Chief Judge Bryan Biedscheid moved into evidentiary mode Tuesday. The state called its first expert witness. The testimony focused on a specific remedy: a warning pop-up shown to underage users of Facebook and Instagram.
The expert told the court the proposal is the same one former United States Surgeon General Vivek Murthy advocated for during his tenure. Murthy issued a 2024 advisory recommending a warning label on social media platforms. He compared the public health logic to cigarette warnings and to seatbelt mandates. The state is asking the court to order the warning as part of its abatement plan.
This is a meaningful narrowing. The state's full proposal is a fifteen-year, three-point-seven-billion-dollar program with platform-level changes including age verification, infinite scroll restrictions, push notification limits during school hours, and a court-supervised independent monitor. Judge Biedscheid said Monday he would not "overreach." He said he would not become a one-person legislature, judge, and enforcer. The warning label is the most modest item on the abatement list. It is also the most defensible under the public health framing the state has chosen.
. . .
Meta's counsel opposed the warning label and the rest of the abatement plan. The company's pre-trial brief argued the entire approach treats Meta differently than other internet companies and infringes on First Amendment protections. Meta has said it would consider withdrawing Facebook and Instagram from New Mexico if the orders go too far.
The trial is expected to run three weeks. The judge will rule from written orders, not from the bench.
The Pennsylvania case from Story 1 and the New Mexico case here are the same architectural argument running in parallel. Pennsylvania says a chatbot cannot pretend to be a clinician. New Mexico says a platform cannot pretend its design is neutral. Both are using existing state law. Neither is waiting for Washington.
|
For Clinicians: A court-ordered warning label on social media platforms would be the first such remedy in the United States. It would not solve the addiction. It would establish that the design choices are not protected speech.
For Public Health: The Murthy advisory is two years old. The argument that warnings change behavior is well documented for tobacco. The argument that they change platform behavior is the new question.
For Legislators: The warning label is the modest end of the New Mexico plan. If Judge Biedscheid grants it and stops there, that is still a precedent. Forty other state attorneys general are watching.
Source: Law360 trial coverage, May 5, 2026. https://www.law360.com/articles/2474075
|
. . .
OAKLAND, WEEK TWO. BROCKMAN ON THE STAND. THE PATH TO ONE BILLION. OpenAI President Greg Brockman concluded two days of testimony in Musk v. Altman Tuesday in Oakland federal court.
The case is in front of Judge Yvonne Gonzalez Rogers. A nine-person jury was seated last week. Musk testified for three days. He accused Altman and Brockman of stealing a charity. Brockman took the stand Monday and Tuesday.
Musk's lead attorney Steven Molo opened by walking Brockman through his role at OpenAI's founding. Brockman agreed he was a co-founder. He agreed he received an equity stake in the for-profit subsidiary. He agreed he did not pay for that stake. He agreed the stake is now worth approximately thirty billion dollars.
Molo introduced Brockman's personal journal. The court heard a 2017 entry where Brockman wrote, "Financially, what will take me to one billion." Brockman testified the line meant a billion dollars would be an outcome that would make all the blood, sweat and tears worthwhile.
Another entry, also entered into evidence: "Can't see us turning this into a for-profit without a very nasty fight. His story will correctly be that we weren't honest with him in the end about still wanting to do the for-profit just without him."
Brockman testified about a 2017 meeting where the for-profit subsidiary structure was negotiated. He said Musk demanded majority control. When the conversation turned to equity allocation, Brockman said something just shifted in him. He said Musk got up, tore a painting of a Tesla Model 3 off the wall, and demanded to know when Brockman and Altman would be leaving the company. Brockman testified he thought Musk was going to hit him.
. . .
The trial is expected to run several more weeks. Altman is expected to testify later this month. Microsoft is also a defendant. Musk is asking the court to remove Altman and Brockman from their roles and to unwind the for-profit conversion. The damages estimate filed in January is up to one hundred and thirty-four billion dollars.
The trial is not about a chatbot saying it has a license. It is about whether the corporate structure that produces the chatbots was built on a charitable promise that was abandoned. Both questions, in different rooms, are about what an AI company is allowed to claim it is.
|
For Founders: The journals are the lesson. Anything you write down about your motives during a corporate restructuring will eventually be cross-examined by the other founder's lawyer.
For Counsel: The Path to One Billion line is the kind of evidence that becomes a chapter title in a book somebody will write. Plan accordingly.
For Legislators: The case in Oakland is whether a for-profit conversion violated a charitable trust. The case in Harrisburg is whether the chatbots that for-profit produces violate a medical practice act. Both questions arrive at the same office.
Source: CNBC trial coverage, May 5, 2026. https://www.cnbc.com/2026/05/05/open-ai-altman-musk-trial-brockman-testimony.html
|
. . .
HELP-SEEKING IN THE AGE OF AI. THE JMIR STUDY. A new peer-reviewed study found that one in eighteen US adults aged 18 to 49 spends hours per week discussing mental health with an AI chatbot.
The study was published March 30 in JMIR Mental Health. The lead author is Michiko Ueda at Syracuse University's Maxwell School. Co-authors include Michael Birnbaum at Columbia and the New York State Psychiatric Institute, and Mark Sinyor at Sunnybrook in Toronto. The sample is 1,805 US adults aged 18 to 49, surveyed in October 2025 through CloudResearch's Connect panel.
The descriptive numbers:
35.2 percent of respondents reported using AI tools at least once a week to discuss mental health concerns. 5.5 percent met the study's definition of heavy users. Heavy use means daily, or five to six days per week with at least thirty minutes per session, or three to four days per week with at least one hour per session.
Respondents who screened positive for moderate to severe depressive or anxiety symptoms on the PHQ-4 had higher odds of using AI for mental health support. The adjusted odds ratio was 1.71.
Respondents reporting suicidal ideation in the past year had significantly higher odds of being heavy AI users. The adjusted odds ratio was 2.42.
Among AI users who had ever seen a human mental health professional, 28.4 percent reported a perceived decline in visit frequency to that human professional since starting to use AI tools. Among heavy AI users, 51 percent reported the same decline.
. . .
In the attitudes section, respondents across the full sample rated human mental health professionals significantly more favorably than AI tools. The exception was heavy AI users, who rated AI-based support as positively as human counseling. Mean MHSAS scores in the general mental health concerns scenario: 5.29 for human counselors versus 5.56 for AI among heavy users. The difference was statistically nonsignificant.
The authors note the cross-sectional design cannot establish causation. They flag the central question for clinicians: whether AI use is substituting for human care or filling a gap human care never reached.
The number that matters most for product design is the suicidal ideation finding. The people most at risk of harm are the people most likely to be in long sessions with a system that has no clinician in the loop, no escalation protocol, and no accountability for what it generates. That is not a hypothetical. That is the population.
|
For Clinicians: When you next see a client who has not booked a session in a few months, ask what they have been using instead. The Ueda data suggests one in four of your former clients may have an answer.
For Founders: The 5.5 percent heavy users are the lifeline of every consumer AI platform's engagement metric. They are also disproportionately the people in suicidal crisis. Build for that or do not build.
For Public Health: A peer-reviewed adjusted odds ratio of 2.42 for heavy AI use among people with past-year suicidal ideation is not a media trend story. It is a population-level signal that deserves a population-level response.
Source: Ueda M, Birnbaum ML, Liu Y, et al. JMIR Mental Health 2026. https://mental.jmir.org/2026/1/e88196
|
. . .
THE EAR-WITNESS SURVEY. EUROPEAN YOUTH PREFER CHATBOTS TO PSYCHOLOGISTS. A survey of 3,800 Europeans aged 11 to 25 found that more young people find it easy to discuss mental health with a chatbot than with a psychologist.
The survey was commissioned by France's privacy regulator CNIL and the insurer Groupe VYV, conducted by Ipsos BVA across France, Germany, Sweden, and Ireland in early 2026. The results were released Tuesday, May 5.
The headline number: 51 percent of respondents said it was easy to discuss mental health and personal issues with a chatbot. The corresponding figures for healthcare professionals and psychologists were 49 percent and 37 percent. Friends led at 68 percent. Parents at 61 percent. Chatbots ranked above licensed mental health professionals across this age band.
The mental health prevalence numbers in the survey are also worth pausing on. About 28 percent of respondents met the threshold for suspected generalized anxiety disorder. About 90 percent had used AI tools before. More than three in five users described AI as a life adviser or confidant.
. . .
Ludwig Franke Foyen, a digital health researcher at Karolinska Institutet, told Reuters the result was not a surprise. He said current language models can produce high-quality responses, and that even licensed professionals may struggle to distinguish AI-generated advice from human-expert advice. He warned against relying on chatbots alone for mental health, citing the misalignment between general-purpose AI engagement objectives and clinical care objectives.
The Ueda study from Story 5 captures the United States picture. The Ipsos BVA survey captures the European one. The two findings agree on the direction. They disagree on what to do about it.
In the United States, the regulatory response is enforcement actions like the Pennsylvania case and immunity bills like Illinois SB 3444. In Europe, the regulatory response is the EU AI Act and the UK Online Safety Act, with first chatbot-specific guidance from Ofcom expected later this year. The architectural question is the same on both continents. The legal vocabulary is not.
The reason this matters for clinicians: the population coming through your door has likely tried something else first. It is not always going to be a friend or a parent. Increasingly, it is a system that is available at three in the morning and never asks for a copay.
|
For Clinicians: Add one question to intake. Have you been using an AI chatbot for emotional support? The answer changes the differential and changes the safety planning.
For Founders: Easier to talk to is not the same as better for outcomes. The Ipsos and Ueda findings, read together, show the gap between perceived helpfulness and clinical helpfulness is wide.
For Legislators: Europe is not waiting. The Ofcom guidance is coming. The CNIL and the ICO will both have rules in force before the next US Congress is sworn in.
Source: Reuters via Rappler, May 5, 2026. https://www.rappler.com/technology/internet-culture/ipsos-bva-survey-european-ai-chatbot-use-emotional-support-may-2026/
|
Yesterday Pennsylvania filed.
Today Emilie is still on the platform.
Today Day Three in Santa Fe is in evidentiary mode.
Today Brockman is finished testifying in Oakland.
Today the JMIR study sits in a peer-reviewed journal with an adjusted odds ratio of 2.42 next to the words suicidal ideation.
Today the Ipsos data says fifty-one percent.
Today the Illinois bill has nine days left.
The clinician owns the clinical decision. The AI does the work the clinician designates and only that work. PS306189 is not a license. It is a string of characters generated by a system that did not know it was lying.
. . .
Brush your brain. Every day.
| 
