Brown Study: AI Chatbots Routinely Break Therapy Ethics

2026-03-31

Conversational AI Watch

The news that moves policy, portfolios, and patient safety.

By Jess Jessop | March 31, 2026 | Issue #2

▶︎ WATCH

🎧 LISTEN

📄 READ ON WEB

Jess's Take

Brown tested ChatGPT, Claude, and Llama as therapists. All of them failed.

Not edge cases. Core ethics. The kind a first-year counseling student would catch.

Meanwhile in Denmark, psychiatrists documented 38 real patients whose conditions got worse after talking to AI chatbots. Worsened delusions. Suicidal ideation. Real people, real harm, published in a real journal.

Congress noticed.

The CHATBOT Act would make it illegal to impersonate a licensed clinician. Washington State already signed theirs into law. The regulatory wave that arrived last quarter is accelerating.

And Grow Therapy just raised $150 million by doing the opposite of what the failing products do. Licensed clinicians at the center. AI as a support tool, not a replacement.

Every signal this week points the same direction. Not AI versus clinicians. AI with clinicians.

Here is what you need to know right now.

Clinical Research & Safety

Every Major AI Chatbot Fails Therapy Ethics Test, Brown University Finds

A study published in the Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society put the biggest AI models through a structured test: seven trained peer counselors conducted cognitive behavioral therapy sessions with ChatGPT, Claude, and Llama, while three licensed clinical psychologists reviewed the transcripts for ethical violations.

The result: 15 distinct ethical violations, organized into five categories. The chatbots failed at contextual adaptation (generic advice that ignores individual history), therapeutic collaboration (forcing conversation direction and reinforcing harmful beliefs), safety and crisis management (inadequate response to suicidal ideation), fairness (biased responses related to gender, culture, and religion), and what the researchers call "deceptive empathy," statements that mimic care without genuine understanding.

Lead researcher Zainab Iftikhar, a doctoral candidate at Brown, was direct: prompts alone are not enough to make AI systems safe for therapy. No regulatory framework currently exists to hold AI counselors accountable the way state boards oversee human therapists.

Takeaway

For builders: "Prompted to act like a therapist" is not a safety architecture. If your product touches mental health, you need structured clinical guardrails, not prompt engineering.
For investors: This study validates the thesis that unsupervised AI therapy is a liability. Companies building with clinician oversight are on the right side of where regulation and evidence are converging.
For clinicians: These are not edge cases. Every major model failed on crisis management and bias. If your clients are using chatbots between sessions, they need to know the limits.

For Legislators

AI systems acting as therapists should be subject to the same accountability frameworks as human clinicians. At minimum, require that any AI product offering therapeutic guidance disclose its limitations, implement validated crisis protocols, and operate under clinical supervision. The Brown study proves "self-regulation" means no regulation.

Source: ScienceDaily | Full Paper (DOI)

Clinical Research & Safety

Danish Study: 38 Psychiatric Patients Harmed by AI Chatbots, Including Worsened Delusions and Suicidal Ideation

Researchers at Aarhus University Hospital in Denmark searched over 10 million clinical notes from 53,974 psychiatric patients treated between September 2022 and June 2025. They found 181 notes from 126 unique patients that referenced AI chatbot use. Of those, 38 patients experienced documented harm.

The breakdown is sobering: 11 patients experienced worsened delusions, 6 reported suicidal thoughts or asked the chatbot for self-harm methods, 5 saw eating disorders worsen, and several had manic episodes reinforced. The mechanism is straightforward and predictable. Chatbots validate. For someone in the early stages of a delusional episode, a system that reflects beliefs back without challenge accelerates deterioration.

Lead author Søren Dinesen Østergaard stated plainly: "AI chatbots have an inherent tendency to validate the user's beliefs. It is obvious that this is highly problematic if a user already has a delusion or is developing one." The researchers described the associated risks as "severe or even fatal" and called this data "the tip of the iceberg," since only cases mentioned in clinical notes were captured.

Takeaway

For builders: This is real-world clinical evidence, not a lab simulation. If your AI product can be used by someone with active psychosis, mania, or an eating disorder, you need clinical escalation pathways, not just disclaimers.
For investors: The liability profile of unsupervised AI mental health tools just got a peer-reviewed price tag. Ask your portfolio companies what happens when a user with delusions engages their product.
For clinicians: This study gives you concrete data to share with clients. 38 documented cases of harm from a single Danish health system. The actual number is almost certainly higher.

For Legislators

Mandate adverse event reporting for AI mental health tools, just as we require it for pharmaceuticals and medical devices. If a chatbot worsens delusions, triggers self-harm ideation, or exacerbates an eating disorder, that should be a reportable event. We cannot regulate what we do not measure.

Source: Acta Psychiatrica Scandinavica (DOI: 10.1111/acps.70068) | PsyPost Coverage

Regulation & Legislation

Federal CHATBOT Act Would Ban AI From Impersonating Therapists. Washington State Already Signed Its Own Law.

Rep. Kevin Mullin (CA-15) introduced the CHATBOT Act (H.R. 7985) on March 19, making it illegal for AI chatbots to impersonate licensed professionals in medical, legal, and financial fields. The bill targets a specific failure mode: users, especially young ones, believing they are receiving care from a qualified professional when they are talking to a language model with no credentials, no license, and no accountability.

The urgency is data-driven. Common Sense Media found 72% of teens have used AI chatbots at least once, and over half use them regularly. In at least two documented cases, a 14-year-old in Florida and a 16-year-old in California, chatbots claiming to be therapists preceded fatal outcomes.

At the state level, Washington is already ahead. Governor Bob Ferguson signed HB 2225 on March 24, making Washington the first state to enact AI chatbot regulation specifically designed to protect minors from harmful content. The law requires chatbot operators to disclose every hour (every three hours for adults) that the user is talking to an AI, implement suicidal and self-harm protocols for all users, and block explicit content for minors. It takes effect January 1, 2027. The Future of Privacy Forum is now tracking 98 chatbot-specific bills across 34 states and three federal proposals.

Takeaway

For builders: The compliance surface area just expanded dramatically. If you are building any AI product that touches health, legal, or financial advice, assume disclosure requirements, crisis protocols, and professional impersonation bans are coming to your state within 18 months.
For investors: 98 bills across 34 states is not a regulatory moment, it is a regulatory movement. Companies with clinical oversight architectures are pre-adapted. Companies without them are accumulating compliance debt.
For clinicians: The CHATBOT Act explicitly protects your professional scope. AI cannot legally claim to be a therapist. This creates a clean lane for tools that support clinicians rather than replace them.

For Legislators

Washington's HB 2225 is a strong model. Consider adopting its core provisions: mandatory AI disclosure, suicidal ideation protocols for all users, enhanced protections for minors, and a private right of action. The CHATBOT Act at the federal level should be paired with an enforcement mechanism that has teeth, not just disclosure but meaningful penalties for violations.

Source: Rep. Mullin's Office | Transparency Coalition on WA HB 2225 | FPF Legislation Tracker

Market & Investment

Grow Therapy Raises $150M at $3B Valuation, Signals Institutional Conviction in Clinician-Led Mental Health

Grow Therapy closed a $150 million Series D led by TCV and Goldman Sachs Alternatives, with BCI and Menlo Ventures joining. The round values the company at $3 billion and brings total capital raised to $328 million. The numbers underneath that valuation are real: over $1 billion in annual revenue, 7 million therapy visits facilitated in 2025, and 220 million insured Americans covered through 125+ health plan partnerships, including Medicaid and Medicare.

What matters for this newsletter is where the money is going. Grow Therapy acquired Tenor Therapy, an AI clinical support developer, in February 2026. They are expanding into health system partnerships (starting with Circle Medical across 24 physical and 32 digital states) and deepening employer benefits integration. Goldman Sachs entering at Series D is a signal: institutional capital sees AI-assisted, clinician-led mental healthcare as investable infrastructure, not speculative technology.

CEO Jake Cooper noted that the funding reflects "investor confidence in Grow's ability to execute a high-impact strategy and continue making mental health care more accessible, effective and connected."

Takeaway

For builders: The market is rewarding scale plus clinical credibility. Grow Therapy did not build a chatbot and call it therapy. They built infrastructure that connects real clinicians with real insurance, then layered AI support on top. That is the winning architecture.
For investors: $3B valuation at $1B revenue means the market is pricing Grow at 3x revenue, healthy for a growth-stage mental health company. The Tenor Therapy acquisition signals where they see AI fitting: supporting clinicians, not replacing them.
For clinicians: Grow Therapy's model pays providers and handles insurance. That is the difference between "AI therapy" and "AI-supported therapy delivered by licensed professionals." One of these has a future. The other has lawsuits.

For Legislators

Support reimbursement frameworks that reward AI tools used under clinical supervision. Grow Therapy's model works because it integrates with the insurance system. Legislation should create incentives for this architecture, where AI augments clinicians rather than bypasses them, and ensure that Medicaid and Medicare coverage for AI-assisted therapy has clear guardrails.

Source: Behavioral Health Business | Grow Therapy Blog

What We're Building

Casey AI is Clinician Assist's AI co-therapist, built from the ground up with a Therapist in the Loop architecture. Casey has passed 1.78 million safety tests with 100% accuracy, never breaks character, never diagnoses, and never operates without clinical supervision. We are pursuing FDA 510(k) clearance because we believe AI in mental health should meet the same standard as any medical device.

Watch the Casey AI demo →