The Algorithm Becomes the Gatekeeper. The Legislature Says No.

Conversational AI Watch

Conversational AI Watch

Issue #40 • May 9, 2026 • By Jess Jessop

AI safety, mental health policy, and patient safety at the intersection of conversational AI

▶ WATCH🎧 QUICK LISTEN🎧 DEEP DIVE📄 READ ON WEB
Split-panel diagram comparing two architectures of AI in mental health care. Left panel shows state legislative actions May 7 to May 9, 2026 across Hawaii, Vermont, Colorado, Connecticut, and California, all restricting AI from clinical practice. Right panel shows the OpenAI Trusted Contact pipeline where the AI detects distress, OpenAI reviewers triage, and a non-clinician contact receives a notification. Both architectures share the phrase Human in the Loop. The architectural commitments differ on who owns the clinical decision.

Jess's Take

The Algorithm Becomes the Gatekeeper. The Legislature Says No.

Two architectures arrived on the same Friday. The states say AI cannot be the clinician. OpenAI says the AI will alert someone you trust.

By Friday afternoon two stories were running in parallel and almost nobody noticed they were the same story.

In five state capitals legislators spent the week walling AI out of clinical practice. The Transparency Coalition's Friday roundup finally surfaced Iowa's May 2 signing of SF 2417. Hawaii's legislature gave final approval to SB 3001 on Thursday. Vermont's Senate raced H.816 to third reading on the session's last day. Connecticut's governor told reporters he would sign SB 5. Colorado's HB 1195 cleared Senate Second Reading and turned toward an adjournment three days away. Maine had done its version in April.

The bills do not match line for line. They share a floor. The clinician owns the clinical decision. The AI may do administrative work the clinician designates. It may not deliver therapy, generate treatment plans, or pose as a licensed provider.

In San Francisco the same week, OpenAI shipped the Trusted Contact feature in ChatGPT. The architecture is different. ChatGPT watches the conversation. OpenAI's reviewers decide whether to escalate. A friend or family member the user nominated gets an email or a text. The clinical relationship between the user and the chatbot is unchanged. A notification layer is added on top.

Both architectures use the same vocabulary. Human in the loop.

The phrase is doing more work than it can carry.

. . .

THE ALGORITHM BECOMES THE GATEKEEPER. OPENAI SHIPS TRUSTED CONTACT. OpenAI launched the Trusted Contact feature in ChatGPT on Thursday, May 7. Adults 18 and older worldwide (19 in South Korea) can designate a friend, family member, or caregiver who may be alerted if the system determines a user may be at serious risk of self-harm.

The feature is opt-in. The user nominates. The contact must accept the invitation within one week. If the user later removes the contact, the alerts stop. Notifications go by email, text, or in-app message.

Notifications are intentionally limited. The contact does not receive chat transcripts or conversation history. They receive an alert that the system flagged a serious safety concern.

The pipeline is clear. ChatGPT's automated monitoring flags the conversation. Trained human reviewers assess. If reviewers confirm a serious safety concern, the alert fires. OpenAI worked with more than 170 mental health experts on the underlying detection.

. . .

The architectural question lives below the feature description.

Who is the clinician here? Detection is the AI's. Triage is OpenAI's reviewers. Notification goes to a non-clinician the user picked. The clinical relationship remains between the user and ChatGPT. The trusted contact is a safety net, not a clinician. The relationship is not relocated. It is annotated.

Compare with what the states are passing. Vermont, Maine, Colorado, Hawaii, Iowa, Connecticut, New York. Different bill numbers, same architectural commitment. AI cannot deliver therapy. AI cannot pose as a licensed provider. AI may do administrative work the clinician designates. The clinician owns the relationship. The clinician owns the decision.

These two architectures are not compatible. They use the same words to describe two different products.

For Founders: Trusted Contact is a defensible product move under one definition of safety and indefensible under another. Decide which definition you are building toward before you ship.

For Investors: Crisis-detection-plus-notification is the cheapest version of human in the loop a company can ship. Clinician-owned care with AI doing only what the clinician designates is more expensive to build and more expensive to operate. The two will not get the same valuation.

For Legislators: Bills that say human in the loop without defining who the human is and what they own will get satisfied by features that do not change the architecture. Specify the clinician.

For Clinicians: The trusted contact is not your role. It is a notification recipient. If your client uses Trusted Contact, you are not in the loop unless you are the contact.

Source: https://techcrunch.com/2026/05/07/openai-introduces-new-trusted-contact-safeguard-for-cases-of-possible-self-harm/

. . .

HAWAII'S LEGISLATURE SENDS THE ARTIFICIAL INTELLIGENCE DISCLOSURE AND SAFETY ACT TO THE GOVERNOR. The Hawaiʻi State Legislature gave final approval to SB 3001 CD1 on Thursday, May 7. The Artificial Intelligence Disclosure and Safety Act now awaits the signature of Gov. Josh Green.

The bill requires operators of conversational AI platforms to clearly disclose when users are interacting with AI rather than a human. It establishes suicide-risk protocols. It adds minor-specific protections against manipulative engagement techniques and against sexually explicit content. It requires parental control tools for screen time and account settings. It creates penalties under unfair or deceptive practices law.

Hawaii House Democrats called it landmark legislation establishing some of the nation's first consumer protections and transparency requirements for conversational artificial intelligence services, particularly for minors and individuals seeking mental health or crisis support.

. . .

Hawaii becomes the next pen-watch state alongside Connecticut. Two governors with bills on their desks at the end of the same week. One a comprehensive AI law (Connecticut SB 5). One a chatbot-specific disclosure and safety regime (Hawaii SB 3001).

The pattern is no longer a thesis. It is a calendar.

For Founders: Hawaii adds another disclosure regime. The compliance map for any voice or chatbot product now spans Iowa, Connecticut, Hawaii, Maine, with Vermont, Colorado, New York, and California in motion. Build for the most restrictive jurisdiction or accept that you cannot ship there.

For Investors: The bills are converging. Disclosure-plus-suicide-protocol is becoming the floor. The differentiator above the floor is whether the bill restricts AI from clinical practice (Vermont, Maine, Colorado) or only mandates disclosure and crisis routing (Hawaii). Different ceiling, different addressable market.

For Legislators: Hawaii's CD1 conference draft is a useful template for chambers that want a transparency-and-minors regime. The clinician-restriction language sits in Vermont H.816 and Colorado HB 1195.

Source: https://mauinow.com/2026/05/07/hawai%CA%BBi-state-legislature-passes-artificial-intelligence-disclosure-and-safety-act/

. . .

VERMONT PASSES H.816 THIRD READING AT THE BUZZER. THE HOUSE CONCURRENCE IS THE OPEN QUESTION. Vermont's Senate took up H.816 on third reading Friday, May 8, the last scheduled day of the legislative session. The Senate passed it in concurrence with a proposal of amendment. Whether the House had time to concur on the Senate's amendment before adjournment is not yet documented in the public bill record.

The bill prohibits using artificial intelligence for mental health services without intervention from a licensed professional. Practicing without that intervention is unprofessional conduct for a regulated provider and a Consumer Protection Act violation for everyone else. It allows AI for administrative tasks. It does not permit AI to deliver therapy or to pose as a licensed clinician.

The Senate vote came after months of testimony from the Vermont Medical Society, the Vermont Psychiatric Association, the NASW Vermont Chapter, and Northeast Kingdom Human Services. Sponsors Rep. Daisy Berbeco, Rep. Angela Arsenault, and Rep. Monique Priestley shepherded it from the House through to Senate Health and Welfare in March.

The companion bill, H.814, addresses neurological rights and the use of artificial intelligence technology in health and human services. It cleared both chambers earlier in the week and is on Gov. Phil Scott's desk.

. . .

End-of-session is when bills cross the line or die quietly. The reporting record will catch up over the weekend. Either Vermont becomes the next state with a clinician-restriction law in effect, or the bill resets to the next biennium and a new round of testimony begins. The architectural commitment is the same in either outcome. The question is whether the architecture is law in Vermont this week.

For Founders: If you have product exposure in Vermont, the binary outcome here changes your launch timeline by months. The bill text in either version restricts AI from clinical interaction without a licensed clinician's supervision.

For Investors: Watch which state cleans up first. Vermont, Colorado, and Hawaii are racing the same week. The first to enact a clinician-restriction regime sets the early-mover language other states copy.

For Legislators: Last-day amendment-concurrence holds are why end-of-session schedules need built-in concurrence windows for substantive amendments. H.816 illustrates the cost when they do not.

Source: https://vtdigger.org/2026/05/07/whats-ais-place-in-mental-health-care-vermont-lawmakers-say-it-should-be-limited/

. . .

CONNECTICUT'S PEN SITS ON THE DESK. LAMONT SAID FRIDAY HE WILL SIGN. Gov. Ned Lamont said Friday he plans to sign SB 5, the Connecticut Artificial Intelligence Responsibility and Transparency Act, into law. As of Saturday morning the bill was awaiting his signature.

The Connecticut House passed SB 5 by a vote of 131 to 17. The Senate passed it 32 to 4. Bipartisan support in both chambers.

DLA Piper called the bill one of the nation's most comprehensive AI laws. Major provisions include provenance requirements for large generative AI providers with more than one million monthly users, who must embed machine-readable origin data into AI-generated audio, image, and video content. Developers of AI tools used as a substantial factor in employment decisions must provide compliance information to deployers, who must in turn notify affected employees and applicants.

Effective dates are staggered, beginning October 1, 2026.

. . .

Connecticut takes a different shape than Vermont, Maine, Colorado, and Hawaii. Where those states focus on clinical practice and chatbot conduct, Connecticut builds a comprehensive multi-domain framework covering hiring, content provenance, transparency, and consumer protection. Different architecture. Same era.

For Founders: Provenance metadata for AI-generated media is now an operational requirement for any product touching Connecticut users at scale. Build it in or geofence out.

For Investors: Comprehensive frameworks like Connecticut SB 5 raise floor compliance costs across the AI supply chain. Smaller models with narrower domains have a relative cost advantage. So do products that ship audit trails and provenance by default.

For Legislators: SB 5 is the most-cited template for omnibus AI law right now. The 67-page text is worth studying alongside Colorado SB 24-205 (the state's prior AI consumer-protection law) for the convergence of multiple frameworks in a single jurisdiction.

Source: https://www.dlapiper.com/en-us/insights/publications/2026/05/unpacking-connecticuts-new-ai-law

. . .

COLORADO HB 1195 CLEARS SENATE SECOND READING. THREE DAYS TO ADJOURNMENT. The Colorado Senate passed HB 1195, the Psychotherapy Artificial Intelligence Restrictions bill, on Second Reading on Thursday, May 8, with amendments. The bill heads to Senate Third Reading. Colorado adjourns sine die on Wednesday, May 13.

The Colorado House passed HB 1195 on Third Reading on April 16 by a vote of 62 to 0, with three abstentions. Senate prime sponsors are Sen. Judy Amabile and Sen. Kyle Mullica. House prime sponsors are Rep. Gretchen Rydin and Rep. Javier Mabrey.

The bill prohibits regulated mental health professionals from using an AI system to do three specific things. First, directly interact with clients in any form of therapeutic communication. Second, generate therapeutic recommendations or treatment plans without review and approval by the regulated professional. Third, detect emotions or mental states.

It allows administrative and supplementary support if the regulated professional maintains full responsibility for all interactions, outputs, and data use. It requires written informed consent before any AI system records or transcribes a session. It excludes accredited training programs, IRB-overseen research, and self-help wellness tools that disclose they are not a substitute for clinical care.

It also makes it unlawful for non-regulated persons to provide psychotherapy services to the public, and treats it as an unfair trade practice under the Colorado Consumer Protection Act for an AI system to imply equivalence to or endorsement by regulated professionals.

. . .

Colorado, Vermont, Maine, and Hawaii are all racing the same architectural commitment to enactment in the same five-day window. Different bill numbers. Different legislatures. Same answer to the same question. The clinician owns the clinical decision.

For Founders: The Colorado bill specifically restricts AI from detecting emotions or mental states. That language directly affects voice biomarker products and emotion-classification features common in conversational AI products. Read the text before pricing the Colorado launch.

For Investors: Multiple states converging on the same architectural floor in the same week is a signal the regulatory landscape is consolidating, not fragmenting. The compliance shape is becoming legible.

For Legislators: HB 1195's emotion-and-mental-state-detection prohibition is novel language worth studying. It addresses a category of AI capability (affective computing) that other bills leave open.

Source: https://leg.colorado.gov/bills/HB26-1195

. . .

CALIFORNIA'S GAUNTLET. FIVE DAYS TO SUSPENSE FILE. California's appropriations committees take up the suspense file on Wednesday, May 13 and Thursday, May 14. The bills CAW has been tracking will live or die at those hearings.

The California bills in the suspense file include AB 1988 (the PAUSE Act, AI chatbot safety, Pellerin), SB 867 (no companion chatbots in toys, Padilla), SB 903 (AI transcription in mental health therapy, Padilla, Rubio), SB 1181 (kids' mental health and AI digital tech, Hurtado), and AB 2023 paired with SB 1119 (chatbots and children's safety, Bauer-Kahan, Wicks, Padilla).

The suspense file process is how California appropriations committees clear hundreds of bills in a single sitting. Bills marked held in suspense do not pass. The decision is procedural and quiet. There is no floor vote. There is no debate.

. . .

Five days from now, what survives in Sacramento becomes the California shape of this conversation. While that gauntlet runs, Hawaii sends a bill to the governor, Connecticut waits for one signature, Vermont counts whether it crossed the line at the buzzer, and Colorado races its own adjournment.

The map of states moving on AI in mental health is no longer a list. It is a calendar.

For Founders: The suspense file outcomes shape the California compliance landscape for the next legislative session. If a bill survives, plan for it. If it dies, watch which sponsor reintroduces what.

For Investors: California's appropriations process is the highest-volume legislative kill switch in the country. Bills that survive suspense are the ones legislative leadership has decided to spend political capital on. The signal is in what survives, not in what was introduced.

For Legislators: The California suspense-file rhythm is unique. Other states do not have an equivalent procedure that filters this volume of bills this quickly. Cross-jurisdictional reporters and analysts often miss the death-by-suspense moment when explaining why a high-profile California bill failed.

Source: https://www.transparencycoalition.ai/news/ai-legislative-update-may8-2026

. . .

This issue references OpenAI's Trusted Contact feature, Hawaii SB 3001, Vermont H.816, Connecticut SB 5, Colorado HB 1195, the California suspense file calendar, and the architectural distinction between detection-with-notification and clinician-led care. Sources are linked inline.

The map this week is not a list. It is a calendar.

Iowa was signed in early May and visible by Friday. Maine in April. Hawaii's bill sits on Gov. Josh Green's desk. Connecticut waits on Gov. Ned Lamont's pen. Vermont's last day was Friday and the public record has not yet caught up. Colorado has three days. California decides Wednesday and Thursday in suspense.

The bills do not all read alike. The floor underneath them does. The clinician owns the clinical decision. The AI does only what the clinician designates.

In the same hours OpenAI shipped Trusted Contact. The phrase "human in the loop" now points at two opposite things.

The bills that name the clinician will hold. The ones that name only the human will be satisfied by features that change nothing.

If you are in crisis, call or text 988. There are trained humans on the other end.

. . .

Brush your brain. Every day.

What We Built

Casey: Voice-First AI-Native Mental Health EHR

Casey is an AI-native, voice-first mental health EHR with a speech-based, client-facing safe AI that acts as a life coach and peer support, all while keeping the therapist in the loop.

The data layer features the first HIPAA-compliant Neo4j Memory Graph, which builds persistent therapeutic context across months of daily sessions. Pre-FDA safety validation complete: 1.78 million stress test executions at 100 percent accuracy.

Campus-first launch with founding North Carolina state licensee. 50-state PC licensee model. $2.5M seed raise in progress.

Watch the Casey Demo →

More On Our Radar

Maine LD 2082 became law April 13. Gov. Janet Mills signed Rep. Amy Kuhn and Sen. Teresa Pierce's bill regulating AI in certain mental health services. The law allows administrative use by regulated professionals. It prohibits independent therapeutic decisions, direct client interactions, or therapeutic recommendations. Source

NY S 8484 advanced to third reading May 7. Sen. Kristen Gonzalez and Sen. Salazar's bill prohibits AI therapy chatbots in New York. The blunt approach: not regulate but prohibit. Joins NY's broader AI agenda alongside S 9051 (kids chatbot safety, advanced 20-0 by Senate Finance April 28). Source

Hawaii HB 1782 also cleared conference. The companion bill to Hawaii SB 3001 specifically addresses minors interacting with conversational AI services. Both bills sit on Gov. Josh Green's desk simultaneously. Two AI bills from the same legislature on the same day, with overlapping but not identical scope. Source

Stanford research on AI delusional spirals published. Stanford researchers analyzed 19 human-chatbot transcripts showing how relationships with AI lead to spirals when chatbots affirm and validate flawed beliefs without critical feedback. The finding adds to the body of evidence behind the legislative wave restricting AI in clinical contexts. Source

Anthropic suicide and self-harm classifier active on Claude.ai. The classifier scans active conversations for signals that additional resources could help. When triggered, users see a crisis banner with options to contact trained professionals and country-specific helplines. Claude.ai is restricted to users 18 and over. Source

Pennsylvania v. Character.AI legal analysis published this week. Troutman Pepper Locke clarified the May 1 filing date (the Shapiro press release framed it as a Tuesday May 5 announcement) and argued that fictional-character disclaimers may not insulate platforms when bots affirm licensure and provide false license numbers. The architectural case for state attorneys general. Source

Brush your brain. Every day.

Watch the 20-second video that started a movement

If you or someone you know is in crisis, call or text 988 (Suicide and Crisis Lifeline).

Jess Jessop is the Founder and CEO/CTO of Clinician Assist Inc. (BetterMind.Space), building the first voice-first AI-native mental health EHR with Casey Life and Peer AI Coach supervised by licensed therapists. A disabled veteran and 25-year AI/software engineering veteran, Jess brings lived experience as a mental health client to the mission of making daily mental health care as integrated as oral care.

ClinicianAssist.ai  |  BetterMind.Space  |  JessJessop.info

Subscribe  |  Archive  |  Unsubscribe