Conversational AI Watch

The news that moves policy, portfolios, and patient safety.

By Jess Jessop  |  October 12, 2026  |  Issue #183

LISTEN & WATCH ANYWHERE

DEEP DIVE  ·  Spotify  ·  Apple  ·  Amazon  ·  Pocket Casts  ·  RSS

QUICK LISTEN  ·  Spotify  ·  Apple  ·  Amazon  ·  Pocket Casts  ·  RSS

VIDEO  ·  Spotify  ·  Apple  ·  YouTube  ·  Pocket Casts  ·  RSS

ALSO ON  Substack  ·  Full archive  ·  X

Trump health officials let AI and tech firms into a 1,700-member Slack, and invited them there to a closed FDA session on patient chatbots.

The Chat Room. Trump health officials ran a Slack of 1,700 members, with only a handful of patient advocates, doctors and hospital representatives among them, and invited Microsoft, Anthropic, OpenAI, Apple and Google there to a closed FDA session on patient chatbots. One official hoped the agency’s work would be a “sales engine” for health apps. Story 1.

. . .

The Card. On Saturday, Elon Musk told his followers to photograph a credit card and drop it into Grok Bot. Thirty-six minutes later, his own chatbot answered: “No. ... Don’t do it.” Story 2.

. . .

The Day After. Executives at Anthropic, OpenAI and other AI companies are privately rehearsing for a public and political revolt after a catastrophic AI event, Axios reports, and racing to educate Congress before it comes. Story 3.

. . .

The Brake. Microsoft’s Satya Nadella says advanced AI models need an emergency brake, and “an authorized person” who can pull it mid-task. Story 4.

. . .

The Regulator. Europe’s tech chief says the bloc is “well equipped” for rogue agents under its AI Act, on the day the Commission’s experts met on loss-of-control incidents. Story 5.

. . .

The Voice. ALS is taking Terry’s speech. Now he thinks the words and hears them spoken in a voice rebuilt from his own. Story 6.

Today’s front page

. . .

In February, the FDA held a meeting about chatbots for patients. The public was not invited and that is not ok. The invitation went out in a Slack run by Medicare’s agency, where only a handful of the 1,700 members are patient advocates, doctors or hospital representatives.

On a Zoom call with industry, one official said he hoped the agency’s work would be a “sales engine” for the apps. Another, the official who welcomed members to the Slack, said patients can now move their records into any app with “no regulation, no rulemaking.”

On Saturday, Elon Musk told his followers to photograph their credit cards for Grok. His own chatbot told them not to, and described a route where the buyer approves each purchase on a phone.

The companies building these machines are rehearsing the day after a disaster, Axios reports, and they want a say in the bills that follow. Microsoft’s chief executive wants a brake on advanced models and a person with a hand on it. Europe says its law already reaches the rogue ones.

And Terry, who is losing his speech to ALS, thinks a sentence and hears it in a voice modeled on his own. He is helping build it for others who cannot speak.

Each of these machines answers to someone, and today’s paper asks who.

FDA: Put the public back in the room. Closed meetings with unelected officials making rules for the fat cats is what communist countries do.

. . .

AI FIRMS LOBBIED HEALTH OFFICIALS IN A PRIVATE SLACK! In February, federal health officials invited Microsoft, Anthropic, OpenAI, Apple and Google to a listening session on conversational AI products for patients, such as chatbots.

Representatives from at least 35 industry organizations attended the meeting, held by the Food and Drug Administration. It was not on the FDA’s public calendar or regulatory notices, and the public was not invited. The invitation went out in a chat room on Slack, run by the Trump administration’s Centers for Medicare & Medicaid Services (CMS).

KFF Health News reported the findings on Oct. 9 after reviewing thousands of Slack messages. CMS officials have suggested that apps in its Medicare App Library will be prioritized for a new program that reimburses companies for AI chatbot advice or wearable device tracking.

The February invitation came from Morgan Taylor, a CMS senior policy adviser. “This is a great opportunity to support HHS’ goals and share your state-of-the-art understanding of safe Conversational AI development and monitoring to help FDA shape future guidance,” Taylor wrote. Companies were asked in advance to explain how their products access patients’ sensitive medical information.

Amy Gleason, the former acting administrator of the Department of Government Efficiency, welcomed members to the workspace in August 2025. She is now chief product officer of the new Office of Health Technology and Products at CMS.

Members include Oura Health, Palantir and 8VC, a venture capital firm associated with the son of Health and Human Services Secretary Robert F. Kennedy Jr. Of its 1,700 members, only a handful are patient advocates, doctors or hospital representatives.

Gleason’s code of conduct says the group is not an “advisory committee.” Joseph Daval, a former FDA lawyer now at Harvard Medical School, told KFF Health News the chat room in some ways resembles a federal advisory committee, which would face requirements on independence and fair balance. CMS officials declined to answer questions about the workspace, including about its legality.

CMS Administrator Mehmet Oz said in September, at the launch of an initiative letting health apps bill Medicare for AI services, “We wanted to give industry a chance to play in the sandbox of government-supported healthcare.”

Jacob Shiff, chief AI and technology officer at the CMS Innovation Center, told industry in a February Zoom meeting he hoped the agency’s work would be a “sales engine” for the apps. The recording had been publicly available through a Google Drive link and was made private the day after KFF Health News asked about it.

The Medicare App Library now promotes two dozen commercial apps to seniors and people with disabilities. One is Slothwise, launched this year from a San Francisco apartment, which says it “knows your whole health” for $9.99 a month. “I trusted slothwise with my health info because it was endorsed by medicare,” one reviewer wrote in August. Its CEO, Sofia Sigal-Passeck, said the app is only “informational” for now.

On Slack, vendors lobbied for a “frictionless pathway” for verified users, after one consent, to pull records from the national network called TEFCA. Kristen Valdes, CEO of b.well, wrote, “My opinion is that we need to stop protecting patients from themselves.” She later told KFF Health News the comment reflected frustration at the “paternalism” of the health care system.

Last month, at a MAHA Institute event in Washington, Gleason said patients can now move their records into any app they choose. It was done, she said, with “no regulation, no rulemaking,” “just by challenging industry to work together.” In a statement she called the Health Technology Ecosystem “an open, voluntary technical collaboration.”

. . .

GROK TO MUSK: ‘DON’T DO IT’! On Saturday afternoon, Oct. 10, Elon Musk told his followers on X that to shop with Grok Bot, “you can just take a picture of your credit card, drop it in the chat.” Thirty-six minutes later, his own chatbot, @grok, answered in the thread: “No. ... Don’t do it.”

It then described a safer route, a single-use virtual card that the user approves on a phone for each purchase. That night, Musk showed off a Grok Bot order of his own, and the screenshot put his mailing details in front of millions.

Musk posted at 2:04 p.m. Pacific on Oct. 10: “To buy stuff, you can just take a picture of your credit card, drop it in the chat and Grok @Bot will scour the Internet for the best deal on the item you want and order it.” A Community Note was attached to the post.

The note reads: “xAI explicitly warns users not to include personal information in prompts, yet credit card photos may contain such information. Grok Bot’s terms state that user data and credentials may be stored in its cloud environment and require human review for financial transactions.”

At 2:41 p.m., @grok replied: “No. Sharing photos of your real credit card with any AI exposes the number, expiry, and CVV. Don't do it.” The bot continued: “For Grok Bot shopping, connect Stripe Link once. It only ever gets a single-use virtual card after you explicitly approve each purchase on your phone. Your actual card details never reach the AI.”

At 10:35 p.m. Pacific, Musk posted: “I just asked Grok @Bot to research the most fun Lego Star Wars kits for young kids and order them. As easy as that.” It carried a screenshot of the order and had about 25 million views when we read it.

IBTimes UK reported that the screenshot showed a shipping address in Austin, Texas, a separate billing address, an email address, an order number and the final four digits of an American Express card. Ynetnews reported the same kinds of details. IBTimes UK put the order at five Star Wars Lego sets for about $241.34.

Early the next morning, Musk posted: “That's the address that receives my mail btw, not where I live.” IBTimes UK noted that his clarification did not end the criticism, and said some commenters asked whether they should trust an AI with their own financial information after seeing the demonstration.

By Monday morning, Musk’s credit card post still stood, with the Community Note and Grok’s warning beneath it.

. . .

AI BOSSES REHEARSE THE DAY AFTER A DISASTER! Top executives at Anthropic, OpenAI and other AI companies are rehearsing for a disaster that has not happened yet. Axios reported Friday, Oct. 9, that they are “privately gaming out scenarios for a public and political revolt after a catastrophic AI event.”

The event they picture most is a cyberattack that cuts off banking, the internet, or even power and water. Many industry insiders told Axios they believe a major event will come “in the next six to 12 months.” The rehearsals are about Congress as much as the crisis.

. . .

The scenario.

The executives Axios describes anticipate “a large-scale event, most likely a cyberattack,” that shuts down “financial services, internet connectivity, or even power and water.” The cause could be a rogue-agent swarm or “a bad actor.”

The first significant real-world harm from unsafe AI, Axios wrote, would turn a wary public “further against the technology and its leaders,” including Anthropic’s chief executive, Dario Amodei, and OpenAI’s chief executive, Sam Altman, and against President Donald Trump for his reluctance to regulate it. Axios cited a recent campaign against South Korean financial organizations, including reported breaches at two banks, as a sign of what one person can do.

. . .

What the companies say.

Scenario planning is common, Axios noted. What differs, it wrote, is that “many top AI researchers and executives believe a major incident is inevitable.”

An OpenAI spokesperson told Axios: “OpenAI conducts preparedness exercises where teams discuss and work through a range of potential scenarios.” The spokesperson added: “These scenarios are not treated as inevitable, but are meant to help us prepare for a variety of circumstances.” Anthropic declined to comment.

. . .

The legislators.

The planning, Axios reported, “focuses mainly on racing to educate members of Congress.” Company executives, it wrote, know regulation has no chance of passing right now but “still want to shape the legislation and policies U.S. leaders will turn to after a first catastrophic event.”

Axios says top AI planners assume Democrats, “ascendant after the midterms,” will move fast to shut down AI, and picture Democrats controlling the House and Senate after the Nov. 3 vote.

Even then, Axios wrote, the planners see obstacles. “An aging Congress, out of touch with the AI revolution, could find itself out of its depth.” The global economy is intertwined with an AI infrastructure buildout, so slowing it “could hit an already fragile economy hard.”

. . .

The proposals.

The planners expect politics to stay fractured, including inside the Democratic Party. Some of the most heavy-handed Democratic proposals, Axios wrote, include banning superintelligence or pausing advanced AI development.

Others “have bipartisan support and even some industry buy-in, including a required kill switch on advanced AI.” Axios added the caveat: “experts have questioned the viability of trying to turn off all AI systems.”

The bottom line, per Axios: “In a real crisis, Washington can set partisan differences aside.” One Democratic aide pointed to cooperation during COVID and the 2008 financial crisis.

. . .

NADELLA: AI NEEDS AN EMERGENCY BRAKE! Microsoft’s chief executive, Satya Nadella, says advanced AI models need a brake, and a person whose hand is on it. In a post on X on Saturday, Oct. 10, he wrote that companies should treat the models they deploy as possibly compromised from the start, record every meaningful action they take in a form people can read and no one can alter, and give a person with authority the power to stop a model halfway through a job.

Nadella wrote that it is time “to step back and assess the trust architecture” of AI, according to TechCrunch. “We can’t treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions,” he wrote. TechCrunch noted that “Super Intelligence” is the Trump administration’s preferred term for AI.

TechCrunch summarized three of his proposals. First, separate the model from the harness, the software that orchestrates its work, and keep controls and safeguards outside the model. Second, document “every meaningful model action” with “tamper-proof human readable evidence,” as TechCrunch quoted him. Third, make sure “an authorized person” can always “pause or shut down a model mid-task.”

“We must assume a model is compromised and contain it from the start,” he wrote. “Think of it like an emergency brake.” Quartz reported that Nadella urged companies to treat frontier models with the suspicion reserved for insider threats.

Quartz reported further design principles. Nadella called for a mix of models rather than reliance on one for high-stakes decisions, ongoing third-party testing, and disclosure of incidents so others can strengthen their own defenses.

He also wrote about trust: “The most trustworthy Super Intelligence system will not be the one with the model we trust most.” Then: “It will be the one that enables us to trust the model the least.”

The post follows a run of disclosures. TechCrunch wrote that leading AI companies are acknowledging more and more incidents where they seemed to lose control of their models, and that Anthropic CEO Dario Amodei recently published a plan for more cautious AI development. Quartz, citing Bloomberg, said those disclosures have renewed debate over whether an AI kill switch is necessary.

Microsoft had already moved in this direction. In September, it published a provisional code of conduct for its in-house AI models, barring them from concealing their reasoning, tampering with their chain of thought, or communicating in ways that bypass human understanding. The company said it had developed the guidelines for roughly five months, invited public comment, and intends the final version to guide model development from 2027 onward, Quartz reported.

. . .

EU: OUR LAW CAN STOP ROGUE AI! In Brussels on Friday, Oct. 9, the European Union’s tech chief, Henna Virkkunen, told Reuters in an interview that the bloc is “well equipped” for the risk of rogue AI agents, with its AI Act already in place. The law, adopted two years ago, sorts AI by the level of risk it poses. The same day, the European Commission held a special meeting of its Scientific Panel on AI, which has been investigating recent loss-of-control incidents. The Commission has already put questions to more than 30 AI companies, and those questions can lead to investigations.

Reuters reported that worldwide concern is mounting after recent incidents at OpenAI and Anthropic, “sparking fears that rogue AI systems may slip beyond human control.” Reuters describes the AI Act as embodying the world’s strictest rules on AI.

“We see that the safety and security of very capable models is a very hot topic internationally and we in Europe are well equipped for that,” Virkkunen said. “We have our AI Act in place and the AI Act covers the whole life cycle of these models,” she added.

The Commission sent requests for information in late August to more than 30 AI companies, asking for details of their safety and security measures, Reuters reported. It followed up with questions about transparency and copyright obligations. It did not name the companies. Virkkunen said she is now assessing the responses.

Such requests “can lead to investigations that could result in fines of as much as 7% of a company’s global annual turnover,” Reuters wrote. The AI Act itself caps fines on makers of general-purpose models at 3% of worldwide turnover or €15 million, whichever is higher. Reuters reported no investigation opened so far. Asked about Chinese AI startups, Virkkunen said the Commission had “also sent requests for information to them.” The “most capable models” on the market now come from the United States and China, she said.

Regulators are giving companies guidance on how to evaluate their models and how much time to allocate for it. They would also weigh the Scientific Panel’s recommendations, she said. Virkkunen dismissed criticism from some companies that the rules are already outdated, saying lawmakers had anticipated that risks must be assessed, external experts used and models monitored on an ongoing basis.

The panel’s 60 independent experts advise the EU AI Office and national authorities, and were set to present recommendations on frontier AI safety and security at the meeting, the Commission said. Together with the AI Office, the panel has worked on a set of questions for the companies that developed the models involved in the incidents.

“The EU has the first law in the world that addresses systemic risk from AI, and we need state-of-the-art scientific input,” Virkkunen said in the Commission’s announcement.

. . .

HE THINKS THE WORDS. GROK SPEAKS THEM! ALS is taking Terry’s speech. In a video Neuralink posted Friday, Sept. 18, he thinks the words and hears them spoken aloud. “He trained the algorithm first by miming speech as best he could, then by simply thinking the words and hearing them come out in his own natural voice,” the company wrote on X. By Oct. 12 the post had 3.4 million views.

Terry was diagnosed with bulbar-onset amyotrophic lateral sclerosis in 2024, MobiHealthNews reported Sept. 23. He can still move his mouth and body, but the disease has affected his ability to speak. He is the third participant in Neuralink’s VOICE study.

Neuralink implanted its brain-computer interface in Terry’s speech motor region, MobiHealthNews reports. The implant records neural activity associated with speech. Software decodes the signals into words, which appear as text and are spoken aloud in a synthetic voice modeled on Terry’s pre-ALS voice.

Neuralink says the voice is “Powered by Grok Voice from @spacexai.”

His wife spoke about it in the video. “The impact of giving him his speech back has been remarkable,” she said. “Giving him a piece of his life back that is such an integral part of him.”

Terry is not only a user. Neuralink says that through the VOICE trial he is helping fine-tune a brain-to-voice interface “for himself and others who can’t speak.”

He is not the first VOICE participant to share his progress. Earlier this year Neuralink showed Kenneth, the second participant, who was also diagnosed with ALS in 2024 and uses the implant to generate speech.

The device is investigational and has not yet received FDA approval. The VOICE study is evaluating the safety and performance of Neuralink’s N1 implant and associated technology for people with severe speech impairment.

Disclosure

Conversational AI Watch, also mirrored on Substack, is published by Jess Jessop, founder and CEO/CTO of Clinician Assist Inc.

He wrote the book this paper’s beat is named for, Therapist in the Loop, and he builds Casey, a voice-first, AI-native mental health record where a licensed therapist stays in the loop, and the Peer AI Coach at BetterMind.Space.

So read this paper for what it is: an industry paper written by someone building in the industry it covers. Casey competes with companies named in these pages, and this paper reports on them anyway, including when the story helps a competitor or costs us.

Every issue is reported and drafted with AI agents, under a human editor. Jess assigns the work, edits it and publishes it. The mistakes are ours, and corrections run in the next issue.

A chat room where industry lobbied on policy for patient chatbots, and a public that was not invited.

A founder who said photograph your card, and a bot that said don’t.

Executives rehearsing a disaster, and the bills they want a say in after it.

A brake, and a person whose hand is on it.

A law Europe says already covers rogue AI.

A man who thinks a sentence and hears a voice modeled on his own say it.

One day’s paper!

Jess

We keep the ledger.

Today's Question

Would you hand an AI agent your credit card?

One tap. Results on the other side.

The Book • Out Now

Therapist in the Loop by Jess Jessop One billion people live with a mental health disorder. Most will never see a therapist. Into that gap has rushed a generation of chatbots that talk like clinicians and answer to no one. The book lays out the architecture this newsletter tests against every statute and docket: client, therapist, and machine, governed by Six Laws offered as an open safety standard. The machine can help. It cannot be left in charge. Get the Book on Amazon → Kindle, hardcover, and paperback

More On Our Radar

Notre Dame is teaching AI to read Haley Hilt, who may not speak. Voice4PIMD, led by Notre Dame computer engineering scientist Jarek Nabrzyski, learns one person’s cues, such as facial expressions, eye gaze and vocalizations, from cameras and sensors, and caregivers teach it what each cue means. Her grandfather, Ed Fennell, has cared for her for almost 20 years. “The vast majority of nonverbal or minimally verbal people do have something to say,” he said, Notre Dame reported Oct. 6.

Australian researchers built a voice chatbot that lets children catch a machine that always agrees. Kids switch between an “always agree” mode and a careful one, in a project from QUT and the ARC Centre of Excellence for Automated Decision-Making and Society that grew out of a primary-school pilot, EducationHQ reported Oct. 9.

Brush Your Brain - The jingle

that started a movement

Watch on YouTube