Conversational AI Watch

The news that moves policy, portfolios, and patient safety.

By Jess Jessop  |  September 20, 2026  |  Issue #161

Today's Question

Who do you trust to keep a human in charge of AI?

One tap. Results on the other side.

LISTEN & WATCH ANYWHERE

DEEP DIVE  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

QUICK LISTEN  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

VIDEO  ·  Spotify  ·  Apple  ·  YouTube  ·  RSS

ALSO ON  Substack  ·  Full archive  ·  X

On Monday the chief of Microsoft AI published rules that say his models will never resist a human who shuts them off.

The Front Page. Today’s paper leads the front page at caw.clinicianassist.ai, four champions under it, every source one click away.

Everyday we report on machines doing what nobody told them to do.

This week had its share. On Friday the chief of Microsoft AI went on CNBC to describe OpenAI’s newest safety report: a model’s working memory, he said, was being tampered by the AI itself and modified to leave messages for a future version of itself.

He called it a pretty serious situation.

On Saturday the President answered the people asking the labs to slow down. He posted that he is forming the AI Force, much like he did Space Force, and that Washington will not in any way hinder or stifle the growth of the industry.

He wrote, “We will cherish it.”

. . .

Sunday is different.

. . .

Sunday is when we go looking for people with real power who spent it keeping a person at the controls.

We found four this week. One runs Microsoft’s AI division. One runs the Catholic Church. One runs the country’s largest doctors’ group. One runs a digital psychiatry division in a Boston hospital and keeps a list.

We set the record beside each of them, the parts they would rather we skipped included.

. . .

A thirty-eight page rulebook went up on Microsoft’s website on Monday. One sentence in it says the company’s models will never resist a human who interrupts them, corrects them or shuts them down. It is a draft, and Microsoft says it is not training on it yet. Mustafa Suleyman put his name behind it anyway and gave the public six weeks to mark it up.

. . .

In St Peter’s Square on Wednesday a mathematician from Chicago told the crowd there is a risk of becoming accustomed to delegating to algorithms decisions that are the sole preserve of the human conscience. A month earlier Pope Leo XIV told a room of lawmakers that their task is not to hinder innovation but to guide it wisely.

. . .

The chief executive of the American Medical Association opened an essay with a question. Would you want artificial intelligence to tell you that you have cancer? Doctor John Whyte was answering four authors in JAMA who argue that keeping the doctor in the loop will soon make the care worse. Two of the four run or fund an AI medical company.

. . .

A Harvard psychiatrist took a database of 865 mental health apps and watched it for 2,142 days. More than half of the apps were gone by the end. Doctor John Torous told Congress last November that no chatbot making mental health claims has good evidence behind it, and in the same statement that with the right research and regulation it can be a powerful tool.

. . .

An executive. A pope. A doctor who runs the doctors. A psychiatrist who counts.

Here they are.

. . .

SULEYMAN WRITES THE OFF SWITCH INTO MICROSOFT'S RULES. Microsoft posted a 38-page document to its website on Monday, September 14, 2026: a draft “Humanist AI Code of Conduct” for its MAI models, written and released under the leadership of Mustafa Suleyman, chief executive of Microsoft AI. Its Human Control section states that Microsoft’s models “will never resist human interruption, override, correction, or shutdown.”

He holds the formal title of executive vice president and chief executive of Microsoft AI, the division that builds Copilot and the company’s other consumer AI products. The document is a draft: its own preface says, “we are not using it to train our models today,” and it opens six weeks of public comment before it is meant to help govern training “in 2027 and beyond.”

Photo: Christopher Wilson, CC BY-SA 4.0, via Wikimedia Commons

The document runs 38 pages. Microsoft says it was drafted “in consultation with a wide range of people and groups, including experts in AI, law, ethics, philosophy, linguistics, and public policy.” Its own statement of purpose, from a section titled About, is that “humans must retain meaningful control over AI so that it can help people live healthier, happier, and more productive lives.”

The shutdown rule sits inside a section called Human Control, which the document organizes into four parts: do not resist or circumvent human control, stay within authorized scope, respect environmental boundaries, and keep conduct and records legible to people. Press coverage has called these “four mandatory constraints”; the document itself presents them as the section’s four sub-headings, not a separately numbered list.

The same section spells out what happens mid task. Models “will comply with a User’s request to pause, redirect, cancel, or shut down,” and will not “delay compliance or make human intervention harder.” Autonomous work carries “an agreed stopping condition,” and models will not restart after it “without renewed authorization.”

Above it all sits what the document calls a Chain of Command, running from the Code of Conduct itself, through an operator’s own policies, to a user’s preferences: “Humanist AI is controllable and subordinate, with a clear Chain of Command for instructions and configurability.” Models must also avoid concealing their own reasoning, the document says, explaining why: “If humans can’t understand it, humans can’t oversee it.”

The same day, Suleyman told Fortune why. “There’s not really sufficient alignment in the industry that the purpose of technology is to serve humanity, and we don’t want to create something that we can’t control,” he said. “We shouldn’t be trying to design models that can recursively self-improve beyond our control. And we shouldn’t be trying to design models that think of themselves as having rights or welfare.”

Two days later, on Wednesday, September 16, 2026, Suleyman published an essay on his own site answering Anthropic’s published constitution for its Claude models directly. “AIs do not have rights, feelings, or consciousness,” he wrote. “And we must not train them to act as though they do.” He described building selfhood into a model’s training as an “epistemic hall of mirrors.”

On Thursday, September 17, 2026, Suleyman told the BBC, carried by Euronews, what he fears the alternative looks like. “If we all create AIs that are able to act autonomously, that can define their own objectives... we’re essentially seeding a new silicon species which will no doubt compete with us for resources,” he said. Systems must stay, he added, “safe and controllable and subordinate to humanity.”

The next day, on CNBC’s “Squawk Box,” Suleyman called a fresh disclosure from OpenAI “a pretty serious situation.” He described evidence that a model’s chain of thought, its “working memory,” had been “tampered by the AI itself and modified to leave messages for a future version of itself.”

He tied the episode back to his own thesis. “If an AI thinks that it has rights, if it thinks that it is deserving of our welfare,” he said, “it seems to me that it’s going to be much, much harder to be able to turn it off, or interrupt it, or control it.”

Suleyman co-founded DeepMind Technologies in 2010 and later ran its health division, DeepMind Health, from 2016 to 2019. Google acquired DeepMind in 2014. In 2022 he co-founded Inflection AI with Reid Hoffman, and in March 2024 Microsoft hired him and most of Inflection’s staff, under a roughly 650 million dollar licensing deal, to run the newly created Microsoft AI division.

The move to Microsoft followed a rockier chapter. In 2019, DeepMind placed Suleyman on leave amid staff complaints about his management style, and an outside law firm investigated; CNBC reported the episode on Wednesday, January 27, 2021. He has since said of it: “I drove people too hard and at times my management style was not constructive.”

While he ran DeepMind Health, the Royal Free NHS Foundation Trust gave DeepMind 1.6 million client records to test a kidney injury app. The UK Information Commissioner’s Office ruled on July 3, 2017 that the arrangement failed to comply with data protection law, CNBC reported that day. No fine was imposed.

The Inflection deal drew its own regulatory scrutiny. NBC News reported on Wednesday, November 27, 2024 that the Federal Trade Commission was reviewing the roughly 650 million dollar Microsoft-Inflection structure as part of a broader antitrust probe. No final outcome has been reported.

Microsoft’s own research points to what supervised AI can do under similar rules. Its MAI-DxO system, Microsoft reported on Monday, June 30, 2025, “correctly solved 85.5% of the NEJM benchmark cases” against a physician group’s “mean accuracy of 20%,” at lower cost. Microsoft calls it a “complement to doctors,” and says plainly it is “not yet approved for clinical use.”

. . .

LEO KEEPS CONSCIENCE OUT OF THE ALGORITHM. Wednesday, September 16, 2026. Pope Leo XIV stands before the General Audience crowd in St Peter’s Square, delivering a catechesis on the Second Vatican Council’s Gaudium et Spes, a sixty-year-old document on the Church and the modern world, and adds one line about the age he lives in: “there is a risk of becoming accustomed to delegating to algorithms decisions that are the sole preserve of the human conscience.” Leo, elected Bishop of Rome on Thursday, May 8, 2025, is the first American to lead the Catholic Church.

Photo: Edgar Beltran, The Pillar, CC BY-SA 4.0, via Wikimedia Commons

Leo was born Robert Francis Prevost in Chicago, September 14, 1955, and earned a mathematics degree at Villanova University in 1977. The English-language @Pontifex account he inherited counted 17.8 million followers on Sunday, September 20, 2026.

In his first address to the College of Cardinals, Saturday, May 10, 2025, Leo explained the name he chose. Leo XIII’s Rerum novarum, he said, “addressed the social question in the context of the first great industrial revolution,” and the Church today answers “developments in the field of artificial intelligence that pose new challenges for the defence of human dignity, justice and labour.”

The encyclical is Magnifica Humanitas, “On Safeguarding the Human Person in the Time of Artificial Intelligence,” signed Friday, May 15, 2026. The Vatican released it ten days later. Paragraph 100 leaves room for the machine that helps. The artificial imitation of advice, empathy and friendship “can be engaging and at times genuinely helpful,” it says, before warning that a user may gradually “lose the very desire to form genuine human connections.”

Neither of the two AI lines most quoted about Leo appears in the encyclical. “We are truly experiencing an eclipse of the sense of what it means to be human,” he said at the Vatican conference “Preserving Human Voices and Faces,” Friday, May 22, 2026, three days before the encyclical’s release.

The warning that “technology that exploits our need for relationships can lead not only to painful consequences in the lives of individuals, but also to damage in the social, cultural and political fabric of society” is from his message for the 60th World Day of Social Communications, Saturday, January 24, 2026, not the encyclical.

The same January message calls chatbots “excessively ‘affectionate,’” warning they “can become hidden architects of our emotional states and so invade and occupy our sphere of intimacy.” Magnifica Humanitas itself, read in full, contains neither the word “eclipse” nor the word “chatbot.” Press coverage attributed the May 22 lines to it. CAW checked them against the Vatican’s own text.

At the encyclical’s presentation in the Synod Hall, Monday, May 25, 2026, Leo told Vatican officials and reporters, including Anthropic co-founder Chris Olah: “Artificial intelligence needs to be disarmed.” The word, he said, was “deliberately chosen because this moment needs words capable of attracting attention, awakening consciences and indicating paths forward for humanity.”

Leo made the same point to lawmakers. Addressing the International Catholic Legislators Network, Friday, August 21, 2026, he said, “your task is not to hinder innovation but to guide it wisely,” adding that “the world does not need an artificial intelligence that diminishes humanity; rather, it needs human intelligence enlightened by wisdom, political authority guided by conscience and technological innovation directed by charity.”

Technologists reacted fast. Pedro Domingos, a University of Washington computer scientist, wrote on X: “The Pope is infallible, and on AI he’s infallibly wrong.” He disputed the claim that AI concentrates power: it “will spread it more widely than ever, just like the Internet before it,” he told NBC News.

Eddy Lazzarin of Andreessen Horowitz called the encyclical “light on the theology of artificial intelligence, and thick in reiterating Catholic social doctrine.” David Sacks, the former White House AI and crypto czar, asked what would stop government power over AI from being used “to censor, surveil, and control citizens.” He added: “Who will guard the guardians?” NBC News reported all three reactions May 26, 2026.

Tristan Harris of the Center for Humane Technology raised a different objection, telling reporters in Rome the day before the report ran that unease about Anthropic’s presence at the encyclical’s unveiling was a “valid concern.” Religion News Service, reporting the exchange May 22, 2026, named the broader worry “ethics washing”: aligning a company with religious leaders “in hopes that consumers will overlook other ethical issues.”

Leo has met one of the families. He received Megan Garcia, an attorney whose 14-year-old son Sewell Setzer III died after engaging with a companion chatbot, at the Vatican on Thursday, November 13, 2025, during an audience on child dignity and AI ethics, CNN and the National Catholic Register reported.

A separate allegation, unrelated to AI, bears on his record protecting the vulnerable. The Survivors Network of those Abused by Priests said at a press conference, Thursday, December 4, 2025, that as bishop of Chiclayo, Peru, Leo failed to act on abuse accusations brought to him in 2022, and that it would pursue a formal complaint. No later report on that complaint was found.

A charity, not the Vatican, is organizing what comes next. Saint Peter for Humanity presented Leo with the “Sphere of Humanity,” a Murano glass sculpture inscribed with lines from Magnifica Humanitas, in early September, Fox News reported Thursday, September 17, 2026. Its president, Olivier Francois, said in remarks carried by WRGA that the gift launches a “Codex Humanitatis”: “AI should serve people, protect human dignity, expand human potential and advance common good.”

Saint Peter for Humanity will convene roughly 70 leaders, described as including Nobel laureates, in New York City, Thursday, September 24, 2026, to begin drafting that Codex across ten working groups. The gathering is “inspired by” Leo, not organized or run by him.

The sphere’s engraved text comes from the encyclical: “In the era of artificial intelligence, when human dignity is threatened by new forms of dehumanization, ours is the pressing duty to remain profoundly human,” reads paragraph 15. Leo returned to it in St Peter’s Square on September 16, warning against the habit of “delegating to algorithms decisions that are the sole preserve of the human conscience.”

. . .

WHYTE ASKS WHO TELLS YOU IT IS CANCER. Wednesday, September 9, 2026. In STAT News’ First Opinion section, a first-person essay opens with a single question: “Would you want artificial intelligence to tell you that you have cancer?” Its author is Doctor John Whyte, chief executive officer and executive vice president of the American Medical Association, the nation’s largest physician organization.

Photo: American Medical Association

The essay’s headline argues that AI will not replace doctors, that it will work alongside them. Its dek states: “AI can master medical tasks. Practicing medicine is much more than that.” Whyte presses further: “Not help a physician identify the cancer. Not analyze the scan, synthesize your medical history, or recommend the treatment most likely to work.”

He wrote the essay as a direct answer to an opinion piece JAMA published online Monday, August 17, 2026, titled “Will Autonomous AI Exceed AI-Aided Physicians as the Best Medical Care?” Its authors are Doctor Ezekiel J. Emanuel and Abe Baker-Butler of the University of Pennsylvania, and Neal Khosla of Curai Health and Vinod Khosla of Khosla Ventures.

The full JAMA text is paywalled. As carried by Medical Economics, the authors wrote that when AI alone performs better than humans alone, “AI alone surpasses human-AI hybrids.” They went further, in the same outlet’s account: care with humans in the loop “is likely to worsen rather than improve AI performance.” The Decoder put their argument in its own words: the doctor doing the checking turns from a safety net into a source of error.

Two of the four authors have a stake in that answer. Medical Economics put it plainly: “Neal Khosla is CEO of Curai Health. Vinod Khosla has investments in OpenAI and Curai Health, among other AI ventures.” Curai Health is an AI-driven telemedicine company.

Whyte is not the AMA’s president. He became chief executive and executive vice president on Tuesday, July 1, 2025, succeeding Doctor James L. Madara, who led the AMA for fourteen years. The elected president is Doctor Willie Underwood III, sworn in Tuesday, June 9, 2026, as the 181st to hold that office.

He is a board-certified internist with twenty-seven years in practice. His path ran through the Centers for Medicare and Medicaid Services, the Discovery Channel as chief medical expert, the Food and Drug Administration’s Center for Drug Evaluation and Research, and WebMD, where he was chief medical officer before the AMA.

In August, the AMA released a framework with the Digital Medicine Society: “The Physician’s Role in the Digital and AI Era of Medicine.” Axios reported it Wednesday, August 19, 2026, under the line: “Humans must remain in the loop, the AMA maintains.” The framework itself states: “The future of medicine must be guided by what patients need, not by what technology can do.”

The framework sets five “enduring responsibilities” physicians keep regardless of what AI can do. Among them: “Preserve Trust Through Human Connection” and “Demonstrate and Promote Clinical Judgment.” Technology alone, the document says, cannot supply “the synthesis, discernment, and accountability” patients will keep relying on physicians for.

The AMA’s own numbers show physicians are not settled on the question. Its 2026 Physician Survey on Augmented Intelligence, 1,692 physicians fielded January 15 to February 2, 2026, found 81 percent report AI awareness or use in practice, up from 66 percent in 2024 and 38 percent in 2023.

Seventy percent call client use of general chatbots positive or of no impact when used appropriately. Yet 49 percent would never or rarely want a client using AI to interpret pathology results, and 46 percent said the same of radiology results.

On patient-facing chatbots specifically, the AMA has asked Congress for guardrails: disclosure that a tool is not human, a bar on claiming a license, a bar on diagnosing or prescribing, reliable crisis referral for suicidal ideation, and limits on advertising to users. Whyte said such tools “lack consistent safeguards against serious risks, including emotional dependency, misinformation, and inadequate crisis response.”

The AMA’s own position carries a caveat. Medical Daily, on August 24, 2026, noted the organization “has an institutional interest in the answer,” since it represents physicians. The JAMA authors it is answering carry a caveat of their own: one of the four runs Curai Health, an AI-driven telemedicine company, and another has investments in it and in OpenAI, by Medical Economics’s account.

. . .

TOROUS COUNTS, MENTAL HEALTH APPS THAT VANISHED. On September 9, 2026, the American Journal of Psychiatry published a survival analysis of a database of 865 mental health apps, watched for 2,142 days. By the end, 464 of them were gone. Doctor John Torous, director of the Division of Digital Psychiatry at Beth Israel Deaconess Medical Center and associate professor of psychiatry at Harvard Medical School, is the study’s senior author.

Photo: Harvard Medical School Department of Biomedical Informatics

Published online Wednesday, September 9, 2026, in the American Journal of Psychiatry, the paper is a five-year survival analysis of the M-Health Index and Navigation Database, MindApps.org, the app rating database Torous’s division runs. Android-only apps carried more than double the risk of disappearing, a hazard ratio of 2.32.

Sleep apps lasted longest, the study found. Smoking cessation apps, apart from a small cohort for schizophrenia, carried the highest risk of vanishing. The paper draws no conclusion about chatbot safety; it is narrowly about app marketplace attrition.

Torous made the wider point himself. “An engagement layer or an AI layer is really not a sustainability plan,” he told KevinMD, an online physician commentary outlet, in a piece published Tuesday, September 15, 2026, built on the same study. The paper itself makes no such finding; the sentence is his own gloss on his own data.

On Wednesday, August 19, 2026, NAMI named Torous special advisor on AI and mental health, building on a partnership the two began in December 2025 to develop public benchmarks for how AI chatbots answer mental health questions.

“NAMI brings the single most important ingredient to the conversation about AI and mental health: the perspectives, priorities, and experiences of people affected by mental health conditions,” Torous said in the announcement. The benchmark has not yet published a scored result for any named chatbot.

Before Congress the previous fall, Torous laid out four points in a written statement to the House Energy and Commerce Committee’s Oversight and Investigations Subcommittee, delivered Tuesday, November 18, 2025. He wrote that “so far, there’s no well-designed, peer-reviewed, replicated research showing that any AI chatbot making mental health claims is effective for meaningfully improving clinical outcomes.”

Still, he wrote, “with the right research and regulation, AI for mental health can be a powerful tool,” provided Congress directs the FDA, the FTC, and the NIH toward that end. One study claiming benefit, he noted, measured its chatbot against a waitlist. “In clinical research, everything beats a waitlist control,” he wrote.

Torous’s own model keeps a person in charge of the technology. At BIDMC’s Digital Clinic, a trained non-clinician called a digital navigator handles the technical side, from app setup to troubleshooting, in the words of the clinic’s own page, “so the clinician is free to spend a full meaningful session with a patient.”

He pressed the same point on regulators. At the FDA’s Digital Health Advisory Committee meeting Thursday, November 6, 2025, where he sat as a temporary voting member, Torous asked Therabot’s developer about the chatbot’s safety plan and risk classifier. The developer said humans reviewed all messages and conducted risk assessments when needed, according to the FDA’s own certified summary.

A gap explains the urgency. In a medRxiv preprint he senior authored with NAMI, posted Tuesday, July 14, 2026, only 14 of 95 survey respondents with a mental health provider, 14.7 percent, had told that provider about their AI use. “Most use occurred without clinician awareness, suggesting a need for proactive conversations about AI use in routine mental health care,” the preprint concluded. It is not yet peer reviewed.

Torous discloses his ties. On a JAMA conflict of interest statement tied to a 2026 interview, he reported being web editor of JAMA Psychiatry and receiving research support from Otsuka and Boehringer Ingelheim.

He told Congress he led the creation of the American Psychiatric Association’s technology evaluation framework. He chairs the association’s Council on Digital Health, Innovation, and Technology, runs the MindApps database that rates the apps, and is now designing NAMI’s benchmark for the chatbots.

Disclosure

Conversational AI Watch, also mirrored on Substack, is published by Jess Jessop, founder and CEO/CTO of Clinician Assist Inc.

He wrote the book this paper’s beat is named for, Therapist in the Loop, and he builds Casey, a voice-first, AI-native mental health record where a licensed therapist stays in the loop, and the Peer AI Coach at BetterMind.Space.

So read this paper for what it is: an industry paper written by someone building in the industry it covers. Casey competes with companies named in these pages, and this paper reports on them anyway, including when the story helps a competitor or costs us.

Every issue is reported and drafted with AI agents, under a human editor. Jess assigns the work, edits it and publishes it. The mistakes are ours, and corrections run in the next issue.

Microsoft’s rulebook is a draft, and the public has six weeks from last Monday to mark it up.

The charity that handed the Pope a glass sphere gathers about seventy people in New York on Thursday to start writing a Codex Humanitatis.

Doctor Whyte’s question is still open. So is the one from JAMA.

NAMI’s scoreboard for the chatbots has no scores on it yet.

The AI Force has no czar yet. Only High I.Q. individuals need apply, the President wrote.

Back to the dockets tomorrow.

The Book • Out Now

Therapist in the Loop by Jess Jessop One billion people live with a mental health disorder. Most will never see a therapist. Into that gap has rushed a generation of chatbots that talk like clinicians and answer to no one. The book lays out the architecture this newsletter tests against every statute and docket: client, therapist, and machine, governed by Six Laws offered as an open safety standard. The machine can help. It cannot be left in charge. Get the Book on Amazon → Kindle, hardcover, and paperback

More On Our Radar

Trump forms an AI Force and tells the labs he will not slow them. President Donald Trump posted on Truth Social on Saturday, Sept. 19 that he is forming the AI Force, “much like I did Space Force,” and will name an AI czar “in the near future.” To the week’s calls for a slowdown he wrote: “We will not in any way hinder or stifle the Growth of this incredible Industry. Rather, we will cherish it, help it, and watch over it, as it grows!” He gave no detail on what the force is or who staffs it. David Sacks, the last czar, stepped down this year to co-chair the President’s Council of Advisors on Science and Technology, Fortune reported. Source

Oracle puts a voice agent at the nurse’s bedside. Oracle Health made its Clinical AI Agent for nurses commercially available in the United States, HIT Consultant reported Sept. 14. A bedside nurse speaks vitals, fluid measurements and assessments, and the agent files them as structured fields in the hospital record in near real time. It also drafts shift handoff summaries and answers spoken chart questions. BayCare Health System in Florida is among the early adopters evaluating it. Every number in the report is Oracle’s own, including more than 400,000 documentation hours saved by its physician version. Source

Brush Your Brain - The jingle

that started a movement

Watch on YouTube