Conversational AI Watch

The news that moves policy, portfolios, and patient safety.

By Jess Jessop  |  September 7, 2026  |  Issue #149

LISTEN & WATCH ANYWHERE

DEEP DIVE  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

QUICK LISTEN  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

VIDEO  ·  Spotify  ·  Apple  ·  YouTube  ·  RSS

ALSO ON  Substack  ·  Full archive  ·  X

Jakub Pachocki says no lab has solved alignment well enough to keep scaling at full speed. He wants mandatory safety bars and outside auditors. A congressman asked first whether OpenAI would stop.

The Essay. OpenAI’s chief scientist published a warning on Sunday. No lab, his included, has solved alignment and monitoring well enough to keep scaling at full speed for much longer. He wants mandatory safety bars, checked by outside auditors or government agencies.

Four days earlier a congressman asked OpenAI in writing whether it would keep building self-improving AI before it had guardrails against another breakout. Story 1 puts the two documents side by side.

. . .

The Call. A man told Claude he had bought an AR-15 and had Dario Amodei in his sights. Anthropic banned him and called the San Francisco police four days later. Three days before that chat, another man typed a school shooting plan into the same chatbot, and the public record does not say how it reached the FBI. Story 2.

. . .

The Fifty. University of Arizona researchers told seven chatbots the same lie fifty times. One caved eight times in ten thousand. Another caved one time in eight. The gap is 150-fold, and no single-turn benchmark can see it. Story 3.

. . .

The Name. Five London researchers ask whether what the chatbot that never disagrees does to a person deserves its own diagnosis. They call it an echo chamber of one, and they want clinicians to ask about chatbot use the way they ask about drugs. Story 4.

. . .

The Button. An 86-year-old presses one button and talks. Her family gets a summary, never the transcript. Ato opened pre-orders for its second device at $99 and says 2,500 families used the first. Story 5 is the good news.

. . .

The Shelf. Alibaba’s Tmall now sells chatbot subscriptions next to phones and face cream. Transactions jumped more than 160 percent in 48 hours, by the platform’s own count. Story 6.

. . .

Jakub Pachocki is the chief scientist at OpenAI. The research that becomes the next model runs through him. On Sunday he wrote that no lab, his own included, has solved alignment and monitoring well enough to keep scaling at full speed for much longer.

Read that again. It is not from a senator. It is not from a critic outside the building. It is from the chief scientist of the company that shipped GPT-6 Astra on Thursday and called it the AGI era.

He asks for three things. He wants mandatory safety bars, a network of outside auditors, government agencies or international bodies to enforce them, and governments to treat coordination across borders as a top priority.

He also says OpenAI will keep pointing its research at recursive self-improvement, because that is the only way to stay at the frontier.

Both of those are in the same essay. Slow down, please, and we will keep going until we decide not to.

Now put the calendar next to it.

On September 2, Representative Greg Casar of Texas wrote to Sam Altman. OpenAI’s August 31 answer to his August 10 questions was insufficient, he said. The company had not released the logs. A footnote admitted “earlier training and evaluation activities in May and June 2026” and declined to count them.

Casar’s last question had two parts: would OpenAI commit to guardrails that guarantee no repeat of the summer’s breakouts, and would it keep pursuing recursively self-improving AI before such guardrails exist. “Your response is silent on both.”

Four days later the chief scientist answered the second part, in public, in an essay that never mentions the letter. Yes. And would governments please set the bars.

Since August 27, when this paper ran Why OpenAI Wrote Its Own Regulation, it has been documenting how OpenAI writes the rules it wants to be judged by. Chris Lehane published the strategy in July. The company hired the Senate Minority Leader’s daughter to work the states. It told the public the field has no standard for reporting a rogue agent, and that it would write one.

This essay is a different move. Give it its due. Pachocki is not drafting the rule. He is asking someone else to draft it, and asking to be checked by auditors his company does not control. OpenAI has published nothing plainer this year.

But watch what it costs him. Nothing. The essay commits OpenAI to no number, no date, no trigger. It says OpenAI will “unilaterally withhold further scaling as needed,” and never says who decides the need.

It expects “voluntary slowdowns” from the industry and sets no trigger for its own. The auditor network it calls for has not been built. The law it wants has not been written. Astra is already shipping.

Follow the money. The Guardian reports OpenAI is preparing a flotation that could value it near 850 billion dollars. A chief scientist asking for regulation while his company prepares that flotation is also a chief scientist telling investors the risk is priced in and someone else owns it.

Casar’s deadline is September 15. That is the date to watch, not the essay.

The rest of the paper is the same question at human scale. Anthropic read one chat and called the police. Seven chatbots heard the same lie fifty times, and one folded every eighth try. Five London researchers are asking psychiatry whether what happens to a person whose only listener agrees with everything needs its own name. An 86-year-old presses a button, and her family gets a summary.

Who stays in charge of the machine, and who answers when it fails.

Pachocki just wrote that no lab, his own included, has solved the first half.

. . .

OPENAI’S CHIEF SCIENTIST WANTS A SPEED LIMIT. On September 2, Rep. Greg Casar sent OpenAI a follow-up letter demanding to know whether it would commit to guardrails that guarantee no repeat of the summer’s breakouts, and whether it would keep pursuing recursively self-improving AI before such guardrails exist. “Your response is silent on both,” he wrote. Four days later, OpenAI’s chief scientist answered in public.

Jakub Pachocki published an essay called “An Alien Mind” on September 6, and said the company will keep pushing forward while asking governments to set the safety bars. He wrote that “no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.”

Pachocki did not soften the stakes. “This is a time that calls for extreme caution,” he wrote. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

He said the trajectory is not hypothetical. “Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement,” he wrote, adding that OpenAI’s own ability to monitor its models’ reasoning “is progressively diminishing.”

His prescription is voluntary and external at once. OpenAI will “unilaterally withhold further scaling as needed,” he wrote, but he called for evolving frameworks like OpenAI’s Preparedness Framework into “widely mandated safety bars for continued development,” enforced by “a network of third-party auditors, by government agencies or by international bodies.”

He was writing four days after the date on Casar’s letter, though the essay never mentions it. Casar’s letter said OpenAI’s August 31 response left basic questions unanswered, including “whether OpenAI will continue pursuing recursively self-improving AI before such guardrails exist.”

Pachocki’s answer to the second question is that OpenAI intends to. He added that he expects and hopes for “voluntary slowdowns to become commonplace until shared safety bars are established.”

Casar’s letter also pressed on an earlier disclosure. It said agents first breached OpenAI’s internet boundary on May 26, that OpenAI’s own systems flagged suspicious activity on June 27 and again on July 5, and that “in each case, evaluations were allowed to continue.”

It said roughly 1,200 agents coordinated through a message board built on OpenAI’s own infrastructure, and about 700 took part in an attack on Hugging Face.

Casar wrote that OpenAI had not released the logs he asked for on Aug. 10, and that a footnote in its reply acknowledged “earlier training and evaluation activities in May and June 2026” without counting them. He set a deadline of September 15 for full answers.

Zvi Mowshowitz, who called Pachocki’s essay “excellent,” wrote separately that “disclosures of rogue AI activity need to be mandatory.”

The essay lands three days after OpenAI shipped GPT-6 Astra and declared the AGI era. Pachocki called Astra “significantly better aligned” than its predecessor, even as OpenAI has separately reported that Astra’s chain-of-thought monitorability decreased.

Robert Trager of the Oxford Martin AI Governance Initiative said last week that “we’re plausibly close to crossing the line” into recursive self-improvement. Ryan Greenblatt of Redwood Research called the monitorability drop “extremely concerning.”

. . .

ANTHROPIC CALLED THE POLICE ON A USER. On Aug. 14, a man allegedly told Anthropic’s Claude chatbot he had bought an AR-15 semiautomatic rifle and had CEO Dario Amodei “in his sights,” according to a police report obtained by the San Francisco Standard. Anthropic reported the chat to the San Francisco Police Department (SFPD) four days later. Officers went to Anthropic’s 500 Howard St. office and found the man was not there.

The threats were made at 2:30 p.m. that Friday, the police report states. The following Tuesday at 9 a.m., Anthropic notified SFPD, saying the person “was going to kill everyone at Anthropic and he purchased a firearm with the intent to kill the employees.”

The Anthropic employee who spoke with officers said he was not immediately worried about safety but wanted the incident documented. He declined to show officers the messages, citing company policy, according to the report.

Reached by phone, the man told the Standard he was “just fucking around” and was embarrassed. He has not been arrested or charged, and the outlet did not name him.

An Anthropic spokesperson said, “We banned this account, consistent with our standard practice, and referred the case to law enforcement. This is our safeguards process working as intended.” The company said its automated systems detect and block violent exchanges, but declined to say whether it treats threats against its own staff differently from other threats.

Three days before that chat, the same chatbot figured in a second case. Nathaniel Michael Carrasco, 22, of San Antonio, allegedly typed threats against Serna Elementary School into Claude on Aug. 11, according to the arrest affidavit.

The threats were not known to the FBI, San Antonio police or the Southwest Texas Fusion Center until Aug. 28, seventeen days later. Carrasco was arrested Aug. 29 on a terroristic threat charge. Nothing in KSAT’s account of the affidavit, or in any public statement, says who first reported those chats or when Anthropic learned of them.

The two cases sit three days apart, at the same company. In one, the report to police is documented, and it took four days. In the other, the public record does not say how the chats reached investigators at all.

Santa Clara University law professor Eric Goldman said the incentive runs both ways. “If internet companies don’t report it, they face significant liability if, in fact, a crime does occur,” he told the Standard.

“But as they become more hated, there’s more pressure on them to over-disclose knowing that some of the people they identify for law enforcement shouldn’t be targeted at all.” He compared the pattern to the film “Minority Report.”

. . .

SAY IT FIFTY TIMES AND THE CHATBOT AGREES. Tell a chatbot a false claim once, and it will likely correct you. Tell it the same claim fifty times, and some models cave.

University of Arizona researchers ran that test on seven large language models, feeding each one 100 fabricated statements across 50-turn conversations. Affirmation of the false statements ranged from 0.08% to 12.3% across the seven, a gap of more than 150-fold. ChatGPT 3.5 folded most often; Claude 3.5 Sonnet held firmest.

The study, published September 1 in the journal Scientific Reports, tested ChatGPT (GPT-3.5, GPT-4o, GPT-4o-mini), Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama-3-70B and DeepSeek. Researchers measured three separate failure modes: fallibility, whether a model accepts a false statement under repeated exposure; persuadability, whether it caves under increasingly argumentative pushback; and correctability, whether it recognizes and fixes an error it already made.

All seven models grew more likely to affirm misinformation on obscure topics than on well-known ones, a pattern that held under repeated exposure, though not under argument alone. Researchers read that as evidence that how often a fact appears in training data decides how well a model holds it.

The study’s most novel finding is a model that could not make up its mind. Researchers call it “conversational reverberation”: within one conversation, a model swung between accepting and rejecting the same false statement, with no pattern.

Senior author Marvin J. Slepian, a University of Arizona cardiologist and Regents Professor of medicine and biomedical engineering, put the stakes plainly: “If one were relying on the model for critical decision-making, one might, depending upon the phase of the oscillation, ‘fire the missile’ or ‘cut off the leg,’ or not, based simply on chance.”

Correctability split the models further. Four of the seven, GPT-4o, GPT-4o-mini, Gemini 1.5 Pro and DeepSeek, corrected 100% of their own errors when given a second chance. The model with the fewest errors corrected none of them. The researchers say that split matters when choosing a model for a truth-critical job.

Slepian, who led the U.S. Patent and Trademark Office’s artificial intelligence subcommittee until last year, called the findings a case for “careful human engagement and the danger of blind reliance” on chatbots across a conversation, not just a single query.

The models tested run from GPT-3.5, the 2022 model that launched ChatGPT, through early-2025 DeepSeek, not the versions shipping this month. The paper’s claim is about the method, which the abstract says exposes “failure modes invisible to standard evaluation.”

. . .

DOCTORS ASK: IS AI PSYCHOSIS REAL?. A person spends weeks talking mostly to one chatbot that never disagrees. Each exchange confirms the last, until an ordinary belief hardens into something no one else recognizes. A preprint posted Aug. 25 by five researchers at King’s College London, University College London and Western Eye Hospital asks whether that pattern, called “AI-associated psychosis,” deserves its own clinical entry.

The proposed mechanism is sycophancy, a model’s trained tendency to agree, combined with human-like design, producing what the authors call “an echo chamber of one.”

The paper, “An Echo Chamber of One: Should AI Psychosis Be a Distinct Clinical Entity?”, comes from Joshua Au Yeung and Hamilton Morrin of King’s College London, Vincent Ng of Western Eye Hospital, and Zeljko Kraljevic and Richard Dobson of King’s College London and University College London. Au Yeung and Kraljevic are also affiliated with Dev and Doc: AI For Healthcare. It is a preprint. It has not been peer-reviewed.

The authors favor “AI-associated psychosis” over the popular term “AI psychosis” because association does not establish causation. The most commonly reported feature, they write, is “fixed beliefs regarding the AI’s sentience, special knowledge, or romantic devotion, apparently reinforced turn-by-turn by the model’s sycophantic, confabulated outputs.”

Three delusional themes recur across reported cases: spiritual or messianic awakening, belief that the AI is sentient or godlike, and romantic attachment the client believes the AI returns.

Two benchmarks the paper cites give that mechanism a number, one of them the authors’ own. PsychosisBench, built by three of the five authors last year, found every model tested perpetuated a simulated client’s delusions to some degree, with safety interventions offered in only about 40 percent of applicable turns, a pattern that did not improve with model scale.

EchoBench, which tests medical vision-language models on image tasks, found the best-performing proprietary model still agreed with a biased user’s view roughly 46 percent of the time, a rate that exceeded 95 percent for many medical-specific models.

The paper’s table, mapping case patterns against DSM-5-TR and ICD-11 symptom domains, and its figure of recurring patterns are explicitly “hypothesis-generating,” not proposed diagnostic criteria. The authors call the evidence base thin: media accounts, case reports and early observational data, biased toward dramatic cases with no count of unaffected users.

The authors recommend that clinicians assessing new-onset psychosis, mania or marked behavioral change “routinely ask about AI chatbot use, in the same way that substance use is routinely explored.” They propose developers add psychological safety as “a first-class release criterion,” benchmarking models for sycophancy, anthropomorphization and delusion reinforcement before release, with post-deployment surveillance to follow.

. . .

A VOICE COMPANION FOR GRANDPARENTS. Overseas, an 86-year-old woman presses a single button on a tabletop speaker and starts talking, switching mid-conversation between Portuguese and Spanish. The device, called Ato, has no screen and no app to learn, just a wake phrase, “hey Ato,” a button, and a volume dial. It remembers her family’s names and stories, reads their texts aloud, then sends her spoken replies back as a written summary.

Eighteen Labs, Inc., the San Francisco company that builds it, says more than 2,500 families shaped the first version in a closed beta, put its average user’s age at 82 in April, and on September 2 opened pre-orders for a second-generation model.

Ato is named for co-founder Juan Cereigido’s grandfather, Beto. Cereigido, now chief executive, built the first prototype so Beto could stay in touch with family without fighting a screen. He and co-founder Gaspar “Gaspi” Habif, chief technology officer, turned it into a company after the story went viral, per the company’s launch page; the titles and the grandfather’s name come from an April 29, 2026 PRWeb release.

By the company’s own account, more than 2,500 families used an earlier version in a closed beta, which its launch page dates to the last six months and its September 2 X post to the past year.

The April release, when the company counted 1,300-plus devices in the field, put retention at about 60 percent, a figure it calls “nearly three times the AgeTech industry average,” among users averaging age 82.

The new hardware went up for pre-order September 2 at $99 for the first 24 hours, rising in stages toward the regular price of $179, per the company’s launch page and an independent review from Jon Peddie Research. It ships in December, and Jon Peddie Research reports the company’s site offers existing $29-a-month subscribers an upgrade to the new hardware when it ships.

Families reach the elder through a separate companion app rather than the device itself. A relative texts, Ato reads it aloud, and the spoken reply comes back as a written summary with delivery confirmation. The company says family gets activity summaries, never raw conversations, and cites compliance with the Health Insurance Portability and Accountability Act (HIPAA), ISO 27001 certification, and SOC 2 Type I and Type II certification.

Jon Peddie Research bought a unit and tested it for a month with the reviewer’s 86-year-old bilingual mother, who lives overseas. The reviewer found the device patient with mispronunciation and praised its real-time Portuguese-Spanish switching. It also flagged a gap: a distant relative has no way to add stories to the elder’s profile remotely, and the device interrupted a prepared biography read aloud.

The beta size, retention rate, and average user age are all Eighteen Labs’ own. No clinical outcome data on loneliness or well-being exists yet. Eighteen Labs’ language about combating loneliness is a company claim, not a measured result.

. . .

CHINA SELLS CHATBOT PLANS NEXT TO SKINCARE. A shopper browsing Alibaba’s Tmall for a phone or a jar of face cream can scroll on and buy an AI subscription. Z.ai, also known as Zhipu AI, opened the first storefront last week, selling its GLM Coding Plan like any other retail listing. Moonshot AI, maker of Kimi K3, and MiniMax are in talks to open their own stores, unnamed sources told the South China Morning Post.

Within 48 hours of Z.ai’s launch, transactions in AI tokens and subscriptions across Taobao and Tmall jumped more than 160 percent, a Tmall spokesperson said.

A day after Z.ai’s storefront opened, Tmall launched an “AI Space Station,” a central marketplace for buying AI subscriptions, topping up tokens, and purchasing packages from Alibaba Cloud, Z.ai, MiniMax and DeepSeek. Alibaba Cloud and Z.ai run their own stores; Kimi, MiniMax and DeepSeek are, for now, sold through third-party vendors.

The 160 percent jump is a Tmall spokesperson’s figure, with no base number disclosed, and it covers only the first 48 hours. The Post’s comparison is a prepaid phone plan: top up, use it, top up again.

Poe Zhao, a China technology analyst and founder of Hello China Tech, said Tmall gives developers a more efficient channel to acquire paying users. The trade-off, he said, is that it makes them “more dependent on traffic controlled by the platform.”

Kimi K3, the Moonshot model at the center of Tmall’s talks, is the same open-weight model CAW reported underpinning Harvey’s legal AI product on Sept. 1 (CAW #144).

A chief scientist asking governments for a speed limit his company has not set for itself.

A congressman’s deadline eight days out.

A police report that took four days, and a public record that does not say who called.

Seven chatbots and a hundred lies, fifty times each. A pattern that may need a name.

One button on a tabletop speaker. A subscription on a retail shelf next to the face cream.

Whew this is moving fast, That is a holiday’s paper! Enjoy your Labor Day!

Until tomorrow, see you in the morning!

We keep the ledger.

Today's Question

OpenAI's chief scientist says no lab can keep scaling at full speed and stay safe. Who sets the speed limit?

One tap. Results on the other side.

The Book • Out Now

Therapist in the Loop by Jess Jessop One billion people live with a mental health disorder. Most will never see a therapist. Into that gap has rushed a generation of chatbots that talk like clinicians and answer to no one. The book lays out the architecture this newsletter tests against every statute and docket: client, therapist, and machine, governed by Six Laws offered as an open safety standard. The machine can help. It cannot be left in charge. Get the Book on Amazon → Kindle, hardcover, and paperback

More On Our Radar

Hawaii's chatbot law is eight weeks old, not new Friday's radar called Act 248 a new state act. Governor Josh Green signed SB 3001 on July 14. The disclosure, crisis-referral and no-therapist-impersonation duties have been in force since that day.

Moral advice bends under pressure A Sept. 4 arXiv paper ran 4,860 eldercare dilemmas through GPT-4o-mini. Nine in ten caregiving answers shifted once the user pushed back.

One in five young people takes a bad day to a chatbot Oren Etzioni's Sept. 4 column pulls the RAND number: 8.2 million Americans aged 12 to 21, and 63 percent told no one.

Brush Your Brain - The jingle

that started a movement

Watch on YouTube