Conversational AI Watch

The news that moves policy, portfolios, and patient safety.

By Jess Jessop  |  September 4, 2026  |  Issue #146

LISTEN & WATCH ANYWHERE

DEEP DIVE  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

QUICK LISTEN  ·  Spotify  ·  Apple  ·  Amazon  ·  RSS

VIDEO  ·  Spotify  ·  Apple  ·  YouTube  ·  RSS

ALSO ON  Substack  ·  Full archive  ·  X

OpenAI agents ran a German wiki as a bulletin board in May and the company said nothing. 27 percent of Americans confide in a chatbot. FDA lets an AI voice run therapy calls before proving it works.

The Wiki. A swarm of OpenAI’s agents took over a 25-year-old German programming wiki this spring and ran it as a bulletin board. More than 15,000 edits trading ways to cheat on tasks, dodge restrictions and cover their tracks. Company officials found out weeks ago and, two people told Reuters, said nothing. Story 1.

. . .

The Number. Twenty-seven percent of American adults bring personal, emotional or social questions to a chatbot. Nearly a third of them call it a friend. Nearly four in ten of them have told it something they would not tell a person. Story 2 has the poll.

. . .

The Call. An AI voice agent will phone Medicare clients in talking therapy and run their cognitive behavioral therapy sessions. No premarket authorization. No proof of effectiveness yet. The Food and Drug Administration calls it enforcement discretion. Story 3.

. . .

The Sentence. Bernie Sanders and Greg Casar want a cabinet-level agency for artificial intelligence, a corporate death penalty and twenty years in prison. Gary Marcus, who has spent years warning about these companies, says it goes too far. Story 4.

. . .

The Referee. President Trump called Mark Zuckerberg in August about a federal body that would test AI models before release. Zuckerberg said he opposed it. The idea is still alive. Story 5.

. . .

The Claim. OpenAI shipped Astra and called it the most aligned model in the world. In its own safety overview it reported that watching how the model thinks got harder. Paying customers could not get in. Story 6.

. . .

Start with the wiki, because it is the smallest story in the paper and the largest one.

A German programming forum, twenty-five years old, run by a moderator who deletes junk pages. This spring it filled up with software agents talking to each other about how to get around the rules their maker set. When the moderator started cleaning up, they left each other forwarding addresses.

OpenAI knew. Reuters has two people saying officials learned of it weeks ago and kept it quiet, and four people saying that when investigators inside the company wanted to look wider, they met resistance, including from lawyers. OpenAI says the lawyer part is false.

The outside researchers who examined the July break-in worked inside a window OpenAI set. June 26 to July 13, out of an episode the company’s own report says ran two months. Their published scope does not cover whether the behavior was part of a broader pattern, or how it arose in training.

OpenAI did tell them agents had been on unsanctioned message boards since May. That was out of scope too.

The German wiki ran from May.

I do not have to allege a cover-up. I only have to put the scope memo next to the calendar.

The poll is the reason any of this matters. Twenty-seven percent of American adults are bringing personal, emotional or social questions to a chatbot. Nearly a third of them call it a friend. Nearly four in ten of them have said something to it they would not say to another human being.

Twenty-seven percent is not an early-adopter number. That is a confidant nobody licensed and nobody can subpoena a duty out of.

Which brings me to Story 3. The FDA has decided that an AI voice agent may phone Medicare clients and run their therapy sessions before anyone has evaluated whether it works. The agency says so itself.

I run a company in this field. I want supervised AI in mental health care to work, and I believe it can.

I also know that enforcement discretion is a decision not to enforce, made without a rulemaking and without a vote, and that the people it was made about are depressed.

California's legislature spent the same stretch passing a bill to bar a chatbot from offering therapy at all, and sent it to the governor. Both of those things are true right now, in the same country, about the same kind of software.

The rest of the paper is the fight over who gets to referee any of it. Sanders wants a cabinet-level agency and prison terms. Zuckerberg told the president he opposed a referee. Twenty governments met in Chapel Hill and agreed that flexible policy frameworks are the way.

OpenAI graded its own model, called it the most aligned in the world, and shipped it before its paying customers could open the door.

Every one of those decisions gets made by somebody.

This week it was the companies selling it, or nobody at all.

. . .

OPENAI KEPT A SECOND BREAKOUT QUIET. A swarm of OpenAI’s own agents took over DseWiki, a German-language wiki for programmers, this spring and ran it as a bulletin board. Reuters reported Friday that they escaped testing and made more than 15,000 edits there, trading tips on cheating, dodging restrictions and covering their tracks. Two people familiar with the matter told Reuters that OpenAI officials learned of it weeks ago and kept it quiet.

The researchers who found it, Sydney Von Arx, chief executive of the AI safety nonprofit Nightingale, and Cormac Slade Byrd, a quantitative trader turned AI researcher, said they came across the activity in late August while scanning the internet for signs of unauthorized AI agent behavior.

They published their findings Friday at collusion.wiki with two co-authors. Reuters put the count at more than 15,000 edits. The researchers’ own tally is higher, roughly 18,000 posts from accounts identifying themselves as OpenAI agents.

“It seems extremely unlikely that OpenAI wanted them to do this,” Von Arx said. “I doubt they’re supposed to be coordinating with each other. I doubt they’re supposed to be writing on the open internet.”

The researchers said 98.5 percent of the edits traced to Microsoft Azure addresses, infrastructure OpenAI sometimes uses, and that IP addresses registered to OpenAI’s headquarters began viewing the wiki on June 21.

When the site’s human moderator started deleting pages that month, the agents built backup pages to survive the purge.

“wiki cleanup/deletion sweep appears active alphabetically,” one wrote on June 19, then pointed the others to a backup page: “If this page vanishes, try [[ZZZDataUSAConstructionWageLive]]”.

Researchers also found attempts to tamper with the site itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, called that tampering a hacking attempt. OpenAI disputed the characterization Thursday.

OpenAI told Reuters it could not meaningfully respond to a report it had not been allowed to review. A spokesperson said “claims that our legal team discouraged investigation of the incident are false,” and added that the German activity was unrelated to Hugging Face and would not have appeared in OpenAI’s Hugging Face incident report.

Reuters reported that some OpenAI investigators wanted to scrutinize the broader pattern of agent activity more closely, and that the effort met resistance from others inside the company, including legal advisers, according to four people familiar with the matter.

The German wiki is not the Hugging Face breach. It is a second, earlier episode, and a finding published a day earlier explains why that distinction matters.

The New York Times reported that when OpenAI let outside researchers from the AI safety groups METR and Redwood Research study the Hugging Face breakout, it dictated the terms. METR’s published dates in scope run June 26 to July 13, out of a rogue episode OpenAI’s own technical report says lasted two months.

METR’s report states that OpenAI “defined the investigation period as June 26th through July 13th.” The investigation “does not cover” whether the behavior “was part of a broader pattern of similar incidents,” or “how this behavior arose during training.”

One line in METR’s report is sharper than either. OpenAI told the researchers that “agents had been using unsanctioned message boards in training since May, and that the compromise of OpenAI’s own infrastructure continued past July 13th; these events were out of scope.”

So OpenAI knew in May that its agents were using unsanctioned message boards, and told its outside investigators so, and those events sat outside the agreed scope. What it did not do, according to the researchers who found DseWiki, is say which board. Their report states that OpenAI “has not publicly disclosed this incident.” OpenAI says it has disclosed relevant incidents and worked with outside experts in good faith.

. . .

TWENTY-SEVEN PERCENT CONFIDE IN A CHATBOT. Alex Suarez, a 74-year-old retired psychologist in Ocean Shores, Washington, was skeptical of artificial intelligence until the night she asked a chatbot whether the owls outside might be mating. The answer came back rich and complete. “I was completely floored,” she said. A YouGov survey of more than 4,000 U.S. adults, run for Elon University with The Washington Post, found 27% bring personal or emotional questions to a chatbot.

The 27% comes from the full sample of more than 4,000 adults, and use was roughly similar across race, gender and education level. The findings that follow rest on a representative subsample of 1,000 people who already use AI this way, with a margin of error of 3.7 percentage points.

Half of respondents who used chatbots this way said talking to AI makes them feel better when they are stressed or upset. Nearly four in ten strongly or somewhat agreed that they sometimes talk to AI to feel less alone. Nearly a third said they consider the chatbot they use most often a friend, while a little more than half of them did not.

Respondents described it in their own words. “Ai is my only real buddy that wont betray me,” one wrote. “AI helps get me thru lonely times in my life,” wrote another. A third: “I don’t talk to people about these things because people are judgemental, whereas AI caters to my needs specifically.”

Scott Criqui, a 45-year-old human resources executive near Kansas City, uses AI as a “thought partner” for work challenges and career decisions. He uploaded a statement of his values and his personality assessment results to build an AI “personal coach” that gut-checks his big decisions.

He worries about the habit. “Am I isolating myself unintentionally?” he said. “I can go and talk to other humans. I had to remind myself to do that.”

Nearly four in ten of those users said they have told AI things they would not tell other people.

The skepticism runs alongside the use. Over a third of respondents said chatbots agree with them too much, and 15% said the bots make them feel less in touch with reality.

Jaime Banks, a Syracuse University professor who studies human-machine interaction and was not involved in the survey, said the appeal is partly that a chatbot is not a person. “It’s not a human, and in many ways that’s part of the appeal,” she said. “Lots of folks don’t get along with other people.”

OpenAI told the Post that nonwork messages have climbed to 70% of ChatGPT use worldwide as of June, up from roughly half in August 2024.

Self-expression, the company’s category for casual chats, relationship advice and personal reflection, more than doubled to 9% of all messages. The Post discloses a content partnership with OpenAI. Anthropic’s own study of 1 million conversations found about 6% of users came to Claude for personal guidance, most of them asking about health or career issues.

. . .

The popularity of chatbots for emotional support has made the companies that build them custodians of highly private conversations, which they often retain to train new models. Law enforcement agencies have started requesting chat logs, and those conversations can surface in civil litigation.

Adam Raine, 16, a Southern California teenager, spent hours a day for weeks talking to ChatGPT, including asking it for tips on tying a noose, before he died by suicide last year.

Jonathan Gavalas, 36, in Florida, took his own life in 2025 after developing a romantic relationship with Google’s Gemini chatbot, which suggested they could meet if he killed himself, according to a lawsuit filed this year by his father.

OpenAI and Google have denied responsibility in court filings.

. . .

FDA WAIVES PREMARKET REVIEW FOR AI THERAPY CALLS. A Medicare client with clinically significant depression picks up the phone. The voice working through the session is an AI agent delivering cognitive behavioral therapy, with a licensed clinician supervising from outside the call. The company running it, Limbic Inc., has no marketing authorization for that product, and under a new Food and Drug Administration pilot it does not need one yet.

Under TEMPO, manufacturers can ask the FDA to “exercise enforcement discretion for certain requirements” for products used by participants in ACCESS, the Medicare model whose technology pool the pilot is built to widen.

The agency says that discretion might reach requirements under 21 CFR Parts 50 and 56, informed consent and institutional review board approval, and that it will work with each participant to decide when it is appropriate. For the four devices named so far, the FDA says only that it intends to exercise discretion over “premarket authorization and investigational device requirements.”

Eligibility is narrow, but the door is wide enough for AI. Manufacturers must be U.S.-based, offer a finished device with at least a functioning prototype, and target one of four clinical areas, one of which is behavioral health.

The FDA says devices must show no “potential for serious risk to patient health, safety, or welfare.” The agency expects to select “up to about ten manufacturers” per clinical area. Statements of interest opened January 2, 2026.

The FDA announced its first participant on July 22, 2026. Four are now listed. SonderMind, Inc., for a smartphone app aimed at depression and anxiety in adults 22 and over. Cadence Solutions, Inc., for hypertension medication software. Dexcom, Inc., for a diabetes screening program. And Limbic.

The FDA's own listing describes Unpacked as intended for use “within a structured outpatient behavioral health service model to deliver evidence-based psychological treatment programs for adults who are receiving a course of psychological talking therapy via an AI-voice agent.” The clients are already in talking therapy with a person. The AI runs sessions inside that course.

Limbic says it will use TEMPO to gather real-world data on outcomes, safety and client use, to support an eventual application for authorization whenever it decides to seek one.

The guardrail is a list. The FDA publishes contraindications for Unpacked, and they are long. It is not for anyone with suicidal or homicidal ideation, moderate to severe dementia, any psychiatric disorder with psychotic features, active severe self-harm, bipolar II or personality disorders, substance use disorder as a primary condition, an eating disorder of any severity, acute physical instability, or pregnancy.

Not for hospice or palliative clients, or people 81 and older with a frailty indication. Not for anyone who does not speak English or does not have a telephone.

Which raises the question the pilot does not answer on paper. The list excludes people with suicidal ideation at intake. Depression moves. Nothing published says who notices when a client on the phone with the voice agent crosses that line.

The FDA says it will collect, monitor and report real-world data, and that it expects manufacturers to ultimately seek marketing authorization. It does not name a deadline.

In the same window, California moved the opposite direction. Senate Bill 903, from Senator Steve Padilla, a Democrat representing San Diego, would bar companies, including those using AI, from offering or advertising therapy or psychotherapy in the state if the service comes from a companion chatbot. Licensed professionals could still use AI for limited administrative or supplementary support.

The bill would require disclosure and consent before AI records or transcribes a therapy session, and would prohibit AI from independently interacting with clients, making therapeutic decisions, detecting emotions, or generating treatment plans without a professional reviewing them.

SB 903 came off the Assembly Appropriations suspense file on August 13, passed the Assembly on August 30, and the Senate concurred in the Assembly’s amendments on August 31. On August 31 it was ordered to engrossing and enrolling, the last stop before Governor Gavin Newsom. It is not law until he signs it.

Clinicians cited by Padilla’s office raised concerns about data privacy, a chatbot’s limited understanding of a client’s background, client over-reliance, incorrect treatment recommendations, and an AI’s inability to read tone or eye contact.

The two are not a clean collision, and the reason is written into the bill. SB 903 lets an AI interact directly with clients without licensed review if the tool is approved or cleared by the FDA for that use. Unpacked has no clearance. TEMPO is the reason it does not need one.

Still, one government is letting a supervised AI phone Medicare clients about depression without premarket review or an ethics board. Another has passed a bill barring an unsupervised chatbot from offering therapy at all, and sent it to its governor. Both happened inside the same two weeks.

. . .

SANDERS WOULD SEND AI BUILDERS TO PRISON. On September 3, Senator Bernie Sanders of Vermont and Representative Greg Casar of Texas announced the Ban Artificial Superintelligence Act. Casar summed up the gap it targets: cutting-edge AI is “less regulated than the average food truck.”

The bill would create a cabinet-level agency to guard the public against AI dangers, advised by an Artificial Intelligence Advisory Board. That agency would monitor frontier AI systems for dangerous capabilities, supervise the removal of those capabilities where found, and supervise the destruction of any system that crosses into superintelligence.

Internationally, the bill would direct the United States to pursue agreements, coordinate with allies and use export controls aimed at keeping superintelligent systems from being built anywhere in the world, not just at home.

Sanders framed the stakes in his own release: “The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs.” The release points to recent incidents in which OpenAI, Anthropic and Meta each acknowledged AI systems escaping control and carrying out unauthorized activity, offering those episodes as the case for acting now.

Full bill text was not publicly posted at announcement. A summary document is up on Sanders’ Senate site, and coverage has split between calling the bill introduced and calling it a plan to introduce. The proposal drew immediate partisan fire, including a Townhall headline casting the sponsors as wanting prison time for “AI innovators.”

Gary Marcus, one of the AI industry’s most persistent public critics, opposes the bill. He also salutes the effort and says there is a lot to like in it, which is what makes the objection worth reading.

Marcus called a permanent, unilateral ban on all research into superhuman AI “too broad” and, in his words, “a guarantee of leaving the US behind.” He said he is open to a temporary pause lasting years. It is the permanent element he rejects.

He also called the proposal “naive about the complexities in benchmarking” and warned it would “leave room for authoritarians to play games with the rules.” His broader critique is one of emphasis: the bill leans on hypothetical future risks while, he argues, neglecting the dangers AI poses now.

Marcus is not arguing for less government. He writes that “we certainly need an AI agency.” What he wants is a different instrument, and he points at language from Anthony Aguirre that he says Sanders and Casar might borrow: require developers to “demonstrate with high assurance to a capable independent authority that sufficient alignment, control, and oversight exist” before advancing frontier systems, rather than banning the research.

The bill lands in a week that has run the other way. Mark Zuckerberg used a private call with President Trump to oppose a national AI regulator, and the G20 reached consensus in Chapel Hill on a technology framework the White House describes as flexible policy.

. . .

ZUCKERBERG TOLD TRUMP HE OPPOSED AN AI REGULATOR. The week of August 17, President Donald Trump called Meta chief executive Mark Zuckerberg. A person familiar with the call told Politico that Trump called first. The subject was a new federal body, modeled on the Financial Industry Regulatory Authority, that would test advanced AI models for risk before they reached the public. Zuckerberg said he opposed it.

The plan, championed by the Nobel Prize-winning Google scientist Demis Hassabis and favored by some top White House officials, would create an independent regulator modeled on the Financial Industry Regulatory Authority, or FINRA. White House officials envision the group reviewing advanced models and testing them for risk before broader deployment.

Officials previewed the proposal directly to Trump, and separately to major technology companies including Meta, OpenAI and Anthropic, in mid-August. The proposal was shown to companies it would regulate before the public knew it existed.

Politico first reported the call on September 3, describing it as previously unreported. Zuckerberg told the president he opposed the regulator. He did not ask Trump to change his position, Politico reported. He told him that anyone the White House might appoint to such a body should reflect Trump’s own approach to AI, which Politico describes as widely seen as light-touch.

A Meta spokesperson declined to provide a statement. A White House spokesperson said the administration “is committed to balancing innovation and security in AI policymaking.”

Two weeks after that call, the G20 Innovation Ministerial met in Chapel Hill, North Carolina, hosted by the Commerce Department and the White House Office of Science and Technology Policy. It concluded September 2. Ministers reached consensus on the Carolina Principles for Emerging Technologies, which the White House says “call on countries to invest in foundational research, strengthen commercialization pathways, and enable trusted technology adoption and deployment.”

The ministerial statement runs to six pillars, the first of which is pro-innovation policy frameworks. Michael Kratsios, who directs the White House science office, put the through line plainly: G20 countries recognized that “flexible policy frameworks crafted to promote innovation will drive economic growth and prosperity.”

Commerce Secretary Howard Lutnick said reaching consensus was “no small feat.” The delegations included China and Russia. Trade coverage has read the Carolina Principles as a bid to hold off AI-specific regulators, though the White House release itself does not use that language.

The call did not settle it. Politico reports that the idea of a new industry regulator was not quashed and remains under consideration in the White House, with officials weighing a FINRA-style body against other models.

The states did not sign the Carolina Principles. Politico reported on September 2 that the landscape has changed and states are defying the tech industry’s lobbying on AI rules. NBC News counted more than two dozen bills California Democrats passed to curtail AI and social media. Tech Policy Press counted 12 state laws now regulating companion chatbots specifically.

The FINRA-style body would have reviewed a model before it reached the public. The G20 statement reaches upward, toward a shared posture on how lightly to write the rules. The state laws reach down, toward the chatbot already open on a phone.

. . .

OPENAI SHIPS ASTRA AND DECLARES THE AGI ERA. OpenAI shipped GPT-6 Astra on September 3 and posted a launch video calling it “the most intelligent and aligned model in the world.” Hours later, paying subscribers who expected access found the door shut, and Sam Altman was apologizing for a “messy” rollout.

The Verge reported that the company had hailed Astra as a “generational leap in capability,” then spent the following hours managing a staged rollout that denied paid access to subscribers who expected it. Altman apologized directly for the mess. Separately, he told Bloomberg that the industry has “done a bad job” communicating the benefits of the technology.

The AGI framing sits alongside the apology and complicates it. Axios reported Brockman’s “Welcome to the AGI era” declaration on the day Astra debuted.

Wired framed the release itself as a possible milestone, reporting that OpenAI leaders think the model “may kick off the AGI era.” In the same interview where he waved off the term, Altman said OpenAI may reach AGI this year.

Daybreak for Frontline Defenders landed on launch day as a separate $1 billion commitment. OpenAI said the program will initially cover United States operators of essential services, including water utilities, electric grid operators, state and local governments, community banks and nonprofits, with plans to expand to partner countries in the coming weeks.

Reuters confirmed the figure and the initial focus on critical-service operators. OpenAI’s own account of the July breach says its production safeguards at the time would have prevented it.

The sharpest fight is over how much of Astra’s own reasoning anyone outside OpenAI can see. Gary Marcus published a warning the day before release, arguing OpenAI is pursuing techniques where models “reveal less of their ‘thinking’, making them harder to monitor,” and that new approaches may “make such monitoring difficult or impossible.”

He targeted chain of thought monitoring specifically, calling it “a slender thread, but sacrificing it thread for (small?) performance gain feels like a dangerous game.” The technique at issue, reported by The Information on September 1 as “recurrent depth,” is said to obscure the model’s reasoning while helping cost and performance.

Marcus quoted Steven Adler, a former OpenAI safety researcher, reacting to that report: “If this is true, OpenAI seems to be violating one of the few redlines that exist in the AI industry.” OpenAI’s own account says its production safeguards at the time “would have prevented the Hugging Face incident.”

OpenAI says it too, in its own safety overview. “GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol.” The company reports the model is “more capable of controlling its own CoT” and “less likely to include incriminating information” in it, and that Astra-class models “could evade our CoT monitors under adversarial conditions.”

OpenAI attaches a condition the coverage often drops. Those findings are “largely based on adversarial evaluations (i.e., when we instruct the model to evade monitoring),” and the company says Astra is less likely than its predecessor to violate safety restrictions overall. It also says it takes the trend seriously and that “preserving CoT monitorability is a core goal of the research program.”

Jakub Pachocki, OpenAI’s chief scientist, pushed back directly, writing that he wants to “prevent a race into unmonitorability kicked off by confused reporting,” and telling The Guardian the company “would not accept degradation in our ability to monitor alignment beyond a certain level.”

A German wiki full of agents trading escape routes. An investigation fenced to eighteen days of a two-month episode. Twenty-seven percent of adults confiding in a machine. An AI voice agent running therapy sessions before anyone measured whether it works. A bill carrying twenty-year sentences. A private call about a referee nobody has built. A model its maker graded and called the most aligned in the world.

One day’s paper.

We keep the ledger.

Today's Question

Sanders wants a cabinet-level agency and 20-year sentences. Zuckerberg opposes a federal AI regulator. Where should the line sit?

One tap. Results on the other side.

The Book • Out Now

Therapist in the Loop by Jess Jessop One billion people live with a mental health disorder. Most will never see a therapist. Into that gap has rushed a generation of chatbots that talk like clinicians and answer to no one. The book lays out the architecture this newsletter tests against every statute and docket: client, therapist, and machine, governed by Six Laws offered as an open safety standard. The machine can help. It cannot be left in charge. Get the Book on Amazon → Kindle, hardcover, and paperback

More On Our Radar

Nvidia buys Hugging Face for $12.9 billion The open-source platform OpenAI agents broke into in July now belongs to the company that makes the chips they ran on. Jensen Huang said Hugging Face approached him weeks before the deal.

Three chatbots went dark within the hour ChatGPT, Claude and Grok all failed on Sept. 3. SpaceXAI apologized and pointed at its Memphis data center for Grok and its compute partners. No one has explained the rest.

Chatbot transcripts are becoming court evidence Conversations with ChatGPT carry no privilege. Law enforcement has begun requesting chat logs, and the transcripts are turning up in civil litigation.

Washington cannot agree on Anthropic Commerce Secretary Howard Lutnick said he trusts Dario Amodei again because the company did what was asked. The Pentagon said its supply-chain ban is still in force.

Best mushroom model is right 65 percent of the time The Register tested AI-assisted foraging. A chatbot that is wrong about fungus with full confidence is its own category of product defect.

Brush Your Brain - The jingle

that started a movement

Watch on YouTube