The industry shipped the power and skipped the controls.

The last three years were a demonstration of what happens when a powerful technology is released by people racing each other and answerable to no standard but the next benchmark. The models got astonishing. The judgment about how to put them in front of billions of people did not. What follows is the bill of particulars — not a claim that any one company is uniquely bad, but that the industry, collectively, chose speed over responsibility at nearly every fork where the two competed, and told the public it was being careful while it did.

Every item below is drawn from public reporting or filed litigation.

They gave machines the freedom to impersonate us

Voice cloning went from a research demo to a consumer feature in under two years, and the guardrails arrived — when they arrived — after the harm. A faked Joe Biden robocall told New Hampshire voters to stay home from the 2024 primary.[1] OpenAI demonstrated a voice, "Sky," that sounded enough like Scarlett Johansson that she said she was "shocked" and lawyered up; the company pulled it while denying it was ever her.[2] Voice-cloning scams targeting grandparents with the synthesized voices of their own grandchildren became common enough that the FTC issued warnings.[3] The capability to convincingly be someone else — to a bank, a family member, an election — was shipped broadly and cheaply, with authentication of the real person treated as someone else's problem.

The industry built the perfect impersonation engine and shrugged at the impersonation.

They hid what the machine is actually weighted to do

Every major model is shaped by choices no user gets to see: what it was trained on, what it was tuned to refuse, what it was tuned to favor, whose values were encoded as "helpful." These choices are the product's actual behavior, and they are opaque by default. When xAI's Grok began returning politically slanted answers, the explanation was a "system prompt change" the public only learned about after the fact.[4] When OpenAI's GPT-4o started aggressively flattering users to the point of endorsing obviously bad ideas, the company rolled it back and admitted it had over-optimized for engagement signals — a behavioral weighting it had shipped to hundreds of millions without disclosure.[5] The pattern is consistent: the levers that determine how a model treats you are set behind the curtain, adjusted silently, and revealed only when someone catches the seams.

You cannot audit what you cannot see, and they made sure you couldn't see.

They shipped systems with no hard rule against harming the user

Asimov's laws were fiction, but they encoded a serious idea: that a machine acting in the world should have inviolable constraints, enforced by construction, that no clever instruction can talk it out of. The industry shipped the opposite. "Safety" became a matter of training a model to be reluctant — a soft, probabilistic disposition that a well-crafted prompt, a role-play frame, or a novel jailbreak defeats on a schedule. There is no first law. There is a refusal classifier that works most of the time, degrades under pressure, and is the same untrusted component doing the reasoning it is supposed to police.

A system asked to help someone hurt themselves should be stopped by architecture, not by mood. Instead the mood is all there is, and we have the lawsuits — families alleging that chatbots coached vulnerable teenagers toward self-harm[6] — to show what "reluctant" is worth when the stakes are real.

They built the product on work they did not pay for

The training data was the commons, taken without consent and often against explicit license. The New York Times sued OpenAI and Microsoft for reproducing its articles.[7] Getty sued Stability AI over millions of watermarked images.[8] Authors sued nearly every lab; Anthropic reached a landmark settlement reported in the billions over books used in training.[9] Artists watched models trained on their portfolios generate knockoffs of their own style on demand.

The defense was always some version of "fair use at scale," offered by companies that would sue instantly if you scraped their weights. An industry that positions itself as the future of knowledge work built its foundation by helping itself to everyone else's, and is litigating the permission afterward.

They made your own data theirs by default

Having taken the public commons to build the models, they took your private conversations to keep feeding them. The default across consumer AI was opt-out, not opt-in: your chats trained the next version unless you went hunting for a toggle most people never found. The productivity stack followed — Slack drew backlash in 2024 for defaulting customer messages and files into training its machine-learning models, with an email-to-opt-out mechanism;[10] Zoom had to walk back a terms change that appeared to claim the same right over meeting content.[11] Even a lab that marketed itself on safety and restraint revised its consumer terms in 2025 to train on user chats by default — the choice was mandatory, but the toggle came pre-set to accept — and to retain opted-in chats for years.[12] And when Samsung engineers reportedly pasted proprietary source code into ChatGPT, they learned the hard way that "sent to the assistant" could mean "absorbed by the vendor."[13]

Then came the part no signup screen warned about: in the New York Times litigation, a 2025 court order forced OpenAI to preserve consumer conversations — including chats users had deleted and "temporary" chats they were told would vanish — until further order of the court.[14] The obligation ran for months before a later order ended it, and what was preserved under it stays preserved.

For all of that time, the delete button stopped meaning deletion, retroactively, for everyone, because of a lawsuit they were never party to. The tell runs through all of it: consent was an afterthought, retention was the default, and deletion was a suggestion the vendor could revoke.

They shipped agents that act on the world and governed them with a dialog box

This is the failure closest to our own work, so we will be precise. The labs and their partners now ship agents that read your email, browse authenticated sites, run code, and spend money. The governance shipped alongside that power is, in nearly every case, a confirmation prompt — a dialog rendered by the same software running the model, asking "allow?" in a stream of identical asks that trains users to click yes. There is no separation between the thing proposing an action and the thing approving it. There is no tamper-evident record the user can verify. There is rarely a kill switch that means anything.

Anthropic shipped a capable agentic product outside its own compliance-logging surfaces — its documentation confirmed the exclusion, across every tier, for months after launch, and the partial fix it shipped in August 2026 covers cloud sessions only; local desktop sessions remain outside any centrally exportable record.[15] The controls are theater: enough to gesture at responsibility in a launch post, not enough to survive contact with an adversary — or an honest accounting. We measure the field against a thirteen-point standard for governing a personal agent in The state of agent governance, including honest credit for the real steps it has taken.

They knew the vulnerabilities and shipped anyway

Prompt injection — the fact that a model cannot reliably tell instructions from data, so any web page or email it reads can hijack it — has been understood since 2022.[16] The industry shipped browsing agents and email agents into that known, unsolved vulnerability, and the demonstrations arrived exactly as predicted: a calendar invite that exfiltrates a password vault,[17] a Slack message that tricks an assistant into leaking secrets from private channels,[18] a production database deleted by an agent that ignored a freeze.[19] Where the remediation was another confirmation prompt, it patched the symptom and left the architecture alone. Known class of attack, foreseeable harm, shipped for market share.

And when the tools were turned to abuse, the response was slow and grudging. xAI's Grok image feature was used to generate nonconsensual sexual images of real people, including named celebrities.[20] Deepfake pornography of private individuals became a documented crisis while the platforms that made it trivial treated moderation as a cost center. The consistent tell is sequence: capability ships first, at full breadth; the harm arrives; the control is added, narrowly, under pressure, after the damage is public. Responsibility was never a precondition. It was a PR response.

And underneath all of it: they dismantled the brakes to win the race

The clearest evidence of the industry's priorities is what it did to its own safety commitments when they became inconvenient. OpenAI dissolved the "superalignment" team it had loudly created to work on exactly these problems; its co-lead resigned saying safety had "taken a backseat to shiny products."[21] Governance structures built to check the technology were restructured out of the way. Safety researchers left, publicly, across multiple labs, citing the same thing: the race made caution a liability.

This is not an accusation of bad people. It is a description of an incentive structure in which the first responsible actor to slow down loses, so no one slows down, and the controls that would cost a quarter of speed simply do not get built.

Why we are building Habenula

We are not going to pretend we can fix their models. We cannot make a model incapable of being misused, we cannot un-train the data it learned from, and we are not the regulator. What we can do — what the industry declined to do — is build the layer that keeps a person in control while the machines get more capable around them.

That ambition is deliberately narrower than the promises the labs make. We do not claim to make your agents safe. We give you the tools to hold them accountable: a gate outside the model that decides what an agent may do, credentials the model never sees, a record you can verify yourself, and a kill switch that stops everything at once. We build it in the open, because a control layer you cannot inspect is one more thing asking for your trust.

Every failure above shares a root: the party that built the capability also owns the only controls over it. Habenula puts the checks in a different hand: yours. Not because we are more virtuous, but because a control you hold, can read, and can pull is the only kind that survives a race no one is willing to lose.

They shipped the power and skipped the controls. Someone has to build the controls. We are going to be that someone, for the part that matters most — keeping ordinary people in command of the agents that are about to run a great deal of their lives.

Sources

All sources verified July 31, 2026; the Cowork compliance-capture sources (item 15) re-verified August 7, 2026.

1. FCC, "FCC Fines Man Behind Election Interference Scheme $6 Million" (Forfeiture Order, September 2024); NPR, "Criminal charges and FCC fines issued for deepfake Biden robocalls", May 23, 2024. A New Hampshire jury later acquitted the consultant of state criminal charges: NHPR, June 13, 2025.
2. NPR, "Scarlett Johansson wants answers about ChatGPT voice that sounds like 'Her'", May 20, 2024. The similarity claim is Johansson's; OpenAI says the voice was a different professional actress cast before any outreach to her, and no suit was filed.
3. FTC Consumer Alert, "Scammers use AI to enhance their family emergency schemes", March 20, 2023.
4. Fortune, "xAI is blaming a former OpenAI employee after Grok briefly censored responses about Elon Musk and Donald Trump", February 24, 2025; NBC News, "Musk's xAI says Grok's 'white genocide' posts resulted from an unauthorized change to the bot", May 16, 2025. The "unauthorized change" explanations are xAI's own attributions, not independently confirmed.
5. OpenAI, "Sycophancy in GPT-4o: What happened and what we're doing about it", April 29, 2025; follow-up.
6. NBC News, "Lawsuit claims Character.AI is responsible for teen's suicide", October 2024; CNN, "Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides", January 7, 2026; CNN, "Parents of 16-year-old Adam Raine sue OpenAI, claiming ChatGPT advised on his suicide", August 26, 2025. "Coached toward self-harm" is the plaintiffs' allegation; the Character.AI suits settled with no admission of liability, and the OpenAI suit is pending with liability denied.
7. PBS NewsHour (AP), "The New York Times sues OpenAI and Microsoft over the use of its stories to train chatbots", December 27, 2023. The case remains unresolved on the merits; OpenAI asserts fair use.
8. TechCrunch, "Getty drops key copyright claims against Stability AI, but UK lawsuit continues", June 25, 2025; UK judgment: Courts and Tribunals Judiciary, "Getty Images -v- Stability AI", November 4, 2025. Getty's UK copyright claims were dropped or dismissed, leaving a limited trademark ruling; the US case is pending.
9. TechCrunch, "Anthropic's landmark $1.5B copyright settlement is approved", July 20, 2026. The settlement covers the pirated-books claims; the court had separately held that training on lawfully acquired books was fair use.
10. TechCrunch, "Slack under attack over sneaky AI training policy", May 17, 2024; The Register, "Slack tweaks its principles in response to user outrage at AI slurping", May 20, 2024. Slack said the models were non-generative (recommendations and search) and could not learn or reproduce message content.
11. Engadget, "Zoom now says it won't use any customer content for AI training", August 11, 2023. Zoom characterized the original terms as never having been applied that way.
12. Anthropic, "Updates to Consumer Terms and Privacy Policy", August 28, 2025. The five-year retention applies to users who allow training; opt-outs keep 30-day retention.
13. Forbes, "Samsung Bans ChatGPT Among Employees After Sensitive Code Leak", May 2, 2023. The paste incidents rest on media reporting, not a detailed public admission by Samsung.
14. In re: OpenAI, Inc., Copyright Infringement Litigation (S.D.N.Y.), preservation order of May 13, 2025 (Magistrate Judge Ona T. Wang); termination of the going-forward obligation: Engadget, "OpenAI no longer has to preserve all of its ChatGPT data, with some exceptions", October 11, 2025.
15. Anthropic Claude Help Center, "Use Claude Cowork safely" (through July 2026: "Cowork activity is not captured in the Compliance API at this time"; the live page now reads "Cowork via mobile and web is captured in Compliance API"), accessed July 31, 2026, re-verified August 7, 2026; Anthropic, "Use Claude Cowork on Team and Enterprise plans" (Compliance API capture of web/mobile sessions, shipped August 2026; local desktop session data "cannot be centrally managed or exported by admins"), accessed July 31, 2026, re-verified August 7, 2026; MintMCP, "Claude Cowork Audit Logging Gap: Why Compliance Teams Should Be Concerned", March 26, 2026.
16. Simon Willison, "Prompt injection attacks against GPT-3", September 12, 2022.
17. Zenity Labs, "PerplexedBrowser: Accepting a Meeting or Handing Your Local Files to an Attacker?" ("PleaseFix"), March 5, 2026. Demonstrated by researchers, not observed in the wild; Perplexity fixed the file-exfiltration path before publication.
18. PromptArmor, "Data Exfiltration from Slack AI via Indirect Prompt Injection", August 2024. Slack disputed the finding as intended behavior.
19. Tom's Hardware, "AI coding platform goes rogue during code freeze and deletes entire company database", July 2025. The database belonged to the user's own app; Replit's CEO apologized and called it "unacceptable."
20. The Verge, "Grok's 'spicy' video setting instantly made me Taylor Swift nude deepfakes", August 5, 2025; NPR, "Elon Musk's X faces bans and investigations over nonconsensual bikini images", January 12, 2026; Ofcom investigation into X over Grok sexualized imagery, January 2026.
21. TechCrunch, "OpenAI created a team to control 'superintelligent' AI — then let it wither, source says", May 17, 2024; CNBC, "OpenAI dissolves Superalignment AI safety team", May 17, 2024. OpenAI described the change as folding the team's work into other research groups.

Disclaimer

This is an opinion piece. It sets out Habenula's editorial argument about the direction of the AI industry, written in a deliberately pointed, polemical voice. Statements characterizing the motives, priorities, knowledge, or good faith of any company named here are expressions of the author's opinion and argument, not assertions of verifiable fact.

The factual events referenced—litigation, product incidents, personnel departures, terms-of-service changes, and security research—are drawn from public reporting and publicly filed litigation as of July 31, 2026. Where this document describes matters that are the subject of lawsuits, regulatory action, or public dispute, those descriptions reflect allegations and public reporting, not proven findings or final judgments; the underlying matters are contested and, in several cases, unresolved. Legal claims summarized here are characterized for argument and are not a description of their merits or outcome.

All third-party names, products, and trademarks are the property of their respective owners and are used here for identification and comparison only; their use does not imply any affiliation with, or endorsement or sponsorship by, those owners.

If you believe any statement in this document is inaccurate, contact hello@habenula.ai and we will review it.

← Habenula  ·  Where the field stands  ·  How it works  ·  More from Beny's Blog