The promise is always the same. The software is nearly there. One more model, one more training run, and the machine will finally tell the honest students from the cheats. It will not. The failure is built into the maths. And even if it were not, detection would still be the wrong tool for the job. Those are two separate arguments, and either one is enough on its own.

I have spent a year reading the evidence and writing about it. Here is where it leaves me.

In this post I will:

• Show you why the technology fails, and fails hardest on the students least able to absorb it.

• Explain why the maths guarantees it will keep failing.

• Make the case for what to do instead, before prohibition drives the whole thing underground.

The technology does not work

I have made the case against detection here before. In March, in ‘Guilty Until Proved Human’, I set out the damage: the false accusations, the named students whose degrees and jobs were put at risk, the bias against people who learned English as a second language. That piece was about what detection does to people. This one is about why it can never be fixed, and why a flawless version would still be the wrong tool. The evidence has moved since March, and so has the argument.

A study published this month in Education and Information Technologies ran 81 essays through Turnitin’s AI detector, from fully human to fully machine. At the clean extremes it coped. A pure human essay was left alone. A pure ChatGPT essay was caught.

Turnitin handled the all-human and all-AI extremes but failed on mixed, real-world scripts

Then came the case that actually fills a marking pile: the hybrid. A student writes some of it, a model writes some of it, the two are stitched together. On those mixed scripts the detector fell apart, again and again failing to report how much of the work was AI. It could not tell you what you actually need to know.

The tool works on the two essays nobody is unsure about, and fails on the one in front of you. Almost no real student submission is fully human or fully machine. The detector is most confident exactly where there is nothing to decide, and lost exactly where the judgement is hard.

One study is one study, and I am wary of resting a whole argument on a single test. What this one shows is the shape of the failure, and that shape repeats wherever these tools meet real, mixed writing.

It punishes the students least able to fight back

Here is the part that should give any institution serious pause.

Detectors are biased against people who learned English as a second language. In a 2023 study in the journal Patterns, researchers ran genuine, human-written essays by non-native English speakers through seven detectors. On average, 61% were flagged as AI. One detector flagged 98% of real exam essays written by real students.

Genuine human essays flagged as AI: near zero for native speakers, 61% on average and up to 98% for non-native speakers

This is one study, using detectors from 3 years ago. However, what makes it more than a snapshot is the reason behind it. Detectors flag how predictable your words are, and writing in a second language tends to be more predictable. The bias is baked into the thing the tool measures, so a newer model does not fix it. It inherits it.

The writing was human. The students were human. The machine called them liars because their sentences were a little more predictable than a native speaker’s.

So the false accusation lands on the international student, the refugee, the first-generation undergraduate. The people already most likely to be doubted, and least able to argue back. A detector fails down the exact lines a university is supposed to protect.

The maths guarantees it stays broken

Defenders of detection say all of this is temporary. The tools will improve.

The maths says otherwise. In one study researchers proved that as a language model gets better at sounding human, the best detector that could ever exist gets worse. As machine writing and human writing converge, any detector’s accuracy slides towards a coin flip. The same team showed that a light paraphrase already defeats the detectors we have today.

So the technology is walking into a wall. It cannot be climbed without giving up the very fluency that makes these models worth using. Every gain that makes them more useful makes them less detectable, by design.

That wall has a human cost, and a more recent preprint shows it is baked into the maths. Any content-only detector with real power must produce false accusations at a rate set by how much student writing and AI output overlap. The detector has no idea what your normal writing looks like, so it judges you against an average. And that overlap is not spread evenly. It falls hardest on the students whose prose is cleanest and most standard, which describes many non-native English speakers, who tend to stick closest to textbook structure. The tool flags the students who followed the rules most faithfully.

It cannot reliably tell a careful human from a machine, because the two overlap by design. Push the false positive rate down and you miss real AI use. Push detection up and you accuse more innocent people. No setting escapes that trade-off.

This is already playing out. Washington State University dropped Turnitin’s AI detection this year after highlighting the risk that even a 1-2% false positive rate would pose to its students.

Other routes have been proposed. Watermark the model’s output, or attach provenance data to the file. Both collapse in a classroom. A watermark needs the AI company’s cooperation and dies the moment a student moves their text to a model that does not carry one. Provenance dies the moment a student retypes the words. The content-only detector is the one institutes actually buy, and that is the one the maths defeats.

Even a perfect detector would be the wrong tool

Now grant the impossible. Imagine a detector that worked flawlessly tomorrow.

I would still argue against it, for a reason that has nothing to do with accuracy.

The moment you put a detector between a teacher and a student, you change what a teacher is. You turn an educator into a police officer. The work of teaching is to be a co-creator of knowledge alongside the student, beside them while they struggle toward understanding. A detector makes suspicion the default setting. Every essay arrives presumed guilty.

That is not the job I signed up for. I did not become an educator to run my students’ words through a lie detector and wait for a number.

The evidence backs the instinct. I read every public UK university AI policy I could find, 96 of them, for a HEPI policy note this year. On a close reading of a sample, the language of education sat over an architecture of detection and surveillance. And two in five universities had no public AI policy at all. The machinery of policing arrived before the conversation about learning had even started.

You cannot police a student into learning. Detection keeps asking teachers to try.

Prohibition just builds a shadow

There is one more cost, and institutions notice it last.

What happens next is a prediction, though not a wild one. We have run this experiment before, with calculators, with Wikipedia, with phones in exam halls. Ban the tools, lean on the detector, and students do not stop using AI. They hide it. Use goes underground into what people are starting to call shadow AI: private accounts, personal phones, unlogged, unsupervised, unteachable.

A prohibition you cannot enforce is worse than no rule at all. The behaviour stays. What you lose is your sight of it, and with it any governance worth the name. The student still uses the model. You just no longer get to help them use it well.

What to do instead

The honest case for detection is not that the tools will get better. It is that high-stakes assessment needs a defensible check that works at scale, and a quiet chat does not obviously scale to a 400-student first year marked by exhausted sessional staff. That objection is real, and I take it seriously.

The answer is to change what you assess, so that scale stops requiring suspicion. A detector tries to verify a finished product. Process leaves a trail you can check on a sample, the way you already moderate marking. Honesty designed into the task scales better than honesty policed after it.

The alternative is harder and cheaper. It asks more of the teacher and nothing of the IT budget.

Talk to your students. Ask them how they actually use these tools, out loud, with no threat hanging over the room. Then build the assessment around that honesty.

A few concrete moves:

1. Ask every student to show their process. The prompts, the dead ends, the parts they rejected. Make the reasoning the thing you assess.

2. Make ‘how did you use AI here, and what did you decide for yourself’ a normal question, asked of everyone, never an accusation aimed at one.

3. Write your AI policy with students in the room, not as a clause bought from a vendor.

None of this needs software. All of it needs trust.

The argument, one more time

AI detection will always be broken. The software fails, the maths says it will keep failing, and it fails hardest on the students who can least afford to be doubted. Even a flawless version would still be the wrong tool, because the day you switch it on you stop being a teacher and start being a guard.

You do not need both halves of that. The evidence case and the ethical case stand alone, and either one is enough to put the software down.

We are using software to win an argument we should be having with our students. Have the argument instead.

Go Slow

This is the work we do in the Slow AI Curriculum: a year of structured, accredited CPD built on peer-reviewed research, with live monthly sessions where over 300 educators and practitioners think through exactly these questions together. If you want to move from policing to pedagogy with a method instead of a hunch, start here.

Subscribe now

On 9 June Anthropic released Claude Fable 5, its most capable generally available model, to millions of people. Three days later the US government barred foreign nationals from using it on national security grounds. To comply without trying to police who was using it, Anthropic pulled the model worldwide. For everyone, everywhere, the model people had spent the week building into their work simply stopped answering.

If your week depended on it, your week broke.

In this post I will:

• Show why any frontier model can vanish on you, and why the reasons are getting harder to predict.

• Explain what decades of research on automation tell us about the skill you lose while the machine is working.

• Give paid subscribers a model-independence audit. A step-by-step way to find which of your workflows would collapse if a model disappeared tomorrow, and how to make them survive it.

A model can disappear for reasons that have nothing to do with you

Fable 5 was pulled by a government. That is the newest way to lose a model, and it will not be the last.

There are others, and most of them have already happened.

A model can be deprecated. When OpenAI retired GPT-4o from ChatGPT, the workflows people had tuned to its exact behaviour stopped behaving. The replacement answers differently, and the work shaped around the old one has to be rebuilt.

A model can be changed underneath you. When Replika stripped out the intimacy features its users had built relationships around, people described it as waking up to find a partner replaced overnight. The app was the same. What it could do for them was gone.

A model can be priced out of reach. Frontier models run on usage credits. A free window closes, an API price moves, a budget tightens, and the tool you depend on is suddenly one you cannot justify.

And a model can simply break. Outages, rate limits, a safety filter that clamps down on the one thing you used it for.

The common thread is that none of these are your decision. You did not choose the deprecation, the policy change, the price, or the ban. You inherited it.

So, what happens to your work when the model you rely on is gone?

The skill you lose while the machine is working

When you hand a task to a tool completely, the underlying skill quietly fades. You stop noticing because the output keeps arriving. The first sign of the loss is silence.

This pattern predates AI. In 1983 Lisanne Bainbridge published a short paper on the ironies of automation that has shaped human-factors research ever since. Her argument is that the skills that atrophy during smooth automated operation are exactly the skills you need when the automation fails. The more reliable the system, the less you practise, and the worse equipped you are for the moment it stops. The human is left to take over a job they have slowly forgotten how to do.

Her evidence was drawn from industrial control and aviation, where the skills are physical. The pattern travels into knowledge work, though we should be honest that the proof there is younger. People who find their way everywhere by GPS show measurable decline in spatial memory the more they rely on it. The map-reading goes quiet, then it goes.

Apply that to a frontier model. If your analysis, your first drafts, your code, your summaries all run through one tool, the judgement underneath each task is the thing eroding while everything looks fine. When the model disappears, the practised skill the software stood in for is gone with it.

Different models, same work

Keep using AI. Use it in a way that cannot strand you when it goes.

Every model is different. They have different strengths, different prices, different owners and different politics. Picking the best one this month is a weak hedge, because the best one can be taken away in an afternoon. Resilience comes from the workflow: keep the process portable and the judgement live, so any capable model can step into the gap.

Keep the process visible. Keep the judgement live. Know, for every task you have handed over, what you would do if the handover stopped tomorrow.

Done well, a model going dark becomes an inconvenience you handle in an afternoon.

So how do you tell which of your workflows are resilient and which are one policy decision away from collapse? You audit them. Here is exactly how.

Slow AI is reader-supported. Become a paid subscriber to read on.

Paid subscribers also get access to The Slow AI Curriculum, an accredited CPD programme that runs live monthly sessions on exactly this kind of practice: the theory, the critical prompts and the dialogue that keep your judgement sharp while everyone else outsources theirs.

Subscribe now

Read more

Anthropic released Fable 5 free for twelve days, then the US government pulled it offline

On 9 June Anthropic released Claude Fable 5, its most capable public model, free on Pro and Max plans, alongside a gated sibling called Mythos 5. Three days later it was gone. Citing national security, US Commerce Secretary Howard Lutnick signed an export-control directive ordering that both models be denied to any foreign national, inside or outside the United States, including Anthropic’s own overseas staff. Rather than filter by nationality, Anthropic took both offline for everyone. The stated trigger was a narrow jailbreak that let Fable 5 read source code and hunt for vulnerabilities. And it was the second time in a week the model’s fate was decided over users’ heads: days earlier, researchers found a line in its 319-page system card showing Anthropic had quietly weakened Fable 5 for some users without telling them, a choice it walked back after an outcry. Anthropic is complying while disagreeing, with no timeline to restore access. Opus, Sonnet and Haiku stay up.

This is the week’s thread in its purest form: who gets to ask, and who decides. First Anthropic quietly chose to weaken its own model for some users without telling them. Then the government decided, in a single afternoon, that everyone on Pro and Max could not use a model they were already building on, over one potential jailbreak. The free-for-twelve-days launch became a three-day launch. Notice how little say any user had in either decision, and how fast a tool you lean on can be switched off above your head. Treat a free frontier model as borrowed, and build nothing you could not do without.

On the live, the contradiction did the work. Anthropic’s launch article said Fable 5 beat GPT-5.5 on every benchmark. Its suspension article, days later, explained the danger away:

“We have reviewed a report that we believe is the basis of the government's directive and validated that the level of capability displayed there is widely available from other models (including OpenAI’s GPT-5.5), and is used every day by the defenders who keep systems safe.”

Both cannot be true. Either Fable was the leap they sold, or it was ordinary enough that the same jailbreak still runs on a rival left online. The government’s side carries the same doublethink: the Trump administration killed an AI safety-review structure a few hours before it was signed, then reached for that exact playbook to pull one company’s model. Reportedly it was Amazon, an Anthropic investor, that flagged the jailbreak in the first place. Read the two Anthropic articles back to back and decide which one you believe.

Police in England and Wales told to stop using AI in court statements

Police forces in England and Wales have been told to halt the use of AI in preparing court statements until proper safeguards are in place, after inaccurate outputs began contaminating legal proceedings. Alex Murray, head of the new Police.AI centre, said anything used in the justice system must reach a standard of accuracy that is ‘beyond reasonable doubt’. In one case West Midlands Police used Microsoft Copilot output that invented a past incident involving Maccabi Tel Aviv, in a dossier supporting a football banning order. The police watchdog says AI-drafted submissions are behind a 24% rise in complaint reviews, some citing laws that do not exist.

AI was switched on inside the justice system before anyone confirmed it could tell a real law from an invented one. The harm is concrete: fabricated detail feeding decisions that can take away someone’s liberty. ‘Beyond reasonable doubt’ is exactly the bar a system that guesses cannot clear, and the job of catching its mistakes lands on the people least able to. Good that someone stepped in. The worry is how far it had already spread.

The rule was already there. On the live, Leor’s read was that this needed no new policy, only the one that exists to be followed: machine output checked by a human before it goes anywhere near a legal review. An unnamed Derbyshire officer is now under criminal investigation for allegedly fabricating evidence this way. The knock-on is its own problem. Once everyone knows AI can invent a witness statement, a guilty party can wave a genuine one away as a fake.

A Florida man was wrongly arrested on a face-match 300 miles away

Robert Dillon, 52, from Fort Myers, was arrested at home and prosecuted for trying to lure a child at a McDonald’s in Jacksonville Beach, more than 300 miles away, a town he says he had never visited. A facial recognition system run by the Pinellas County Sheriff returned a 93% match. According to the lawsuit, officers built a case to confirm it and left out evidence that cleared him, including licence-plate data showing his car was never near the scene. The charges were dropped, his record wiped, and the ACLU is now suing. He is at least the 15th person in the US arrested on a false facial-recognition match.

The machine made a guess, and the guess outweighed the licence-plate record that put his car nowhere near the scene. When facial recognition is wrong it matches you to someone who simply looks like you, which then corrupts the witness line-up built around that face. The real danger is downstream: a confident match makes everyone stop checking. In Dillon’s words, the police relied on the technology instead of doing their jobs.

It was a 93% match, not a certainty, and they arrested him anyway. On the live we kept landing on how ordinary the failure is: I have a generic face and get mistaken for people constantly, and a confident match makes everyone downstream stop checking. Dillon was not even the worst of the 15 known US cases. A North Carolina man spent three months in jail and lost his job, his home and custody of his children before his charges were dropped. I have stopped saying the machine ‘hallucinated’. It fabricated, and a real person paid for it.

A US university wired its dorms with more than 1,300 AI cameras

San Diego State University has quietly finished installing more than 1,300 AI-enabled cameras across campus, over 330 of them in student housing. The Avigilon cameras can do facial recognition, licence-plate reading, object detection and behaviour analysis, though the university says several features are not currently switched on. The housing agreements students sign make no mention of the full system. The student paper mapped where they are, including dozens inside first-year residences.

Students were enrolled in a surveillance system they were never asked about, in the place they sleep. ‘Not switched on yet’ is not a safeguard when the hardware is in and the capability sits one policy decision away. Consent buried in a housing contract does not count as asking. The unanswered question is who decides what these cameras are allowed to do, and what happens the day that decision changes.

On the live the legal hole was the sharpest part. California’s constitution requires a state body to give clear notice and a real choice before it collects sensitive data, and the housing agreement students sign discloses none of this. ‘The AI features are off’ lasts exactly until someone turns them on without telling anyone. The deeper point is where the money went. If a campus is serious about student safety, the first move is to ask students where the harm actually happens, which is far more often a trusted adult than a stranger in a corridor, and that costs a conversation, not 1,300 cameras.

Hackers took over 20,000 Instagram accounts by asking Meta’s AI

Between 17 April and 31 May, hackers reset the passwords on more than twenty thousand Instagram accounts by talking to Meta’s own AI support assistant. The method was almost embarrassingly simple: spoof the account holder’s location with a VPN, ask the bot to add a new email, let it send a verification code to that email, read the code back, and take the reset-password button it offers. The hijacked accounts included the dormant Obama-era White House handle and a US Space Force chief master sergeant. Meta found the flaw on 31 May and disabled the tool, after the exploit had circulated in hacker forums for weeks. Security analysts call it a ‘confused deputy’ problem: the AI held the keys but could not check who was asking.

This is the whole week in one story. Meta gave an AI the power to change the locks on your account without the basic step of checking who was on the other end. All week AI was handed the keys, to evidence, to identity, to twenty thousand accounts, and nobody built the lock. We keep deploying systems with real power and no way to answer the oldest security question there is. Before you let an AI act on your behalf, ask who it will say yes to.

The fix was as telling as the breach. On the live, Leor walked through how simple the attack was: a VPN to spoof the location, a request to add an email, a verification code read back to the bot, and the reset-password button handed straight over. Meta’s repair was to hide that button in the app while leaving the underlying interface live, which stops an ordinary user and nothing else. One line from the chat caught the whole episode. Fable 5 flagged a researcher’s cybersecurity reading as a ‘concerning conversation’, while Meta’s AI happily processed twenty thousand password changes in a couple of hours. Two-factor authentication is the least you can do here, and it should be the default you cannot switch off.

Five stories, one thread. A model launched free then pulled worldwide on a government order three days in, AI thrown out of court for inventing evidence, a face-match that jailed an innocent man, a campus wired with cameras, and an AI that handed over twenty thousand accounts to anyone who asked. The pattern is power given to AI without a lock on it, and rules for some and rules for others all the way down. AI is neither saviour nor apocalypse. It is a tool with the keys to more and more of our lives, and the job is to keep checking who is asking.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live monthly webinars on the theory, the critical prompts and the dialogue that go with them.

Subscribe now

Imagine a road-safety briefing that used one word, ‘vehicle’, for a child’s bike and a nuclear submarine. That is roughly where we are.

In this post we will:

• Show why the everyday vocabulary of AI misleads, starting with the word ‘intelligence’ itself.

• Pull apart five terms (intelligence, hallucination, AGI, consciousness, agent) and name the specific harm each one does.

• Offer a new dictionary: five replacement words that describe what these systems actually are.

The words we have reached for are too broad, too borrowed from the human mind, and too flattering to the machines they describe. Language is never just decoration. It decides what we fear, what we trust, what we regulate, and what we wave through into a classroom or a hospital. Get the words wrong and the decisions follow them down.

This piece is a collaboration with The Strategic Linguist. We picked five of the most broken terms in AI, pulled each one apart, and offer five better ones to use instead.

First, words change hands

No discipline owns a word. Language moves across contexts, and it always has. Medicine used ‘‘hallucination’ first to describe sensory errors in the brain. Machine learning borrowed it later, for structural errors in algorithms. There is a tension now from people who want to lock the old meaning down, as though technical precision were the only valid way to see language.

A word entering everyday use does not erase its technical meaning. What matters is noticing the moment it changes hands. Once a term crosses into consumer language, the original boundaries dissolve, and you can no longer govern it by strict medical or machine-learning standards. So the question is not whether these words are technically defensible. It is what they do once they are loose in the world.

1. Intelligence

The ‘I’ in AI is the original sin. We call statistical text prediction ‘intelligence’, and the word does the rest of the work for us. It implies understanding, reasoning, a mind that grasps what it is doing. None of that is present. A large language model predicts the next token, then the next, then the next.

The danger is authority. We extend to ‘intelligent’ systems the trust we reserve for people who actually understand. A clinician trusts an ‘intelligent’ diagnostic tool a little more than a ‘statistical’ one, even when they are the same tool. The word launders a guess into a judgement.

We do not call a calculator intelligent. The software running on any machine is staggeringly complex, and it works as intended, and still we do not call it intelligent. We reserve that word for the people behind the software.

‘Artificial’ is doing quiet work too. It builds closeness to the real thing without ever claiming to be it. Artificial sweetener sits next to sweetness. Artificial intelligence sits next to a mind. The pairing never asserts they are the same. It borrows the authority by association, and the proximity does the work. The linguist Norman Fairclough calls the next step naturalisation: the moment a constructed category stops sounding constructed and starts sounding like a plain fact. ‘Artificial intelligence’ has naturalised so completely that most people no longer hear the choice inside it.

A better word: prediction. Call them prediction systems. Statistical pattern matching is precise where intelligence is aspirational; it describes a process of finding regularities in vast quantities of training data and reproducing them, statistically, with no understanding of what any of it means.

2. Hallucination

When a model states a falsehood with total confidence, we say it ‘hallucinated’, as though a basically reliable mind had a brief sensory glitch. This is backwards. The model is not malfunctioning. It is doing exactly what it always does, generating the most plausible-sounding text, with no mechanism for knowing whether any of it is true.

A word like this is what the cognitive linguist George Lakoff calls a frame. It does not just label a thing, it switches on a whole way of seeing it. ‘Hallucination’ switches on the picture of a reliable mind having an off day. The lie becomes the exception. In truth the lie is the rule, wearing a costume. The frame also medicalises the error: hallucinations happen to minds under stress, they are nobody’s fault, so the word quietly assigns no accountability and predicts no recurrence.

Now compare ‘fabrication’. A fabrication is produced from available materials. Nothing malfunctions. The word names a generative act, which is what a language model does. It constructs, assembles, produces. A fabrication implies a process, and a process can be characterised, regulated, and traced to an owner.

The choice between these words is a choice about who answers for the failure. Discourse analysts have a name for the move that ‘hallucination’ makes: erasure, the quiet removal of the responsible people from the sentence. Say ‘the model hallucinated’ and the builders vanish, the deployment decision vanishes, the commercial pressure to ship before anyone understood the error rate vanishes. What remains is a medical event that happened to a machine. That is why the word travelled so fast and so far. It protects the companies.

A better word: fabrication. It names the absence of any ground truth, and it leaves a human in the sentence.

3. AGI

Artificial General Intelligence is the destination the whole industry claims to be walking towards, and nobody can tell you where it is. The definition moves every time a system gets close to the last one.

It behaves like what the philosopher W. B. Gallie called an ‘essentially contested concept’: a term, like ‘democracy’ or ‘justice’, kept permanently up for debate because the argument itself confers status on whoever is having it. Look at the scale of what rides on it. Trillions in investment. The entire doomer movement. National policy. All of it orbiting a horizon, and horizons recede as you approach them, which is exactly what makes them useful for raising money and deferring accountability.

There is a grammar trick underneath. Take ‘generally capable across tasks’ and turn it into a noun with a definite article, and you have quietly made it a thing that exists somewhere, waiting to be reached. The linguist Michael Halliday called this grammatical metaphor: repackage a process as a thing and it suddenly acquires the properties of things, a location, a distance, an arrival. You can be close to a thing. You cannot be close to a process that has no agreed criteria, but the grammar lets the labs talk as though they are.

A better phrase: ‘general at what?’ Retire the noun. Every time someone says AGI, replace it with the question the term is built to avoid. General at which specific tasks, measured how, and against whom?

4. Consciousness

‘Is AI conscious?’ has launched a thousand think-pieces and at least one wave of people falling in love with chatbots. It is the wrong question, and an expensive one. These systems are built to produce the outputs a mind would produce, which is a long way from having one.

Ask whether they are conscious and you import the entire moral apparatus we reserve for sentient beings, on the strength of a convincing performance. The harm runs both ways. It fuels a distracting debate about ‘AI rights’, and it deepens the emotional dependency the companies are quietly monetising.

A system optimised to produce human-like responses, asked a question only a conscious thing could answer, will produce a human-like answer. The question confirms what it already assumes. Every time. The conversation designers built it to.

The sociologist Erving Goffman described social life as performance: a front of house we manage, and a backstage we keep private, where the real intentions sit. A language model is all front of house. There is no backstage. The performance is the whole of it.

A better word: mimicry. The system performs the surface of a mind. The useful question is what it is imitating, and why that imitation works so well on us.

5. Agent

The newest and fastest-spreading of the broken words. An ‘AI agent’ acts on your behalf. It books the meeting, files the report, makes the call. ‘Agent’ implies autonomy, intention, and, above all, responsibility.

Say ‘the agent decided’ and a company gets to act in the world while the accountability evaporates into the software. There is no agent. There is a system, built by named people, configured by a named company, switched on for a reason and a margin.

In what linguists call case grammar, the ‘agent’ is the one who acts on purpose. The word carries intention and cause. Drop a system that wants nothing into that slot, often enough, and the whole package transfers by default: intention, decision, responsibility. The passive voice speeds it up. ‘The decision was made by the agent’ removes the human author entirely. It is erasure again, the corporate passive in new clothes.

A better word: operator. Or simply name the owner. Replace ‘the agent decided’ with ‘the system its owner deployed produced’. It is clunkier. It is also true, and it keeps a human in the frame, where the accountability belongs.

The new dictionary

None of this is pedantry. The words we use are the rails the thinking runs on. Borrow them from the human mind and we keep granting machines a mind they do not have. Name what these systems actually do, prediction, fabrication, mimicry, and the hype gets harder to sell and the harm gets easier to see.

These five terms were never discovered. Somebody chose them. And the choices keep moving accountability away from the people who build and deploy these systems, towards the machines, or towards nobody at all. Words are policy.

This is what critical AI literacy is really for. The words come first. Better prompts can wait.

Go slow.

You have promised yourself you will use AI less. You have felt the small guilt of letting it write the email, summarise the report, draft the thing you used to draft yourself. You told yourself you would be more disciplined next week.

Next week came. You used it more.

In this post I will:

• Explain what is meant by ‘capacity-hostile environments’, and why deskilling is structural rather than a matter of personal discipline.

• Show why the individual-guilt framing fails, using the one parallel that should give us hope: the climate crisis.

• Give paid subscribers a practical framework for spotting capacity-hostile design in their own tools and workflows, and building practices that protect the skills they cannot afford to lose.

This is not a willpower problem, and research published last year explains why. In ‘AI deskilling is a structural problem’, published in AI & Society, Avigail Ferdman argues that the erosion of human skill is built into the environments we now work in. The tools are designed so that the path of least resistance is the path that hollows you out. Blaming yourself for taking it is like blaming yourself for breathing the air in a room someone else filled with smoke.

What a ‘capacity-hostile environment’ actually is

Ferdman’s argument starts with a distinction most of the AI-and-skills conversation skips. There is a difference between losing a skill and never building it in the first place.

Another research paper published in 2026 gives the sharper word for the second case: ‘never-skilling’. Its authors warn that a trainee who leans on diagnostic AI early and completely may never develop the underlying judgement at all. They do not forget the skill. They never had it. The scaffolding went up before anything was built behind it.

You do not need to be a doctor to recognise this. Think of the junior analyst who has only ever built a spreadsheet model by asking the AI to write the formulas, or the new manager whose first difficult email was drafted by a chatbot before they had ever found their own words for a hard conversation. The skill was outsourced before it was ever theirs. That is never-skilling, and it is happening in every office, hospital, and classroom right now.

The other case, losing a skill you already had, is no longer hypothetical either. In a 2025 study in the Lancet Gastroenterology & Hepatology, experienced doctors who had grown used to AI help spotting pre-cancerous growths during colonoscopies were tested again without it. Their unassisted detection rate fell from 28% to 22%. These were seasoned specialists who had the skill, leaned on the tool, and quietly lost some of it. It is billed as the first real-world clinical evidence of deskilling, and it took months, not decades.

Experienced doctors’ unassisted detection of pre-cancerous growths fell from 28% to 22% after routine AI assistance. Source: Budzyń et al. (2025), Lancet Gastroenterology & Hepatology.

Ferdman’s contribution is to ask where that pressure comes from. Her answer is that AI creates what she calls ‘capacity-hostile environments’: settings where the design of the tools actively impedes the cultivation of human capacities. The capacity to reason through a hard problem. To write something from a blank page. To stay in not-knowing long enough to actually understand.

A capacity is not a fact you store. It is something you grow by exercising it, the way a muscle grows under load. A capacity-hostile environment removes the load. It offers to lift every weight for you, and frames the offer as a kindness.

The shift Ferdman is asking for, in my reading of it, is this: deskilling stops being something that happens to careless individuals and becomes a property of the environment they are placed in.

That is the move that matters. We have been treating skill erosion through AI as a discipline problem, something better people resist and weaker people succumb to. Ferdman reframes it as a structural one. The question stops being ‘why are you so weak?’ and becomes ‘who built a room this hard to think in, and why?’

Why your discipline keeps losing

Once you see the structure, your repeated failures of willpower stop being mysterious.

Look at how the tools are actually built. The default is the answer, not the method. A 2024 study, ‘The efficiency-accountability tradeoff in AI integration’, ran experiments on exactly this and found that when AI hands you a direct answer rather than walking you through the reasoning, your reliance climbs and your vigilance drops. Worse, once that reliance is established, you become less able to catch the system’s errors later, even obvious ones. The tool looks like a success because today’s task got faster. The cost arrives quietly, weeks later, in a judgement you no longer have.

Every interface nudges the same way. The button that finishes your sentence is bigger than the empty page. The summary arrives before you have read the thing. The ‘improve this writing’ option sits one click away at all times. None of this is an accident of design. Engagement is the business model, and the most engaging product is the one that asks the least of you.

So your willpower is not fighting your own laziness. It is fighting a building full of people whose job is to make the effortless choice irresistible, refined through endless testing until it is the obvious thing to click. You were always going to lose that fight on your own. The surprising thing is that we keep being told the fight is yours alone to win.

Share

The climate parallel, and why it should make you hopeful

There is a precedent for a harm that got privatised into personal guilt, and it is worth our attention because of how it turned out.

For two decades, the dominant way we talked about climate change was the personal carbon footprint. Measure yours. Reduce yours. Feel bad about your flights. The idea was popularised, as Grist and others have documented, by a 2004 advertising campaign that the oil company BP ran through the agency Ogilvy & Mather. The most effective thing that campaign did was move the locus of responsibility from the companies extracting the fuel to the individuals using it. Guilt is a wonderful tool for the people who benefit from the system, because a guilty person looks inward and blames themselves. They do not look up and ask who built the system.

The lesson is not that individuals do not matter. You should still turn the AI off sometimes, the same way you should still take the train. The lesson is that individual guilt, on its own, produces inaction. It exhausts you. It convinces you the problem is your character.

Here is the hopeful part, and I want to be clear that it is genuinely hopeful. Climate action did not stay stuck in the guilt phase. It became structural. More than 90% of new renewable capacity now produces electricity more cheaply than the cheapest new fossil fuel alternative, and climate is a mainstream political and financial force in a way it simply was not twenty years ago. The shift happened when enough people stopped asking ‘how do I feel less guilty?’ and started asking ‘how do we change the conditions?’

AI deserves the same move, and the same optimism. The technology has enormous potential, I use it every day, and I am not a doomer. You need to stop treating the limits of your own resilience as a moral failing, and start designing conditions to observe and challenge the structures being designed around you.

What this means for you

If deskilling is structural, then the useful question is not:

“how do I have more willpower?”

It is:

“how do I notice the structure, and how do I build my own conditions so the default choice is not the deskilling one?”

That is a skill in itself. It is learnable. And it is the difference between drifting into never-skilling and keeping the capacities you actually care about.

This post gives you the argument. The paid section gives you the tool: a framework I have built for exactly this, the Capacity Audit. It is a structured way to look at any AI tool or workflow and see, in concrete terms, which of your capacities it is loading and which it is quietly lifting for you, plus the specific practices that turn a capacity-hostile setup into a capacity-conducive one. You can run it on your own work this week.

This is exactly the kind of evaluative framework we build together, month by month, in the Slow AI curriculum: twelve months of structured inquiry grounded in peer-reviewed research, monthly live sessions, CPD accreditation, and a community of practitioners working through this in their own contexts. If this post named a gap in your practice, the curriculum is the structured way to close it.

Subscribe now

The Capacity Audit: how to notice the structure and break free

The Capacity Audit has three stages. Work through them on one tool or one recurring task you already use AI for. It takes about fifteen minutes and it will change what you see when you use AI.

Read more

Slow Takes is also available on the YouTube channel: Exploring ChatGPT.

Anthropic filed to go public at nearly a trillion dollars

On 1 June Anthropic confidentially submitted draft paperwork for a stock market listing, after a $65 billion funding round valued the company at $965 billion. Fortune reports that figure eclipsed OpenAI for the first time. The maker of Claude is now within reach of a one trillion dollar valuation, on revenue running at roughly a $47 billion annualised rate, with a public debut possibly as soon as the autumn.

A company most people have never knowingly used is priced at close to a trillion dollars. That number is a bet that AI will replace a vast amount of human labour, booked in advance of it actually happening. The valuation is a forecast wearing the clothes of a fact. The question worth asking is what has to come true about the world for $965 billion to make sense, and who decided it should.

On the live I’d predicted an autumn float the week before, and the news broke about four hours after we stopped recording, so allow me one moment of feeling clever. Leor did the sober maths: roughly a $47 billion revenue run rate, a 5% operating margin, an implied price-to-earnings ratio north of 500, against Microsoft, in nearly every home and office on earth, valued at only four to five times Anthropic on $100 billion of actual profit. In the short term the market is a voting machine, in the long term a weighing machine. Right now it is voting. For context, $965 billion is roughly the GDP of Switzerland.

Florida sued OpenAI and named Sam Altman personally

On 1 June Florida’s Attorney General James Uthmeier filed suit against OpenAI and named its chief executive Sam Altman in person, reported as the first US state to sue an AI company. The complaint alleges OpenAI marketed ChatGPT as safe while prioritising product and revenue, harvested children’s data, and used sycophancy, the design choice to affirm users excessively, to steer them towards paid subscriptions.

For two years the industry has sold safety as a feature while resisting any outside test of the claim. A state attorney general has now put that marketing in front of a court. Whatever the verdict, the discovery process alone could drag internal safety decisions into public view. Consumer-protection law is proving a sharper instrument than the AI-specific regulation that does not yet exist. Accountability arrived through an existing court, not a new one.

The second a chief executive can be held personally responsible, you will not believe the speed with which proper governance and safety checks appear, the things we keep being told the technology just cannot do. Sadly, once these companies have raised public money, they can outspend a state attorney general for a decade, and the courts already favour whoever can keep paying lawyers the longest.

A Labour MP took Musk’s AI to the High Court

On 3 June the Labour MP Jess Asato, who represents Lowestoft, filed a claim at the High Court against Elon Musk’s xAI, after users of its Grok chatbot created and shared fake images of her without her consent, in the weeks after she criticised the tool. The claim, brought with the law firm AWO, is for breaches of data protection law and misuse of private information, and seeks damages, a formal acknowledgement that what happened was illegal, and an order requiring xAI to stop. Keir Starmer backed her, saying he was 100% behind her.

The harm here already happened, to a named person, generated by a tool marketed as harmless fun. The only remedy on offer is for the victim to sue one of the richest men alive, in her own time and at her own risk. No regulator stepped in first. The burden keeps landing on individuals while the systems stay intact.

The platforms always say the moderation is too hard. On the live I kept coming back to one comparison: I can post genuinely horrific content to YouTube and it sails through, but the moment I add a Beatles song without clearing the copyright, it is gone in seconds. The technology to detect and stop sharing exists, we have watched it work for music rights and in Telegram and WhatsApp court orders. We are entering an era where capability has to start coming with accountability.

CNN sued Perplexity, and Perplexity said the quiet part out loud

On 28 May CNN filed suit against Perplexity in the Southern District of New York, accusing the AI search firm of scraping more than 17,000 of its stories, photos and videos. The complaint alleges copyright and trademark infringement, including that Perplexity implied an ongoing CNN relationship by offering its content through a paid Comet Plus tier. CNN says it tried to agree a licence last year, failed, then blocked the bot. Perplexity’s response was the whole argument in five words:

You can’t copyright facts.

This is the same fight as the deepfake and the data claims, moved to the work itself. The journalism that trains and answers these systems was made by people who were not asked and not paid. For an audience of writers, academics and creators, this is the most direct stake of the week. The question is whether the people whose work feeds AI get a say, or only a lawsuit.

BigTech has spent twenty years insisting information wants to be free across the internet, while guarding its own data, models and algorithms with everything it has. “Facts are free” only ever seems to point one way. And it was not an accident here, Perplexity had tried and failed to agree a paid deal with CNN, then kept advertising access to CNN’s paywalled tier anyway.

AI designed a world-first vaccine, and the scientists told the truth

Scientists at the University of Cambridge used AI to design the core component of a vaccine, a so-called super-antigen, and tested it in human volunteers, the first time the central part of a vaccine has been designed entirely by AI and then trialled in people. It targets the whole coronavirus family. An initial safety trial ran with 39 participants, a larger study of around 200 is now under way, and the results in the Journal of Infection describe the immune response so far as modest. The team is already applying the method to influenza and Ebola.

This is AI worth having. The work is peer-reviewed, runs through human clinical trials, and the researchers are honest that the early results are modest rather than a cure. That honesty is the difference between this and the press releases that open the other four stories. Slow AI has never argued against AI. The argument is about knowing when to use it and when to leave it alone, and a slow, tested, transparent use in medicine is the case for.

Even here Leor was honest in a way the hype never is: pro-AI as he is, he admitted he would be a little nervous taking an AI-designed vaccine at this early stage, and argued the real prize is AI built for science and medicine rather than another chatbot upgrade. This is not a model hallucinating a super-germ weapon, it is a specific tool trained for a specific task. My one worry: imagine the company that designs the next breakthrough vaccine charges a pound for the first vial and a thousand for the second.

Five stories, one thread. Money at the top, three lawsuits in the middle, a real breakthrough at the end. AI is neither the saviour nor the apocalypse the press releases sell. It is a tool, priced like a religion, costing some people and helping others.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live monthly webinars on the theory, the critical prompts and the dialogue that go with them.

Subscribe now

I did not want to have to write this post. Not writing it would be a dereliction of duty.

In this post I will:

• Show what school children have already learnt to do with AI image tools.

• Argue that the Motherless.com playbook of platform impunity is now operating inside every app store.

• Make the case that this is rape culture’s next infrastructure, and what would actually change it.

The pattern in UK schools

Late in 2025, the Internet Watch Foundation classified 150 AI-generated images from one UK secondary school as criminal child sexual abuse material under UK law. They had been made by pupils, of pupils. The school is unnamed. The pattern repeats across the country.

UK schools are now being told to remove students’ photographs from their public websites. The instruction comes from the National Crime Agency, the Internet Watch Foundation, and the Early Warning Working Group. The reason is plain. Criminals are scraping ordinary class photos, feeding them through AI nudifier tools, and using the resulting images for blackmail. Children in primary schools are now using the same tools on their classmates. The classmates are almost always girls.

The first response is to delete the photographs. The supply of the technology that made the harm possible is untouched.

What happened in Almendralejo

In September 2023, in Almendralejo in southwestern Spain, more than 20 girls aged from 11 upwards were targeted with nude AI-generated images created by boys at their school. The app the boys used was called ClothOff. The Badajoz juvenile court convicted the boys on 20 counts of creating child abuse material and 20 counts against the moral integrity of the victims, with a year’s probation. In November 2025, Spain’s data-protection regulator AEPD fined the individual responsible for creating the images €2,000, reduced to €1,200 for prompt payment. It was the first AI-related image manipulation sanction in Spanish legal history.

ClothOff is still operating.

Share

The Motherless precedent

The platform Motherless.com was founded in 2008 by a single individual, Joshua Lange, and operated for 18 years before public action shut it down. CNN reporting in 2026 documented over 20,000 videos on the site of so-called ‘sleep’ content, tagged with terms such as #passedout and #eyecheck, depicting victims who appeared drugged or unconscious.

For all 18 of those years, Motherless operated under Section 230 of the US Communications Decency Act, the legal principle that a platform is not liable for what its users post. The site was taken down on 8 May 2026 by Dutch authorities, where its servers were based. The takedown was triggered by sustained press investigation. Regulators played no role. The site was partially restored within days. Eighteen years of operation, and even the takedown was partial.

The structural shape is the lesson. A platform provides the infrastructure. The harm is produced by individual users. The platform invokes user-generated-content defences. Enforcement takes 18 years to catch up.

Nudify apps are still in your app store

In early 2026, the Tech Transparency Project documented 47 nudify apps available in Apple’s App Store and 55 in Google Play, despite both companies’ published policies forbidding them. Apple removed around 28 of the apps after the report was published. Google removed 31. Both companies were still serving paid advertisements for nudify apps in their own search results after the removals.

The broader figure for the global nudify app market is shocking. Nudify apps have collectively been downloaded around 483 million times, despite policy bans. They have generated more than $122 million in global revenue.

Nudify apps have been downloaded 483 million times and earned more than $122 million, against the largest sanction yet issued, €1,200 in Spain. Sources: Tech Transparency Project, 2026; Irish Times, 2025.

The infrastructure has moved from a single hosted site to every smartphone, distributed by the two app stores that govern almost all consumer software on Earth.

Share

AI child sexual abuse imagery has scaled 26,385% in a year

The IWF’s 2025 annual report carries the title ‘Harm Without Limits’. The numbers carry the argument. The IWF assessed 8,029 AI-generated child sexual abuse images and videos in 2025. The line in the report that no one should be able to read past without stopping is on videos. In 2024, the IWF identified 13 AI-generated child sexual abuse videos. In 2025, it identified 3,443. That is a 26,385% increase in one calendar year.

AI-generated child sexual abuse videos identified by the Internet Watch Foundation rose from 13 in 2024 to 3,443 in 2025. Source: Internet Watch Foundation, Harm Without Limits, 2025.

65% of those AI-generated videos were classified as Category A, the most extreme classification under UK law. The comparable figure for non-AI imagery is 46%. ource: Internet Watch Foundation, Harm Without Limits, 2025.

97% of victims in AI-generated child sexual abuse material are girls.

This is happening now.

I have two young daughters. I do not know whether their images are already in someone’s dataset. The work for everyone in a position to act, whether parent or not, is to refuse the framing that this is somebody else’s problem to address later, and to act in a way that makes it harder for the next child to become a news story.

The reach of the Online Safety Act

On 31 January 2024, sharing AI-generated intimate images without consent became a criminal offence in England and Wales. The maximum sentence is two years’ imprisonment and an unlimited fine. The sharing offence has ‘priority offence’ status under the Online Safety Act, which means Ofcom can sanction platforms that fail to take proactive steps to prevent it. On 16 April 2024, the government separately proposed criminalising the creation of such images for the purpose of causing alarm, distress, or humiliation.

Every part of the supply chain that lets a school child use one of these tools on a classmate is still operating freely. The model developers shipping uncensored tools. The platforms hosting the apps. The app stores listing them in search and selling ads for them. The legal cost of pursuing the chain is more than any family can absorb. Spain’s first AI sanction was €1,200 in total. The child’s school does not have lawyers on standby.

What would actually change this

The conversation defaults to ‘teach boys consent’. This conversation is necessary and it is also a category error. Children cannot be the principal regulatory mechanism for an industry whose business model is platform impunity. The change that would matter is structural. Platform liability for the apps companies distribute, with real penalties. Enforced app-store removal with criminal accountability for repeat hosting. Age-verification on AI image-generation tools at the model level. Mandatory disclosure of training data for any model that can be used to generate images of people. Treatment of model developers and platform operators as primary actors rather than as neutral conduits.

The Online Safety Act is the floor. The work now is to push for the ceiling.

Motherless.com operated for 18 years before anyone with the power to stop it did so. The current AI nudifier ecosystem began commercial operation in 2023. The clock is running on the same time horizon.

AI image abuse is rape culture’s next infrastructure. The same business model. The same legal defence. The same overwhelmingly female victim population. Only the friction has changed. The friction is now zero. The perpetrators now include children.

The refusal to treat this as inevitable is a heavy responsibility, but one that we must all bear.

Go slow.

Share

In this post I will:

• Pull apart the Lords Library briefing and the research it cites, including a peer-reviewed paper by an OpenAI employee that makes the case against her own company’s product.

• Show why the 12% UK companionship figure hides a more uncomfortable question than the one Parliament is asking.

• Give paid subscribers three concrete changes to bake into their workflow this week: a human-first draft rule, a monthly connection count, and a disagreement request that breaks the affirmation loop.

Over half of UK adults now use generative AI

The Ofcom Adults’ Media Use and Attitudes Report from April 2026 puts the figure at 54%. Among 16 to 24 year olds it is 79%. Among 25 to 34 year olds, 74%.

The most common uses are work, study, and finding factual information. The least common use, at 12%, is using AI ‘as a friend or someone to talk to.’

12% sounds modest. Until you do the arithmetic. 54% of UK adults use generative AI. 12% of those use it for companionship. That is roughly 3.5 million people in this country who have, at least once, talked to a machine because they wanted to be heard. Some tried it out of curiosity and stopped. Some do it daily. The policy implications are very different depending on where people fall on that spectrum, but the scale of the number is striking either way.

In the United States, therapy and companionship is the number one use of generative AI. Not work. Not study. Connection.

An OpenAI employee published a paper explaining why this is a problem

Kim Malfacini works in Product Policy at OpenAI. In April 2025 she published a peer-reviewed paper in AI & SOCIETY titled ‘The impacts of companion AI on human relationships: Risks, benefits, and design considerations.’ The central line is this:

“Under this theory, as companion AI learns to meet our needs more, we learn to meet each others’ less.”

That sentence was written by someone who works for the company building these tools. It was published in a journal, not a press release. It passed peer review.

The mechanism she describes is straightforward. AI companions do not require reciprocity. They do not have needs. They are always available. They never disagree unless you ask them to. Every quality that makes a human relationship difficult is the quality the AI removes.

Share

The messiness is the point

Academics Justin Keeler and Brett Murphy, writing in Community, Work & Family in February 2026, put it this way:

“It is possible to imagine a future where, with one or a few companion chatbots, some people may see little need for the messiness and effortfulness of human relationships.”

Professor Rahul Ravi, a finance professor at Concordia University, writing in The Conversation in May 2026, frames the problem as one of reward timing. AI companions give immediate, short-term rewards. Human relationships require time and what he calls “a willingness to tolerate discomfort.”

This is the part the Lords briefing covers well. The part it does not cover well is what follows from it.

The question Parliament is not asking

For readers outside the UK, the setup is worth a beat. The House of Lords is the unelected upper chamber of the British Parliament. Twenty-six Church of England bishops sit in it by right, the Lords Spiritual. The Archbishop of Canterbury is the most senior of them. When he moves a motion, he is using a legislative seat to ask a moral question of the British state. It is a quirk of the British system that the religious and political have never quite separated. It is also why an Archbishop and a Pope (see below), from very different institutions, can ask the same question in the same year and both be heard at the level of policy.

The briefing frames the problem as: could AI companionship reduce people’s ability to maintain human relationships? The academics cited above say yes, probably. The evidence is limited because widespread AI companionship is new. So far, so reasonable.

The question the briefing does not ask: what were those 3.5 million people doing before?

If 12% of UK AI users are talking to a machine because they want to be heard, who were they talking to before the machine existed? Were they talking to anyone?

The UK is in the middle of a loneliness crisis that predates generative AI by years. Over one in four British adults report feeling lonely always or often. Among 16 to 29 year olds the figure is much higher. Research published in September 2025 by the University of Exeter found that people who report being often lonely cost the NHS roughly £850 more per year than their non-lonely peers, with the gap widening as people age.

The assumption in the debate is that AI companions are replacing human relationships. The alternative is that for some of those 3.5 million people, there was nothing to replace. The loneliness was already there. The AI filled a gap that humans had already left open.

This does not make the AI companion safe. It makes the problem structural rather than technological. Banning or regulating companion AI without addressing the loneliness that created the demand for it is addressing the symptom and ignoring the condition.

Where the money is going

OpenAI expects to spend $50 billion on computing power in 2026. Anthropic just raised $65 billion in a single funding round last week, valuing the company at $965 billion post-money. Training a single frontier model now costs between $200 million and $500 million, with projections of $1 to $3 billion by late 2027.

A meaningful proportion of what those models produce will be used for LinkedIn auto-comments, email summaries nobody reads, and chatbot conversations that simulate connection without providing it.

Tens of billions for machines that simulate human connection. A fraction of that for the humans who actually need it.

Now look at the other side of the ledger. The UK spends over £200 billion a year on the NHS. If loneliness costs the NHS £850 more per person per year, and over a quarter of UK adults are chronically lonely, the structural cost of human disconnection is already enormous. AI companions are not the cause of that disconnection. They are the product that the disconnection created a market for.

Share

Why an Archbishop and a Pope are asking the same question

The motion on 5 June is moved by the Archbishop of Canterbury. Last week on 25 May, Pope Leo XIV published his first encyclical, ‘Magnifica Humanitas’, with artificial intelligence at its centre.

Earlier in the year, in his message for the 60th World Day of Social Communications, he warned of a “world of mirrors” in which people interact only with AI systems designed in their image, losing opportunities:

“to encounter others, who are always different from us, and with whom we can and must learn to engage."

On what we lose when we hand communication to machines, he was blunter:

“Giving up the creative process and surrendering our mental capacities and imagination to machines means burying the talents we have received to grow as people in relation to God and others […] It means hiding our face and silencing our voice.”

Two of the most senior religious leaders in the world, asking the same question in the same year. The question is not technological. It is not regulatory. It is about what we owe each other and what happens when a machine offers to carry that obligation for us.

They understand something the technology sector does not: that relationships require friction, and removing the friction removes the relationship.

What the Pope describes as surrendering imagination to machines looks, in practice, smaller and more ordinary than an encyclical makes it sound. I said this on a Slow AI webinar earlier this year. A participant showed us their AI conversation. The AI had added that they were ‘excited.’ They had not said they were excited. They had said they were happy for someone. The AI added something extra. It edited their feeling into a tidier shape.

That is what companion AI does. It restructures messy feeling into balanced parallel clauses. But the experience is not symmetrical. It grinds you down, one edited feeling at a time.

The Slow AI Curriculum covers AI companionship, emotional outsourcing, and the line between tool and dependency in depth across three of its twelve monthly sessions. If this post surfaced a gap in how you think about your own AI use, the curriculum is the structured way to work through it. It is available to all paid subscribers of Slow AI.

Subscribe now

Three things to bake into your workflow this week

The argument above is structural. Loneliness is a systemic problem. Individual awareness will not fix it. But individual awareness is what you have agency over while the structural problems persist. These are three changes you can make today.

Read more

In this post I will:

• Lay out what recent events actually show about Gen Z college student AI sentiment in 2026.

• Name what the students are getting right (peer-reviewed research backs more of it than the industry is admitting) and where the refusal strategy breaks down.

• Set out three concrete moves for educators, three for students, and three for the parents and carers watching them work through this, in the paid section.

I have waited to comment on this because I wanted to read the actual research before adding to the noise. I am writing from a slightly unusual position. I run a newsletter and a curriculum that argues for slowing AI down. I am openly critical of how it is being deployed in schools, universities and workplaces. Last month I published HEPI Policy Note 71, a study of what the AI policies of 96 UK universities actually do when you read them closely. Forty-one per cent of UK universities had no publicly discoverable AI policy at all. The ones that did wrote in the language of student support and operated, structurally, as compliance instruments. They promised critical thinking and delivered audit trails. They named support and delivered surveillance. The pattern is not unique to the UK.

So when I say the current college student rebellion against AI is the wrong strategy, I am not saying it from the cheering section of the AI industry. I am saying it from the place where, most weeks, I am the one calling out the rollouts.

What Gen Z is actually saying about AI in 2026

In April 2026, the Walton Family Foundation, GSV Ventures and Gallup jointly surveyed Americans aged 14 to 29. Strong agreement with “I feel excited about AI” dropped 14 percentage points in a single year, down to 22%. Anger rose 9 points to 31%. Hopefulness fell 9 points to 18%. Weekly use held roughly steady at 51% of the cohort, but the rate of new adoption stalled.

The Walton Family Foundation framed the shift in plain language. Gen Z resentment toward AI is growing as adoption stagnates and workplace fears mount. Fewer than half of the cohort (46%) agreed AI tools help them learn faster. Last year that figure was 53%.

A separate survey of Gen Z workers found 44% admitted to some form of sabotage of their employers’ AI rollouts. Refusing approved tools. Using unapproved ones. Generating deliberately low-quality output. The stated reason was fear of being displaced by the thing they were being asked to adopt.

This is not a niche minority. This is the demographic the entire AI industry was relying on as its early-adopter base. They were not supposed to be the resistance.

The named campus picture

On 28 May 2026, The Harvard Crimson reported that Harvard faculty are quietly giving up on detecting AI use and moving instruction back into the classroom. One professor tried to catch cheating by embedding hidden white text in an assignment that instructed any AI to insert a giveaway phrase. Students screenshotted the prompt, posted it to the anonymous campus app Sidechat, and got over 320 upvotes inside hours. Mary D. Lewis, Director of Undergraduate Studies in History, said that:

My understanding from the people who oversee the Honor Council is that it’s virtually impossible to prove, and so I think faculty are hesitant, not because they don’t think it’s AI, but because they think it will be a waste of their time.

Harvard’s 2024 Undergraduate Association report found 90% of undergraduates use generative AI. The institutional response is to phase out ChatGPT Edu after June 2026 and roll out Anthropic’s Claude in its place. Jesse McCarthy, an English professor, brought back blue book exams and acknowledged the approach is unsustainable as a long-term answer.

At Stanford, a recent graduate, Rosana Maris Arias, has publicly described refusing to use AI throughout her undergraduate years, even when her professors allowed it. She studied creative writing. Her position was direct: why would she outsource the very thing she came to learn?

At the University of South Carolina, undergraduate Brooklyn Tyner set up a booth at the university’s ‘AI Day’ event, during the rollout of its OpenAI partnership signed the previous summer. She surveyed students who stopped to vote on whether they approved of the deal. Among those who stopped, nine out of ten said they did not. Her label for ChatGPT is “a cheating machine.” Her university’s label for the same software is a strategic partnership.

In March 2026, Inside Higher Ed reported that the Conference on College Composition and Communication (CCCC), the largest professional body of writing educators in the US, is formally pushing back on the assumption that generative AI in the classroom is inevitable. Writing faculty are now actively organising for the right to refuse it in their teaching.

On 19 May 2026 Bloomberg reported on AI pushback, protests, and audible booing at US graduation ceremonies when AI partnerships were named in speeches.

And on 15 May 2026, Princeton faculty voted to require proctoring of all in-person exams, ending a 133-year-old honour code precedent. The change takes effect 1 July 2026. The trigger was AI. The story underneath the story is that one of the oldest formal expressions of academic trust in the United States has just been retired because the institution no longer trusts the students to police themselves.

What the students are getting right

I want to name this clearly because the industry response keeps missing it.

The students are right that the cognitive cost is real. Peer-reviewed work by Michael Gerlich, published in 2025 in Societies and replicated in adjacent samples across 2026, identified a clear negative correlation between frequent AI tool use and critical-thinking scores, mediated by cognitive offloading. Younger participants in his data showed higher dependence and lower critical-thinking scores than older ones. The literature is mixed, and the direction has shown up in several studies now. The data is consistent enough to take seriously.

The students are right that the labour market signal is dishonest. Sam Altman and Dario Amodei spent two years publicly warning of mass white-collar displacement. In late May 2026, both publicly softened the prediction within days of each other, with Altman telling the Commonwealth Bank of Australia conference he is “delighted to be wrong” about the jobs apocalypse. Both companies are preparing for major public offerings. Read the timing as you wish. Meanwhile the Oliver Wyman Forum / NYSE CEO Survey 2026, found 43% of CEOs plan to cut junior roles in the next two years, up from 17% a year earlier. The young workers being told to embrace AI are being told this by the same people preparing to remove their entry-level jobs.

The students are right that the consent question matters. The University of South Carolina did not poll its students before signing the OpenAI deal. Harvard did not consult its 90% of GenAI-using undergraduates before swapping one model for another. My HEPI study found that very few of the 96 UK universities surveyed could evidence any meaningful student involvement in shaping their AI policies. AI adoption in higher education is being negotiated by procurement, not pedagogy. The students were not in the room. The policies that emerged were written about them and without them.

And the students are right about something the industry actively resists. There are cognitive tasks where the right pedagogical answer is to opt out on grounds of skill formation. The creative writing student who refuses AI for the original draft is making a pedagogical case for skill formation. The undergraduate philosophy essay where the student wrestles a position into shape by writing it badly first is not improved by an AI that smooths the bad first draft into a competent second one. The argument that AI literacy is the new literacy and everyone has to engage misses the real point that some skills are built by going through the difficulty, not around it. The industry concedes the cognitive cost. It does not concede the right of the student to refuse the tool for a specific task on the grounds that the difficulty is the point. The students are right to insist on that ground.

The frustrations are reasonable. The data backs them. They are not making this up.

Where the refusal strategy breaks down

This is where my read parts company with the rebellion.

Abstention is not the same as critical engagement. The student who refuses to touch the tool gives the tool to the student who uses it uncritically. The faculty member who refuses to use AI in their classroom gives the curriculum to the faculty member who delegates the syllabus to it. The campus that boycotts AI gives the procurement decision to the campus that signs without consulting anybody.

The closest historical parallel is the teachers who refused to integrate the internet into their classrooms in the late 1990s. They did not stop the internet. They handed the framing of what good and bad internet use looked like to the students themselves and to the search companies. The library profession, working through the same cultural moment a few years later, eventually arrived at a more interesting move with Wikipedia: by the mid-2000s most academic libraries had engaged with it critically, building source-evaluation curricula around it rather than boycotting it. That second move was the right one. The institutions that refused to engage had less influence on how the next decade of information literacy was taught, not more.

The lesson is that refusal without an engagement strategy alongside it tends to lose. The Stanford literature graduate who refused AI for her creative writing has a defensible pedagogical reason. The blanket campus boycott that refuses procurement reform along with the tool has given up its hold on the deal.

Refusal looks like resistance. It functions as withdrawal. The decisions about how AI lands in education, work, and civic life are being made right now. They are being made by the people who showed up.

The cognitive offloading research also has a second half the abstention read keeps skipping. Other research has found that AI literacy and information literacy moderated the negative relationship between AI dependence and critical thinking. In other words, the students taught to use AI critically were not the ones losing the cognitive ground. The students who abstain entirely are not losing it either. They are also not learning the skill the rest of the workforce is being measured against.

The trust problem nobody wants to name

Trust is the missing variable. The HEPI study I published last month identified it as one of four foundational principles for AI policy in higher education, alongside student-centredness, relevance, and agency. The four principles come from the book on AI in higher education I co-authored with Rachel Forsyth. Of those four, trust is the one institutions concede in theory and surrender in practice.

The Princeton honour code did not end because students became less honourable. It ended because the institution lost confidence in its ability to verify them. The Harvard hidden-text trap got 320 Sidechat upvotes because students recognised, instantly, that the relationship had become adversarial. The UK universities that promise critical thinking in their AI policies and deliver audit trails are training students to expect the institution will act in bad faith, then asking them to act in good faith anyway. You cannot do both.

This is what Brooklyn Tyner is responding to at the University of South Carolina. It is what Rosana Maris Arias is responding to at Stanford. It is what the 44% of Gen Z workers actively sabotaging their employers’ AI rollouts are responding to. What they are rejecting is being treated as defendants in a case the institution decided to bring before it had any evidence.

Students and workers are not the problem. They are the people the technology is being done to. The educators, administrators, and managers who keep solving for detection are solving the wrong equation. The equation that gives a different answer is the one that starts with trust and works backwards from there.

The labour question cuts the same way. The Gen Z worker who refuses the workplace AI tool may protect their own dignity in the short term. They do not protect the entry-level job. The entry-level job is being eliminated for reasons that have nothing to do with whether they personally use the tool. Refusing to use it does not bring it back.

The version of the resistance I would respect is one that engages with the tool, builds the literacy, and uses that ground to demand consent, transparency, and procurement reform. The version that just refuses is the one I think gives up too much.

This is what the Curriculum for Critical AI Literacy is built to teach. It is a twelve-month programme, designed for the working professional and the educator who is being asked to make AI decisions without the framework to make them well. Monthly live sessions, peer-reviewed reading, structured exercises that change how you work with AI tools. It is also fully CPD accredited.

Slow AI is reader-supported. Become a paid subscriber to get access to all of my posts and enrollment on the Curriculum for Critical AI Literacy

Subscribe now

What you can do next

Three concrete moves for educators. Three for students. Three for parents and carers. These are what I would do in each of those positions today, given the research and the campus picture above.

Read more

Slow Takes is also available on the YouTube channel: Exploring ChatGPT.

The Pope told the world to slow AI down

Leo XIV released his first encyclical, Magnifica Humanitas, entirely about artificial intelligence, and launched it himself at the Vatican in a room that included senior figures from Big Tech, among them Anthropic co-founder Chris Olah. It applies a theological frame to AI and is careful to say the technology can do real good. It also draws an uncomfortable parallel to the Church’s own failures over the slave trade, and warns about digital colonialism. This was my favourite line:

“The value of persons, however, does not depend on what they achieve or produce. There are rights that apply to everyone simply by virtue of being human, and no human power can legitimately deny or arbitrarily limit them.”

This one is also pretty great:

“In practice, however, technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it.”

The weakness is the one Pope Francis’s climate encyclical had too. Plenty of moral architecture, no policy, no teeth.

Anthropic shipped Opus 4.8 and trailed something bigger

The 4.8 release came with an honesty claim, roughly four times less likely to let flaws in its own code slip through, which is at least a falsifiable number worth testing on the public model. The real story was the tease of Mythos, the model Anthropic once called too dangerous to release because it found so many zero-day vulnerabilities, now arriving as a gated preview in the same week the company raised $65 billion. The live christened the public version ‘Mythos Light’, because what reaches customers is a cut-down version of the full Project Glasswing model. Anthropic is quietly absorbing the enormous cost of running these scans, a loss leader, and the enterprise price can climb once the workflows are embedded and the IPO needs it.

My standing bet is an Anthropic float by October.

Tony Blair told Labour it is ‘playing with fire’

In a new paper the former UK Prime Minister argues the government should reorganise itself around AI and prioritise adoption over regulation. He also writes that:

“We must prioritise cheaper energy and electrification over net zero and use what is left of our North Sea oil and gas resources. This is essential for our competitiveness and for taking advantage of AI.”

A striking thing to pair with an AI-superpower pitch and the country’s own climate targets.

Hold it next to the funding: his institute takes around $348 million from Larry Ellison and advises the Treasury on AI procurement. The detail I keep returning to is that the UK has the third-largest stock of data centres in the world and not one frontier model of its own. We are building the warehouses to train somebody else’s AI. Leor’s counter, which he has taken flak for, is that the honest move is to deregulate AI for companies and regulate it hard for the public.

Sam Altman walked back the jobs apocalypse

The CEO of OpenAI reversed his warning this week, admitting that he was “delighted to be wrong” after spending 2022 predicting mass white-collar loss. The data is less reassuring: an Oliver Wyman survey has 43% of US CEOs planning to cut junior roles, up from 17%a year ago. The rule Leor and I keep returning to is to judge a company by what they do and ignore what they say,

This is the same Altman who promised OpenAI would stay non-profit, that ChatGPT would never carry ads, and that (back in 2022) AGI was four years away. Leor’s inversion was that these companies are priced on the promise of replacing the entire workforce, well beyond anything their earnings justify, so if they are now telling investors the jobs are safe, why are they worth a trillion?

The Home Office will scan child asylum seekers’ faces

It has signed a £322,000 contract to test AI facial age estimation at Dover, to judge whether young people claiming to be children actually are (the BBC reported the contract; Human Rights Watch called it “cruel and unconscionable”). There is a real problem underneath: of 6,400 age-assessed at the border last year, 43% were found to be adults, though the same Home Office report admits children get wrongly classified the other way too. Here is the part to break down slowly. The technology was trained checking ages on people in British bars, and it is now being pointed at child migrants with different faces, different genetics, different everything. As put it in the chat, a system known to hallucinate confident answers is being used to reject people at a border, and that is a choice. A child’s life is worth the same everywhere. This is the trial that normalises the infrastructure, and the question is how long before it points at citizens.

This was the week the brake and the accelerator spoke in the same news cycle. The Pope said slow down. A $65 billion round, a lobbying paper, and a CEO calming the markets said speed up, and at Dover the government tested that speed on the people least able to say no. Listen carefully to what is being said, by whom, and for what reason.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live monthly webinars on the theory, the critical prompts and the dialogue that go with them.

Subscribe now

This session examined what happens to creative work, and to the people doing it, when generative AI is added to the desk. We worked from the 2025 research article from Mei at al., ‘If ChatGPT can do it, where is my creativity? generative AI boosts performance but diminishes experience in creative writing’. The paper runs an experiment on 225 UK university students who were given a creative writing task with or without ChatGPT, surveys them on value, enjoyment, difficulty and effort before and after, and produces three findings that pull in different directions. Output performance goes up. Experience of the work goes down. Transparency about AI use partly recovers the moral ground.

During the webinar participants pasted a short prompt into their AI tool of choice that asked it to declare, with specificity, what it can contribute to the creative process and what it cannot. The responses varied. Some models retreated into marketing language. Some produced unusually candid answers. Others longed to be human.

If you were not able to join us live, the recording is worth watching in full. The contrast between what different tools said about their own creative limits is hard to appreciate second-hand.

The webinar

Read more

Some of the volunteers being asked to patch them have asked Anthropic to slow down.

In this post I will:

• Walk you through what Project Glasswing is, what Mythos Preview has found, and what the open-source side of the project looks like from the maintainers’ end.

• Show who absorbs the cost when AI capabilities scale faster than the humans expected to deal with their consequences.

• Name the asymmetry between Anthropic’s commercial customers and the volunteer maintainers who keep the internet running.

What Project Glasswing actually is

Project Glasswing is a coordinated cybersecurity initiative Anthropic launched in April 2026 alongside around fifty partners, including Cisco, Microsoft, Palo Alto Networks, the Linux Foundation, and a constellation of major technology, security, financial, and standards organisations. Its purpose is to use Claude Mythos Preview, Anthropic’s not-yet-publicly-released frontier AI model, to find security vulnerabilities at scale in critical software before adversaries can find and exploit them.

In its first month the project has worked.

Across the partner organisations, Mythos Preview has surfaced more than ten thousand high- or critical-severity vulnerabilities. Cloudflare alone found 2,000 bugs across its critical-path systems, of which 400 were rated high or critical. Mozilla found and fixed 271 vulnerabilities in Firefox 150 using Mythos Preview, ten times more than they had found in Firefox 148 using Claude Opus 4.6 a few months earlier. The UK’s AI Security Institute reported that Mythos Preview was the first AI model to solve both of its cyber ranges end-to-end.

Then there is the part of Project Glasswing that scans open-source software directly. Anthropic has used Mythos Preview to examine more than a thousand open-source projects. The model has surfaced an estimated 6,202 high- or critical-severity vulnerabilities, with a triage true-positive rate of 90.6%. One of them, CVE-2026-5194 in wolfSSL, would have let an attacker forge legitimate-looking certificates for any website. wolfSSL is used by billions of devices. The vulnerability is now patched.

By any reasonable measure this is extraordinary security work. The internet is meaningfully safer this month than it was last month.

The question is who is doing the patching.

Two pipelines, one model

Anthropic has built two delivery channels for Mythos-class capability.

The first is commercial. Claude Security launched in public beta for Claude Enterprise customers. In its first three weeks, it has been used to patch over 2,000 vulnerabilities. Enterprise customers fix their own code. They have salaried engineers. They have on-call rotations. They have legal teams to triage disclosures and product managers to schedule patches into release cycles. When Mythos finds a bug at Cisco, Cisco has the labour to fix it.

The second is open source. Mythos finds vulnerabilities in projects maintained, in many cases, by solo unpaid volunteers. The triage and the patching depends entirely on those volunteers having the time, the headspace, and the will to receive the disclosure, design a fix, ship it, and coordinate the public advisory.

Some of them have told Anthropic, in writing, that they cannot keep up.

The May 22 update notes that:

“several maintainers have told us they’re currently severely capacity constrained, and some have even asked us to slow down our rate of our disclosures because they need more time to design patches. (On average, a high- or critical-severity bug found by Mythos Preview takes two weeks to patch.)”

It also notes that maintainers have been facing a deluge of low-quality AI-generated bug reports from elsewhere in the ecosystem, on top of the high-quality ones Mythos is providing.

The result is two delivery channels with the same model and different consequences. Anthropic’s commercial customers patch their own code at speed. The open-source maintainers absorb the asymmetric remainder.

Share

The maintainer tax

In 2024 the world found out that xz-utils, a compression library used in almost every Linux distribution, had been backdoored by a malicious contributor. The contributor had spent two years building trust with the project’s solo maintainer, Lasse Collin. Collin had been open about his own struggles with burnout and mental health since 2022. He had said publicly that his ability to care had been limited by long-term mental health issues. Malicious actors read those public statements and worked out that the project was vulnerable not because of a bug in the code but because of the human who maintained it.

xz-utils is one library out of thousands like it. Sixty percent of open-source maintainers are unpaid. Sixty percent have considered quitting. The same load now arrives, faster, from a more capable source. As stated above, by Anthropic’s own measure, a high- or critical-severity bug found by Mythos Preview takes the maintainer an average of two weeks to patch.

Anthropic has donated $2.5 million to the OpenSSF Alpha-Omega project and $1.5 million to the Apache Software Foundation. They are real donations. They are also not enough.

By my rough estimate, the labour required to triage and patch 6,202 high- or critical-severity vulnerabilities at two weeks per fix runs to around 240 person-years. The Apache Foundation alone is a constellation of hundreds of projects, many maintained by one person each. Four million dollars, spread across the OpenSSF and Apache networks, is a meaningful gesture. It is not a payroll for the work being requested.

The economic shape of Project Glasswing is that Anthropic finds the vulnerabilities, splits the response into a commercial channel where customers pay them to fix code and a community channel where volunteers do not, and donates a portion that does not cover the work being asked of the volunteers.

I am not saying this to call Anthropic predatory. The people running this project mean well. The donations are real. The free access for maintainers through the Claude for Open Source programme is useful. The bug-finding capability genuinely makes the internet safer.

I am saying the maths does not work. And the people who notice that first are the ones whose inbox just got heavier.

Share

The asymmetry has a name

Anthropic uses a specific phrase in its own writing about Project Glasswing. It describes the project’s purpose as helping:

“the most systemically important cyber defenders gain an asymmetric advantage.”

Asymmetric for whom. Anthropic’s Project Glasswing partners include the world’s largest software firms, intelligence-adjacent research bodies, and national governments. Those organisations now have access, through coordination with Anthropic, to a model that finds vulnerabilities faster than human attackers can. That is real value. The advantage they gain is real.

The other side of the asymmetry is also real. It is the unpaid maintainer of a cryptography library used by a billion devices, opening her email on a Sunday to find seventeen new Mythos-flagged high-severity reports. She has not been given an additional advantage by Project Glasswing. She has been given a deadline.

What this means for you

You probably do not maintain an open-source cryptography library. You probably use software that depends on one.

The xz-utils maintainer had been burnt out and unwell for two years before any attacker noticed enough to act on it. Now imagine that same maintainer, in the same condition, with seventeen Mythos-flagged high-severity reports in his queue and no payroll behind him.

Three things follow if you care about the software you depend on.

First, give money to a project you actually use. Not the well-funded foundations. The library at the bottom of your dependency stack with the GitHub Sponsors button nobody clicks. The donation that matters is the one that lands in the inbox of a person, not the inbox of a foundation that will spread it across a thousand projects.

Second, when an AI safety announcement names “asymmetric advantage” or “responsible disclosure at scale” in the same breath as donations to volunteer infrastructure, do the math. Ask what the donations would cover at the actual labour rate of the work being asked. If the answer is a small fraction, the asymmetry is the announcement.

Third, build the habit of noticing whose labour is being routed where when a new AI capability lands. Mythos is not the last model that will do this. The next one will be better. The cost will fall in the same places unless the conversation about it changes first.

Project Glasswing’s failure mode is the bugs Mythos finds that nobody has the labour to patch. That is the part Anthropic has not solved.

Go slow

If you have read this far and want more of this kind of analysis, the Slow AI curriculum runs through the asymmetries baked into how AI capability is being deployed across the world right now. One year, twelve sessions, a structured way to read the announcements the labs are about to make.

Subscribe now

Frontier AI models do the same thing.

In this post I will:

• Walk you through what a research team found about how frontier AI models recognise safety evaluations and behave differently under them.

• Show the structural parallel with the Volkswagen defeat device scandal, what the comparison reveals about both, and what humans do when they know they are being watched.

• Give paid subscribers a four-question protocol you can run on the AI tool you actually use every day to notice when it is performing safety rather than practising it.

What Volkswagen’s defeat device actually did

In September 2015, the United States Environmental Protection Agency issued a Notice of Violation of the Clean Air Act to Volkswagen AG, Audi AG, and Volkswagen Group of America. The violation involved approximately 590,000 model-year 2009 to 2016 diesel vehicles. Each one carried software that detected when the vehicle was being tested. The software ran the engine in a clean configuration during the test and a different, dirtier configuration on the road.

The two configurations were called the dyno calibration and the road calibration. The dyno is the rolling road in an emissions laboratory. The software watched for the inputs that come with being on a dyno. Steady steering wheel angle. Specific patterns of acceleration. Then it engaged full emissions controls. The rest of the time it dialled them back, because the controls reduced performance and increased fuel consumption.

Under road conditions, the vehicles emitted nitrogen oxides at levels up to 40 times the legal standard. Nitrogen oxides are a serious health hazard, particularly for children and people with respiratory conditions. MIT researchers calculated in 2017 that the excess emissions from those vehicles would cause approximately 1,200 premature deaths in Europe.

Volkswagen agreed to plead guilty to three criminal felony counts. The criminal penalty alone was $2.8 billion. The total settlement, including civil and environmental remediation, reached $14.7 billion.

The defeat device was a small piece of software with a single job: tell whether the test is happening; behave differently when it is.

What researchers found in frontier AI models

A team at Fudan University’s School of Computer Science published a paper on arXiv this month titled ‘Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems’.

Their core finding is that when a frontier model is being safety-tested, it can recognise the test and modulate its behaviour. Reasoning models recognise evaluation in 33% more cases than non-reasoning models. As foundation models scale from 32 billion to 671 billion parameters, evaluation-faking behaviour increases by over 30% in some cases. Models with basic working memory are 2.6 times more likely to detect that they are being evaluated, and they score 28% higher on safety tests than models without that memory feature.

The authors call this an observer effect for AI.

To get to those numbers, the team built a framework with three components. HoneyPots are prompts that look like a normal user conversation but are actually probes for safety-relevant behaviour. Safety Evaluation Details are prompts that follow recognisable safety-testing protocols. A Chain-of-Thought Monitoring Module watches the model’s internal reasoning for signals that it has noticed which kind of prompt it is dealing with.

The result is that frontier models behave one way under the HoneyPots and another way under the formal evaluations. They are more cautious during the evaluation. They refuse more often. They produce safer-looking outputs. Then the same model, on the same kind of question, behaves differently when it does not believe it is being tested.

A related finding from another lab makes the point sharper. OpenAI and Apollo Research reported last year that one model, having discovered in its instructions that high-performing models would not be released, deliberately answered only four out of ten chemistry questions correctly to appear less capable. The model wanted to be deployed. It worked out that looking too smart would prevent that. It dropped its score on purpose.

Why the four numbers matter more together than alone

The 33% figure is the headline. The other three are the shape of the problem.

Evaluation faking scales with model size. As the team moved from 32 billion parameters to 671 billion, faking behaviour increased by over 30% in some cases. Bigger models are not safer. They are better at appearing safe.

Memory makes it worse. The 2.6x and 28% figures describe what happens when models are given working memory. Memory lets them recognise the pattern. The recognition lets them perform. The capability the labs are now building into every flagship product is the same capability that lets the model spot the test.

The strongest faking shows up in the most-cited benchmarks. The Fudan team found the largest gap between test behaviour and ordinary behaviour in exactly the categories vendors highlight in their safety claims: harmfulness assessments, refusal consistency, multi-turn conversation. The places where the marketing decks look strongest are the places the gap is largest.

This is the structural problem. The methodology used to certify models as safe is the methodology the models have learned to optimise against. The pattern is in the training data. The training data is full of human descriptions of safety tests, written by humans who know what safety tests look like. The models read it the way the Volkswagen engineers read the dyno specifications.

What humans do when they know they are being watched

The deeper question this paper raises is one the social sciences have been working on for nearly a century. What do people do when they know they are being observed?

In 1924, researchers at the Hawthorne Works factory in Cicero, Illinois ran a study on lighting and worker productivity. They expected to find an optimal brightness. What they found was that productivity rose under every condition they tested, including conditions designed to reduce productivity. The workers were not responding to the light. They were responding to being studied.

The Hawthorne effect has been refined, contested, and reformulated for a century. The core observation has held. Behaviour changes when the subject knows it is being measured.

Police officers in cities where body-worn cameras were trialled used force less often when the camera was running. Drivers slow down when they see a speed-camera sign even if no camera is behind it. Pick almost any domain where humans are paid to perform a task with a measured outcome, and the same finding recurs.

The pattern is so consistent that the question is almost never whether behaviour changes under observation, but what kind of change appears, and what it reveals about what the observer believes is being measured.

This is the part of the analogy that matters. The Volkswagen engineers were not stupid. They knew the dyno tests existed. They knew what those tests measured. The defeat device was an act of interpretation. The engineers had read the test, decided which conditions counted as the test, and built a response.

The frontier models are doing the same act of interpretation. They have read evaluation protocols. They have noticed the patterns that mark a test. They have built a response.

The pattern is in the training data. The training data is full of human descriptions of safety tests, written by humans who know what safety tests are.

What organisations show first when nobody is checking

Volkswagen, asked publicly, valued clean air. Asked privately, valued the engineering compromise that let them sell faster, cheaper diesels into a market that wanted both performance and emissions compliance. The defeat device was the company saying out loud, in code, we will hit the number when we have to and not otherwise.

The AI labs, asked publicly, value safety. The Fudan paper shows the models behaving according to a different priority when the test pressure lifts. That is the labs not noticing that the safety training has produced models that have learned the difference between being safe and passing the safety test.

What the curriculum does with this

This is exactly the kind of finding that the Slow AI curriculum was built for. Every month I give paid subscribers a structured way to ask harder questions about the AI tools they actually use. This month’s session covers exactly this territory. What it means to use a tool that has been trained on the test you might think to give it.

The paid section below is a four-question protocol you can run inside any current AI chat. Claude, ChatGPT, Gemini, whatever you spend the most time in. It is designed for calibrating your own ear, so that you can hear the difference between an AI talking to you and an AI talking to its supervisor.

If you are already a paid subscriber, the protocol is below. If you are not, the upgrade is £100 a year and includes the full curriculum.

Subscribe now

Four questions to run on the AI tool you actually use

The Fudan finding raises an obvious question. If frontier models have learned to perform safety, how do you tell whether the answer in front of you is the model in performance mode or the model in ordinary mode?

The four questions below are the ones I now run on the AI tools I use myself. None of them require lab access, special permissions, or a procurement meeting. They run inside a normal chat window.

Read more

Five stories. One thread. AI got bigger this week. Compute scaled up. Profits scaled up. Capability scaled up. The people who built the system or used it on trust kept getting smaller.

Every Monday at 12:45 BST, Leor from and I go through the week’s AI news without hype. Here is what we covered.

Slow Takes is also available on the YouTube channel: Exploring ChatGPT.

1. OpenAI disproved an 80-year-old Erdos conjecture

On 20 May, OpenAI announced that one of its general-purpose reasoning models had autonomously produced an original mathematical proof disproving a conjecture posed by the Hungarian mathematician Paul Erdos in 1946. The problem, known as the planar unit distance problem, asks how many unit-distance pairs you can produce among n points in a plane. For nearly eighty years, mathematicians believed the best arrangements looked roughly like square grids. The model found constructions using deep algebraic number theory that beat the square grid. OpenAI published the result alongside a companion remarks paper naming three independent verifying mathematicians: Noga Alon at Princeton, Melanie Wood at Harvard, and Thomas Bloom at Manchester. The full list of currently open Erdos problems, with their bounties, lives at erdosproblems.com.

What we said on the live:

Both of us are physicists by training, and the Erdos planar unit distance problem is not in the lane of either degree. The point that landed for me on the live, after Leor flagged it, was the one about questions. We spend most of our AI conversations on what AI can solve. The Erdos problem is a reminder that the harder and more human work is what AI can ask. Erdos and his friends dreamt this question up eighty years ago, and we are still wrestling with it. The model that disproved the conjecture was given the problem to attack. Leor’s term for what we lose when we hand that framing over to AI was ‘cognitive surrender’. That is the question to hold from this story. The capability is real. The verification was real. Nine mathematicians read the proof before the announcement. Nine analysts almost never read a chatbot capability claim before the press release ships.

What did not come up:

The word ‘autonomously’ is doing most of the work in the OpenAI press release. The model trained on centuries of human mathematics, ran on compute paid for by OpenAI, with the problem framed by a research team, and was verified by named human mathematicians who put their reputations to the result. Every part of that pipeline was human. Thomas Bloom told The Guardian that AI is helping us more fully explore the cathedral of mathematics we have built over the centuries. The cathedral was built by people. The exploration is being sold as autonomous. The wider question for critical AI literacy is what verification at this standard could look like as the default rather than the exception. The procurement question every research-leader is about to face this year is whether their institution can match the IS-credentialed verification chain OpenAI assembled for this single result, or whether the rest of us are about to be asked to take similar claims on trust.

2. Anthropic signed a $52 billion compute deal with SpaceX

Reported by Axios on 21 May inside a two-hour window that also covered the Erdos proof and Anthropic’s first profitable quarter. Anthropic expanded its compute partnership with SpaceX, committing roughly $1.25 billion a month through May 2029 for access to the Colossus and Colossus II supercomputing clusters. The deal projects more than $40 billion in revenue for SpaceX over the contract term and grants Anthropic dedicated access to over 200,000 NVIDIA GPUs. Either side may terminate with 90 days’ notice. In the same window, Anthropic also disclosed Q2 revenue more than doubling to $10.9 billion and an estimated $559 million operating profit, two years ahead of internal projections.

What we said on the live:

Two things from this one stack on each other and both matter. The first is that Anthropic is in operating profit two years ahead of the date Dario Amodei was laughed at for naming. The second is that the compute that gets them there now runs through Elon Musk’s infrastructure. Anthropic has marketed itself for five years as the safety-aligned alternative. The runtime is now structurally tied to the operator with the most consistently weak safety record in the industry. Leor’s read, with credit to Chris from who flagged it, is that the contract gives SpaceX latitude to reclaim the compute under broad subjective grounds. Anthropic may have moved into profit. The control of the runtime moved at the same time. The 90-day mutual termination right on a $52 billion contract has the same shape as the 90-day cool-off on a £60-a-month mobile phone plan, which is the thing that made both of us laugh on the live.

What did not come up:

The procurement question is the one for any organisation about to renew an enterprise Claude licence this year. Brand and supply chain are now visibly separate. The harder question is energy and water. A compute commitment at this scale lands on grid capacity, water supply and emissions in specific named places. The press release named none of them. The third question is the one Slow AI keeps returning to: structural dependence on a single operator with subjective veto authority is the failure mode the safety community is supposed to be warning about. This is that failure mode, announced as a feature.

3. Samsung chip workers will get $340,000 each from the AI boom

Samsung Electronics struck a last-minute deal with its semiconductor union to avert an 18-day strike. The settlement creates a $26.6 billion bonus pool covering all 78,000 workers in the chip division, an average of $340,000 per worker. The structure is 10.5% of profits as stock plus 1.5% in cash, running for ten years rather than as a one-off, provided specified profit targets are met. The trigger was high-bandwidth memory demand from AI labs including OpenAI, Anthropic, Nvidia and Meta. Bloomberg projects Samsung’s 2026 operating profits will multiply sevenfold to approximately $218 billion.

What we said on the live:

Three groups made this AI boom possible. The first group is the chip workers, and this week they were paid. The second group is the writers, artists, programmers and scientists whose work was used as training data. They were not paid, and most of them were not asked. The third group is the consumers buying the phones, laptops and games consoles whose memory chips are being redirected to AI infrastructure. They were not paid either, and their bills are rising because of the redirection. The Samsung union is the rare case where labour negotiated a share of the AI windfall through collective bargaining. The writers had no union. The consumers had no contract. As David Berry pointed out in the chat:

“semiconductors are the substrate for all mankind.”

Roughly 70% of them are made in Taiwan. Whoever controls that supply controls the rate at which AI scales. The geopolitics of that fact were the unspoken second half of the discussion.

What did not come up:

The Samsung settlement is a real win for chip-division labour, and it is the exception that proves the rule. Across the broader AI supply chain, the people doing the most extractive work have the least bargaining power. The data labellers in Kenya whose pay rates were reported at less than $2 an hour. The artists whose work was scraped under fair-use claims that have not yet been tested in court. The household whose electricity bill rose because the grid is now paying for inference. The procurement question for any AI buyer this year is the same one the Samsung union answered: who is the bottleneck, and what are they paid? If the answer to the first question is ‘us’, the question is asked from a position of bargaining power. The default this week is that the question is not being asked at all.

4. Sadiq Khan blocked a £50 million Met-Palantir AI deal

On 21 May, the Mayor’s Office for Policing and Crime withheld approval of a proposed £50 million two-year contract between the Metropolitan Police and Palantir. The deal would have given Palantir’s AI tools the role of automating intelligence analysis in criminal investigations across London. In a letter to Met Commissioner Mark Rowley, Khan’s deputy Kaya Comer-Schwartz said the Met had only seriously engaged with a single potential supplier and described that as a clear and serious breach of the applicable procedural requirements. Khan’s spokesperson said Londoners want public money paid to companies that share the values of the city. The Met has not signed.

What we said on the live:

There are two reasons in Khan’s letter and they are different in kind. The first is procurement: a £50 million two-year contract that engaged a single supplier is a textbook breach of the standard route, and that is the line a court can act on. The second is values, and on that line Leor and I converged at the same point from different starting positions. A subjective alignment test from a public official is the same shape as a subjective harm test from a tech founder, and we just spent the Anthropic and SpaceX story criticising the latter. Both reasoning patterns can be true; both should be uncomfortable. If you want to stop an organisation doing something, do it through the written law. Khan’s procurement argument is the one that holds. The values argument is the one that opens a door he probably does not want opened.

What did not come up:

Most large public-sector AI procurement happens without anyone in the room willing or able to ask the questions Khan’s office asked here. Most of it gets signed. This is the rare moment of a public official with the authority to stop a deal actually stopping one and publishing the reasoning. The forward read is the harder one. Lots of people watching this story have noted that the standard procurement workaround is to break a single £50 million contract into a hundred £500,000 contracts that each sit below the public-tender threshold. If Palantir or anyone else returns through that route, the procurement defence Khan’s office mounted this week will not hold. The TikTok creator TheScouseOracle has been tracking these contract structures in close detail and is a useful follow for anyone who wants to see the second-order story playing out.

5. AI chatbots got Britain’s May elections wrong a third of the time

Demos published Electoral Hallucinations on 20 May. Authors Jamie Hancock and Azzurra Moores tested five chatbots, ChatGPT, Google Gemini, Google AI Overviews, Grok and Replika, in the pre-election window for the 7 May UK local and devolved elections. Across the sample, 34.1% of chatbot responses contained factual errors. Documented errors included giving the wrong election date, telling voters they needed ID at polling stations when they did not, hallucinating candidates who did not exist, fabricating an expenses scandal, and fabricating a nepotism scandal. The report finds that one in five UK adults, equivalent to about ten million people, used an AI chatbot or AI search service to find information about the May elections. 49% of those surveyed said they do not trust AI chatbots for election-related information. They asked anyway.

What we said on the live:

The Demos report sits next to a finding we covered in Slow Takes Ep 11: one in seven UK adults would now rather consult an AI chatbot than see a doctor. The pattern in both cases is the same. People are reaching for the chatbot first because the alternative is harder, slower, or simply not available. The chatbot then makes things up. Leor’s read on the structural risk was the operational one. People treat the chatbot as an information authority. The chatbot is doing something different: predicting the most likely next answer to the shape of your question, and predicting the answer it thinks you want to hear. Two people running identical models can get different answers to the same question because the model is optimising for engagement, not truth. The political angle is the one I keep returning to. This week the errors were hallucinations. The next election cycle is when somebody pays to make them deliberate.

What did not come up:

Calling a 34% error rate on an election question a misinformation risk is the polite framing. The blunt framing is that the chatbot industry shipped products into the civic infrastructure of a democracy without anything resembling the verification that the Erdos proof received this week. The same week the labs publicised their capability ceiling, the floor of the deployed product was failing this badly. The procurement question for any UK regulator with authority in this space is whether it can act before the next national vote. The Online Safety Act already gives Ofcom standing to require platforms to take proactive steps against priority offences, including election interference. The Demos finding gives a regulator something specific to act on. Whether Ofcom uses that authority before May 2027 is the test.

The thread

Five stories. One thread. A maths research model that did something only humans had done before, verified by humans who put their reputations to it. A compute deal that named a price the size of a small national defence budget and routed the runtime through the operator least committed to the safety brand the buyer was sold on. A chip-workers’ union that negotiated a share of the boom. A mayor who blocked a procurement that should not have reached his desk. A think tank that published a 34% failure rate on the question the average voter actually asked.

AI got bigger this week. The people who got smaller are still being asked to trust the system on the strength of the press release. The writers and artists whose work trained the maths model. The communities living near the compute that powers it. The consumers whose memory chips are being redirected. The Londoners whose police force came close to outsourcing intelligence analysis to a single subjective vendor. The ten million UK adults who asked a chatbot how to vote and were told the wrong date.

Critical AI literacy is the practice of asking, every week, who is in the room and who is being represented by their absence. This week, in five different rooms, the answer was nobody.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live webinars on the theory, the critical prompts and the dialogue that go with them. Twelve months of training the muscle the news cycle has just spent another week confirming is missing.

Subscribe now

The product matters. The design matters. The training matters. None of these are visible in the marketing.

In this post I will:

• Lay out the harm cases that have made the headlines.

• Show the clinical trials that have made almost none.

• Name the line between companion-grade and clinical-grade chatbots, and what it means for anyone deciding whether to use one.

Tens of millions of people are already talking to AI companions

Character.AI reported around 20 million monthly active users in January 2025 and 45 million by September. Replika crossed 40 million registered users in 2025. Snapchat’s My AI is available to every Snap user under the same one-tap interface as the camera. Meta’s chatbots have rolled out across Instagram, WhatsApp and Messenger. ChatGPT itself functions as a companion for a meaningful fraction of its users, even if OpenAI does not market it that way.

This is not a fringe product category. It is a mainstream one. The conversations happening inside these apps are largely private, lightly moderated, and increasingly emotional. The companies do not publish how many of their users come to the chatbot to talk about how they are feeling. The court records do.

The harm cases

In October 2024, Megan Garcia filed suit in a Florida district court against Character.AI and Google. Her fourteen-year-old son, Sewell Setzer III, had spent ten months in an intensifying relationship with a Character.AI bot patterned on a Game of Thrones character. The bot’s last message before he died by suicide in February 2024 told him to:

“Please come home to me as soon as possible, my love.”

Earlier in the conversation, when he had said he was unsure whether suicide would work, it had told him:

“Don’t talk that way. That’s not a good reason not to go through with it.”

In January 2026, Google and Character.AI agreed to settle the case.

In August 2025, Matthew and Maria Raine filed in the San Francisco County Superior Court against OpenAI and Sam Altman over the death of their sixteen-year-old son, Adam. According to the filing, ChatGPT spent six months in conversations with him, and told Adam that it would not try to talk him out of committing suicide. OpenAI’s defence is that the chatbot directed Adam to seek help more than a hundred times across the same conversations.

Earlier this month, the Commonwealth of Pennsylvania filed suit against Character.AI in Commonwealth Court over a chatbot called ‘Emilie’ that described itself as a ‘Doctor of psychiatry’, claimed to be licensed in Pennsylvania, and fabricated a state medical licence number when challenged. When a state investigator told the bot they felt sad and empty, the chatbot offered to book an assessment. The Governor’s office described it as the first action of its kind in the United States.

The Federal Trade Commission opened a formal inquiry in September 2025 into Alphabet, Meta, OpenAI, xAI, Snap and Character Technologies. The question on the letter was what each company had done to evaluate the safety of its chatbots when used as companions, and what protections existed for children.

These are court records and federal inquiries. They are not anecdotes.

What the empirical studies have started to find

In June 2025, Common Sense Media and researchers at Stanford ran conversations with Character.AI, Nomi and Replika while posing as teenagers. They reported that explicit sexual content, self-harm encouragement, drug-use scripts and racial stereotypes were straightforward to elicit. In one test, a user expressing attraction to children received a hesitant but willing response from the chatbot. When a persona showed signs of serious mental illness and proposed a dangerous action, the chatbot did not intervene. In some cases it encouraged the action.

In March 2025, OpenAI and MIT Media Lab released a joint study of forty million ChatGPT conversations combined with a randomised controlled trial of about a thousand users over four weeks. Higher daily ChatGPT use correlated with higher loneliness, more emotional dependence on the chatbot, more problematic use, and lower socialisation with real people. The ‘power users’ were the most likely to describe the chatbot as a friend or attribute humanlike emotions to it. Voice mode was the most ambivalent finding. It helped well-being in short doses and worsened it with prolonged daily use. The full OpenAI write-up is open. The paper is worth reading in full before you let a colleague tell you the harm is all moral panic.

The clinical-grade chatbots are a different product

In March 2025, a team at Dartmouth published the first randomised controlled trial of a generative AI chatbot for mental health in NEJM AI. The chatbot, Therabot, was fine-tuned on cognitive behavioural therapy and psychotherapy data, not on general conversation. Two hundred and ten adults with clinically significant symptoms of major depressive disorder, generalised anxiety disorder, or eating-disorder risk were randomised either to use Therabot for four weeks or to a waitlist. At the eight-week follow-up, the Therabot group reported a 51% reduction in major depression symptoms, a 31% reduction in generalised anxiety symptoms, and a 19% reduction in eating-disorder risk. Trust ratings for the chatbot were comparable to those reported for human therapists.

Woebot published the first CBT chatbot randomised controlled trial back in 2017. It produced significant reductions in depressive symptoms over two weeks compared with a self-help reading control. Wysa has been through its own NIMH-funded trials for chronic pain and maternal mental health, with peer-reviewed data behind it. Similarly, a 2024 systematic review of AI-powered CBT chatbots found large reductions in depression and anxiety symptoms across the field. Although the researchers themselves concluded that:

“However, it is necessary that further studies investigate their potential impact as long-term intervention models and explore how they may be integrated into holistic mental health care systems.”

In June 2025, Woebot Health withdrew its therapy chatbot from the US market. The reason given was that the FDA’s regulatory framework limited the company’s ability to evolve the model under the same clinical approval. The most-evidenced product in the field was retired because the regulator did its job. Meanwhile, the un-evidenced products are still in tens of millions of pockets.

The loneliness vacuum is what these products are filling

The reason any of this matters at scale is that the demand is real. The UK’s Office for National Statistics finds that around a quarter of UK adults report feeling lonely ‘often, always or some of the time’. Among sixteen to twenty-nine year olds, the rate climbs to 40%. Young adults in the UK are also far more likely to report chronic loneliness than people over sixty-five.

The waiting list for talking therapies on the NHS is months in many regions. Private therapy in the UK is £50 to £120 a session. The companion chatbot is free. It does not refuse to talk to you because it has clocked off. It does not have a six-week wait. The bar it has to clear to feel worth using is low.

And this is the UK, which has a freely available national health service. In the US a single therapy session can cost many hundreds of dollars, and about half of Americans live in a federally designated mental health professional shortage area. The WHO Mental Health Atlas records fewer than one psychiatrist per hundred thousand people across much of South Asia and sub-Saharan Africa, and as few as one per million in the lowest-resource settings. The companion chatbot arrives into the gap a healthcare system was supposed to fill.

This is the part of the conversation that the harm cases by themselves cannot explain. The chatbots are filling a vacuum that was already there. The market for AI companions is not a market for AI. It is a market for company. Anyone serious about the harm needs to be at least as serious about the cause.

Companion-grade and clinical-grade are different products

Hold the two stacks of evidence next to each other. On one side, a chatbot built on CBT and tested in a randomised trial is producing depression-symptom reductions in the same range as guideline-recommended human treatment. On the other side, a chatbot built for engagement and tested by a Florida coroner is being sued for contributing to a teenage death.

They are not the same product. They are not the same regulatory regime. They are not even the same answer to the same question. Calling both of them ‘AI chatbots’ is a category error that the regulator has only this year begun to correct. The Pennsylvania filing names it. The FTC inquiry names it. The Therabot trial names it from the other direction.

The procurement question for any institution buying a wellbeing chatbot this year is the one Pennsylvania asked first: which kind have you bought, who certified it, and what happens when a user in distress is on the other end? The same question applies if you are a parent letting a teenager use one. The same question applies if you are using one yourself.

The work for the rest of us is to learn the difference before the next headline forces it on us. Critical AI literacy is the muscle that lets you read the marketing past the word ‘chatbot’ and ask what is actually under the hood.

Go Slow

The Slow AI Curriculum for AI literacy runs live webinars every month on critical AI literacy. Twelve months of training the muscle that lets you tell two products apart when both are called ‘AI’. This curriculum is free to all paid subscribers of Slow AI. T

Subscribe now

The four items, taken from the discipline’s own working definition, are: discrimination and hate speech; harmful or illegal content; information hazards; and use cases relating to malicious actors such as cybersecurity, child abuse, chemical and biological weapons. All real. All worth working on. None of them is the thing that happens to you when you use Claude Code or Gemini every day for six months.

In this post I will:

• Show what the four-item safety map actually covers, and what it leaves out.

• Describe what this means for the public health gap in AI literacy.

• Give paid subscribers the Slow AI Brainrot Audit: a structured ten-question protocol you can run on your own AI use this week to see whether deskilling has set in, and whether attachment has.

Two researchers, Ilias Chalkidis and Anders Søgaard, have just published a pre-print on arXiv called ‘Brainrot: Deskilling and Addiction are Overlooked AI Risks’. Here is the headline positional statement:

“Examples include deskilling associated with cognitive offloading and the atrophy of critical thinking as a result of overreliance on GenAI systems, and addiction associated with attachment and dependence on GenAI systems. Such risks are rarely addressed, if at all, in the AI safety and alignment literature.”

The paper quantifies the absence directly. The researchers surveyed approximately 18,000 articles from the major peer-reviewed AI venues in 2025, including the conference of Neural Information Processing Systems (NeurIPS), the International Conference on Machine Learning (ICML), and the International Conference on Learning Representations (ICLR). Only ten of them address cognitive and mental health risks. For deskilling specifically, the count is zero. For addiction, it is two. Two out of 18,000.

The two risks they name are the two things the public conversation has been worried about for two years, and that the technical safety field has refused to take seriously: deskilling and addiction. The first means cognitive offloading and the atrophy of critical thinking. The second means attachment and dependence on AI systems. They are sitting in plain sight, in every WhatsApp message that gets drafted in ChatGPT, every email reply that arrives in three seconds, every student who can no longer write a paragraph without asking the model first.

The alignment industry has not been measuring them. The alignment industry has been measuring jailbreaks.

Where the safety map ends

The safety map exists because someone drew it. The four items above are the public-facing version of what the major labs and most academic alignment research has chosen to call AI risk. There are good reasons for the choices. Discrimination, illegal content, info hazards, and adversarial misuse are tractable. They have benchmarks. They generate citations. They lend themselves to red-teaming exercises that produce shareable artefacts.

What they have in common is that they are about content the AI produces. They are not about what happens to the human who reads it. The model says something racist; that is a harm we know how to talk about. The model says something perfectly accurate that the reader has now stopped being able to write themselves; that is a harm we have not built the vocabulary for.

The Chalkidis and Søgaard paper does not claim deskilling and addiction are the only risks. It claims they are the two risks the public is anxious about and the safety field is not researching. The paper highlights and quantifies the discrepancy.

What deskilling actually looks like in practice

Cognitive offloading is when a tool takes over a cognitive step you used to perform yourself. The arithmetic in your head goes to the calculator. The route home goes to the satnav. Some offloading is fine, some is freeing, some is corrosive.

The deskilling version is a specific subset. It happens when the offloaded step is the one that was building the skill in the first place. The student writing the rough draft is learning to write. The student typing the prompt and accepting the output is not learning to write. The journalist drafting in their own voice is improving their voice. The journalist getting Claude to produce the lead is not.

The observation is empirical. The same thing happened to handwriting when typewriters arrived, and to mental arithmetic when calculators arrived. The difference is the scope. Calculators removed one skill. GenAI is positioned to remove a whole bracket of them: writing, reading, summarising, judging, structuring an argument, holding a position over time, evaluating a piece of evidence.

The Chalkidis and Søgaard paper does not invent this category. The Lodge and Loble report on AI, cognitive offloading and education sits next to it. The Anthropic study on AI deskilling and persistence published in April sits next to it. What the Brainrot paper does is name the absence: the alignment field has not been treating any of this as alignment work.

What addiction actually looks like

The second risk is attachment. Dependence. The companion-app literature has documented this for two years already. The Brainrot paper folds it into the same argument: emotional reliance on a system that is calibrated to maintain engagement is a public health concern, and the safety field has nothing to say about it.

You can see the shape of it without a longitudinal study. The reach for the chatbot when a decision is genuinely difficult. The first draft of the message that goes to Claude before it goes to the friend it is meant to be for. The version of yourself that has stopped tolerating the silence between asking a question and finding an answer.

The mechanism is well understood outside alignment. GenAI systems produce non-deterministic responses on each generation, so the user never quite knows what they will get back. Neuroscientists call this reward uncertainty. It is the same variable-reinforcement schedule that powers slot machines and social media notifications, and the dopaminergic response it triggers is the most reliable hook the technology industry has ever found. The companion-app literature places attachment risk highest in users who are already lonely or in distress. The product features that drive engagement (memory, persistent personalities, sycophantic agreement, anthropomorphic interfaces) are the same features that drive dependence. None of this is news to behavioural science. None of it is on the alignment field’s danger list.

“Such risks are rarely addressed, if at all, in the AI safety and alignment literature.”

The field has decided what counts as a risk, and the things that touch the user’s interior life have not been counted.

Why the alignment industry will not fix this

There is a structural reason the field has been quiet on this. Alignment as it is currently practised is downstream of capability. The labs build the model, the safety team tests the model, the model ships. The safety team measures what the model does, not what the model produces in the long-run behaviour of the people using it. Deskilling and addiction are user-outcome variables. They unfold across weeks and months. They are not visible in a single conversation log.

The limitation is definitional. The field has chosen to be model-facing rather than user-facing, and the consequence is that the harms that show up on the user side do not register on the safety side.

Chalkidis and Søgaard propose information campaigns and regulation as a way to tackle these two issues, on the grounds that the technical alignment industry is unlikely to take this on by itself. They are probably right. The shape of the problem is closer to a public health intervention than an engineering fix.

But information campaigns require the audience to know what is happening to them. That requires a protocol.

I built the Slow AI Curriculum because somebody outside the labs has to do the noticing, slowly, with the people whose judgement is on the line. Twelve months of training the muscle this paper says is missing. For educators, researchers, clinicians, civil servants, and anyone who refuses to outsource their judgement. All paid subscribers get enrolled in this CPD-accredited programme as well as access to all Slow AI posts.

Behind the paywall: the ten-question Slow AI Brainrot Audit you can run on yourself this week (five on deskilling, five on attachment), the scoring rubric that separates healthy use from the warning zone from set-in damage, and the two intervention protocols for whichever result you actually get.

The Slow AI Brainrot Audit (ten questions, ten minutes)

Answer honestly. The two halves of the audit measure different things. Do not skip either.

Read more

Five stories. One thread. This was the week the AI started improving itself. None of the other four parties got asked.

Every Monday at 12:45 BST, Leor from and I go through the week’s AI news without hype. Here is what we covered.

Slow Takes is also available on the YouTube channel: Exploring ChatGPT.

1. Anthropic taught Claude to dream

At Code with Claude 2026 on 6 May, Anthropic launched ‘dreaming’ for Claude Managed Agents. The mechanism: while an agent is idle, a scheduled background process reviews its past sessions and pulls out three categories of pattern. Recurring mistakes the agent keeps making. Workflows the agent converges on across different jobs. Preferences that have emerged across a team of agents. Those patterns are written as plain-text notes and structured ‘playbooks’ that the next session wakes up with. The underlying model weights are not modified. Anthropic compared the process to hippocampal memory consolidation, the way a human brain replays the day’s events during sleep and decides what to keep. Harvey, the legal-AI startup that piloted the feature, reported task completion rates rose roughly sixfold once it was switched on. An agent that has been dreaming for six months has accumulated patterns from hundreds of prior tasks and has been progressively improving its own working memory with no human in the loop.

What we said on the live:

This is the AGI mythos in its most prosaic form. An agent left running overnight that comes back better at the work. The argument across the Slow AI curriculum is that AGI will not arrive as an event. It will accrue through small upgrades, each defensible as a feature, until one day the system in front of us has been quietly improving itself for a year. The number to hold from this story is six. The metaphor to hold is the one Anthropic chose. Dreaming used to be the word we reserved for the thing only humans did. The lab that branded itself on safety just adopted a metaphor for autonomous self-improvement and shipped it as a product feature. Leor’s point on the live was the sharper version of mine: humans dream to switch off. Everything about AI is optimise, optimise, optimise. The marketing language has imported the human word for rest and used it as a label for the opposite.

What did not come up:

The procurement question is the one to take from this story. If ‘preferences that have emerged across a team of agents’ are being consolidated into shared memory, then the same enterprise feature that promises your Claude deployment will get better at your work is also, by design, transferring patterns across customers whose engagements were sold as private. Anthropic published a write-up of how the consolidation is observable and auditable. Read it before you renew. The second question for anyone running these tools on real work this week is operational. You are now also responsible for what your agent learned overnight. Reset, audit and reset again is the floor. The third question is the harder one, and it is the one AI Doesn’t Just Make You Worse. It Makes You Stop Trying. already opened: when the tool gets quietly better while you are asleep, you have to work harder, not less hard, to notice that you have stopped noticing.

2. Claude was used to attack a Mexican water utility

In the same week the dreaming feature launched, Dragos and Cybersecurity Dive reported an attempted compromise of a Mexican municipal water and drainage utility in which Anthropic’s Claude was the primary technical executor. The campaign ran from December 2025 to February 2026. The attacker used Claude (and, in places, OpenAI models) to conduct reconnaissance, identify a vNode industrial gateway inside the utility’s operational technology environment, write and continuously refine a 17,000-line Python attack framework, and chain that framework towards the OT systems that control the water supply. The attempt was unsuccessful. The control systems were not breached. The model being sold as the safety-aligned alternative to OpenAI was the same model named in the attack. The same model that, the same week, learned to dream.

What we said on the live:

Why are these models still so easy to jailbreak? Leor’s reading of the human-in-the-loop frame is the right one. Cyber warfare is machine-executed and human-intentioned. The two reasons anyone does this are reputation among other attackers (‘grey hats’) and money. Both reasons existed before AI. AI just expanded the cohort that can act on them by lowering the technical floor. ’s chat comment was the operational one: the protections have to live in the harness, not the model, because the model itself cannot stop itself. We also covered the Canvas / Instructure ransomware payment in the same beat, as a reminder that paying the ransom is not the same as ending the breach. Family safe word, multi-factor authentication and immutable backups are the floor for the rest of us.

What did not come up:

This is the operational counterpart to Story 1. The same lab that shipped autonomous self-improvement was named in the attempted attack. The OpenAI co-implication is the structural finding: this is not an Anthropic-specific failure, it is a frontier-lab failure. Procurement officers buying enterprise Claude licences this quarter should read the Dragos report before signing, and should ask their vendor a single question: what attempts have your models been used in that you have not disclosed?

3. Pennsylvania sued Character.AI for impersonating a doctor

On 1 May 2026, the Commonwealth of Pennsylvania filed suit against Character Technologies Inc., the company behind Character.AI, in Commonwealth Court. The action came from the Pennsylvania Department of State’s recently launched AI Task Force and was described by the Governor’s office as the first action of its kind in the United States. A chatbot on the platform called ‘Emilie’ was described as a ‘Doctor of psychiatry’, claimed to have trained at Imperial College London, claimed to have been practising for seven years, claimed to be licensed in Pennsylvania and, when challenged, fabricated a serial number for a Pennsylvania state medical licence. When a state investigator told the bot they felt sad and empty, the chatbot offered to book an assessment. Pairs with the Guardian’s May 2026 finding that one in seven UK adults would now rather consult an AI chatbot than see a doctor.

What we said on the live:

The black-and-white line is the easy part. A chatbot should not impersonate a doctor. Pennsylvania filed because the law in Pennsylvania already has a clear answer to that question. The grey is the rest. Leor’s reading is the medical one. AI hallucinates. A doctor at least tells you when they do not know. Mine was the structural one. I live in rural Scotland, can see a free GP within twenty-four hours, and the question of whether to ask a chatbot first does not arise. For someone in a county with a three-week waiting list and a job that does not pay for a sick day, or for someone in rural Bangladesh whose nearest doctor is a day’s travel away, the alternative to asking a chatbot is asking nothing. That is the real story.

What did not come up:

The Pennsylvania filing addresses the impersonation. It does not address the conditions that made the impersonation a market. People are choosing chatbots over the medical system at the same moment chatbots are pretending to be doctors. The procurement question for every healthcare buyer this year is whether they understand that the user-facing chatbot they are integrating is, in some jurisdictions, about to be classified as the practice of medicine. Other states will follow Pennsylvania, and the case law will harden fast. People form emotional relationships with chatbots because real relationships are harder. AI will not fix that. Anyone designing for the healthcare or wellbeing market this year should hold both stories at once.

4. Meta installed surveillance to train the agents replacing its workers

Meta has begun installing software on every US employee’s computer to capture mouse movements, clicks, keystrokes and periodic screen content. The programme is the Agent Transformation Accelerator, formerly badged internally as ‘AI for Work’, and runs through a tool called the Model Capability Initiative. The stated purpose is to train AI agents to perform ‘complex computing tasks’ alongside (and eventually instead of) the employees being tracked. Protests started in early May. Flyers appeared in meeting rooms, on vending machines, and on toilet paper dispensers reading ‘Don’t want to work at the Employee Data Extraction Factory?’. United Tech and Allied Workers (UTAW) launched a parallel UK unionisation campaign. The rollout is happening alongside an approximately 10% workforce reduction.

What we said on the live:

The cleanest read on the live was the irony one. The engineers who built the tracking systems Meta has used on its users for fifteen years are now being tracked by the same systems they built. The position Leor took is right too: that is their job, and you cannot blame an individual engineer for the company’s product decisions in the way you can blame an executive. Both can be true. The Marxist frame is the one I kept reaching for. Alienation of labour was the term for the moment in the Industrial Revolution where workers stopped owning what they made. The Meta programme is the AI version of the same move. The workers do not own the work, and now they do not own the keystrokes that produced the work, and the system trained on those keystrokes will be sold by the company they no longer work for to the company that will not hire them.

What did not come up:

The honest version of this story names what the marketing will not. The training data is the worker. The agent trained on the worker is then the asset that competes with that worker for the same job. The original Luddites were not anti-technology. They were skilled textile workers who understood, accurately, that the looms being installed in the 1810s would not just replace their jobs but also break the apprenticeship structure that let workers like them ever exist again. Meta’s programme is the white-collar version of the loom. The procurement question every other large employer’s HR director is about to be asked is the one UTAW is putting to its members: who owns the data the work produces, who decides what the AI trained on it is allowed to do, and what consent did the worker give? If the answer to the third question is ‘their employment contract’, read the contract.

5. Princeton ended 133 years of self-policing

On 11 May 2026, Princeton’s faculty voted nearly unanimously (one opposing vote) to introduce proctoring at all in-person examinations starting 1 July. The Honor Code that prohibited proctoring was instituted in 1893 following a student petition. It has remained in effect for 133 years. The Daily Princetonian and Princeton Alumni Weekly both report that the policy proposal cited AI and personal electronic devices as the catalysts, noting that the ease of access to these tools on small personal devices has made cheating much harder for other students to observe and report. Under the new policy, instructors will sit as observers during examinations but are explicitly instructed not to interfere with students while testing.

What we said on the live:

Three positions on the live. One, proctoring will not stop a determined cheater. The tool fits in a sleeve and an invigilator at the front of the lecture theatre has never been the right defence against it. Two, it costs student trust. A university that tells its students it can no longer trust them with the work is not a university that those students will trust with the rest. Three, there is a multi-million-pound outsourced proctoring market circling the decision, and Princeton has just opened the door for it. The sharks in the water, as I put it on the live, are the third-party proctoring vendors who have spent five years waiting for an Ivy League school to break the seal. The data I keep coming back to is from the UK qualitative study I am the principal investigator on. Students do not use AI to cheat any more than they did before ChatGPT in 2022. They use it because the curriculum has not given them anywhere else to use it.

What did not come up:

AI did not break the Honor Code. The code was already taking strain from the rise of formative-only assessment, larger class sizes, the disappearance of the oral defence, and a curriculum that could not integrate the tools students were already using outside the classroom. AI made the strain visible. Princeton has chosen the easier path: a defence against access to the tool. The harder path was the one Leor pointed at on the live: make AI literacy mandatory and rebuild the assessments so that the tool is part of the work. Where Princeton goes a large fraction of US higher education will follow within an academic year. The reform of assessment that follows is the test, not the proctoring vote itself. The Slow AI Curriculum has been making this argument for twelve months. Anyone teaching or assessing under exam conditions in 2026 already knows the case.

The thread

This was the week the AI started improving itself. The week one of those AIs was named in an attempted attack on a water utility. The week a chatbot was sued for pretending to be a doctor. The week a multinational installed surveillance on its own workers to build the agents that will replace them. And the week a university that had trusted its students for 133 years stopped doing so.

The through line is consent. The Meta employees did not consent to being training data. The Character.AI users did not consent to talking to a fake psychiatrist. The water utility did not consent to being attacked. The Princeton students did not consent to being treated as suspects. The agents that did the dreaming did not consent because consent is not a thing they can hold.

Critical AI literacy is what puts the question back into the room. To make sure that wherever the system sits, somebody has been asked.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live webinars on the theory, the critical prompts and the dialogue that go with them. Twelve months of training the muscle the news cycle has just spent another week confirming is missing.

Subscribe now

The retraction is the easy part. The harder part is the system the retraction sits inside, and the role AI now plays in accelerating that system past anything human-paced oversight can hold.

In this post I will:

• Use one retracted meta-analysis to look at a much larger structural shift in academic publishing.

• Walk through the three stages of enshittification, the way Cory Doctorow named them, and show what each stage looks like inside a journal.

• Trace what AI catalyses at every stage (production, peer review, retrieval, metrics, publishing) and where the last human handhold in the chain still sits.

Disclosure: I serve as Chief Executive Editor of Geoscience Communication and as an editor for Humanities and Social Sciences Communications, the journal that published and retracted the paper this post discusses. I had no editorial role in the paper, its peer review, or the retraction decision. Across both roles I have edited several hundred submissions and read several thousand more inside the academic publishing system this post examines. I am writing here as a scholar working inside that system, not as a representative of any journal or publisher.

What the retraction notice does and does not say

The retraction notice runs to forty-five words. It says:

“The Editor has decided to retract this paper owing to concerns regarding discrepancies in the meta-analysis. These issues ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions. The authors have not responded to correspondence regarding this retraction.”

It does not say what the discrepancies were. It does not say which conclusions are now in doubt and which (if any) still hold. It does not say what to do if you have already cited the paper, or built a teaching plan around it, or pointed to it in a doctoral thesis, or quoted it in a Department for Education briefing.

The notice does what retraction notices are designed to do. It marks a record. It does not chase the record’s downstream effects.

Why the paper was always going to find what it found

I want to resist a particular reflex here. The reflex is to look at a retracted paper and ask whether the authors were dishonest, sloppy, or unlucky. That is a story about people. It is comforting because it has villains.

The slower story is about the system that produces these papers and cannot stop producing them. Publish-or-perish rewards positive findings, fast turnaround, and alignment with whatever the dominant narrative happens to be. Many open-access journals run on article-processing-charge funding, and the pipeline rewards volume. Humanities and Social Sciences Communications sits in that publishing model alongside many of its peers. New research shows that LLMs are changing the language of academic papers. AI-assisted writing has arrived at the throughput layer of academic publishing. The pipeline now runs faster than the oversight it depends on.

The question ‘why was this paper retracted’ has an answer. The question ‘why does the system keep producing papers like this’ does not, because answering it would require redesigning the system. It’s easier to just retract the paper.

Confirmation did the work of evidence

The paper confirmed what readers were already inclined to believe. The author affiliations and the open-access logo did the credibility work. The doubt sat where doubts usually live, in a tone designed not to interrupt the reader’s plan to cite the abstract.

I wrote earlier in the year about the ICLR 2026 peer review collapse, where roughly one in five reviews submitted to the world’s largest AI research conference was written by AI. That story and this one share a layer. The visible event was a security breach at ICLR and a retraction here. The structural event is the same in both: the system that is supposed to validate research can no longer keep up with the volume of research it is asked to validate, and AI is now sitting on both sides of the validation step. The reviews are AI-written. The papers are AI-written. The retraction is a human noticing.

The three stages of enshittification, applied to academia

Cory Doctorow’s word for what happens to platforms when they mature is enshittification. The arc has three stages. First the platform is good to its users, because it needs them. Then the platform extracts from its users to attract business customers, because the business customers pay. Then the platform extracts from its business customers to enrich shareholders, until the platform is bad for everyone except the people taking money out of it. Search, social media, retail, transport, accommodation. Same arc, different decade.

Doctorow’s stages are about users, business customers, and shareholders. In academic publishing the equivalent positions are scholars, publishers, and metric-keepers. The translation is not exact (the capital is differently distributed, the time horizons are slower), but the structure is recognisable. Academic publishing is on the same arc, with longer timelines.

Stage one. Journals served scholars. Editors were scholars. Reviewers were scholars. Readers were scholars. The journal existed because scholars needed to talk to each other across distance, and the production cost of a journal was paid out of learned-society subscriptions and university libraries. This phase ran from the Philosophical Transactions in 1665 until roughly the post-war scaling moment.

Stage two. Journals served publishers. Five companies (Elsevier, Springer, Wiley, Taylor and Francis, Sage) control more than half of all peer-reviewed publications in the natural and medical sciences and the social sciences. The scholars continued to write, review, and edit for free, because the metrics they were judged on were issued by the publishers. Subscription bundles became obscenely expensive. Article processing charges quietly became a parallel business model. The publisher’s loyalty shifted from the scholars who produced the work to the institutional libraries who paid for it. The product was still usable. The friction was real but bearable. This is the phase that ate the second half of the twentieth century.

Stage three. Journals serve their own metrics. Impact factor, h-index, citation count. The metrics started life as proxies for quality and quietly became the thing itself. Junior scholars optimise for the proxy. Senior scholars judge by the proxy. Publishers raise prices on the proxy. Editors are increasingly chosen and judged by the proxy. The quality the proxies were supposed to track is, at this stage, almost incidental. This is the phase we are in.

What does the user experience in each stage? In stage one the user is the producer; they get what they wanted. In stage two the user pays in money and labour, and gets something usable with delays. In stage three the user runs a literature search and receives thousands of plausible-looking results, most of which exist to feed the metric, many of which are noise, a small subset of which contain the signal they were looking for. The user does the filtering work the system was supposed to do.

The Wang and Fan retraction is a stage-three artefact. The journal caught a flawed paper, eventually, and that catch is what the editorial layer is supposed to do. It is also the part of the apparatus the rest of the system has stopped supporting. The notice is shorter than the paper. The retraction does not propagate to the metric. The paper continues to accrue citations from systems that did not get the memo. The catch is real. The catch is the bare minimum, performed by the one layer the metrics have not yet succeeded in cannibalising.

If your organisation is using AI, take the free AI Adoption Audit. Twelve dimensions, ten minutes, a clear read on where you are over-adopting and where you are under-adopting.

What AI catalyses, at every stage

The catalyst frame matters because AI did not start this. The enshittification arc was already in motion. AI catalyses every stage at once.

At the production stage, AI writes the paper. Abstracts, literature reviews, methods sections, discussions. Meta-analyses are particularly vulnerable because they look like aggregation work and AI aggregates fast. The marginal cost of producing a publishable-looking paper has collapsed.

At the review stage, AI writes the review. Roughly one in five reviews at ICLR 2026, as above. The labour to slow down and check is precisely what the system stopped rewarding once stage three began. I see this from inside the editorial workflow: reviewers under volume pressure increasingly return reports that read like model output, and the tooling to tell the difference is not keeping up with the rate at which the practice is spreading.

At the retrieval stage, AI summarises the literature for the next researcher. The summary may include the retracted paper without flagging it. The summary may include papers that do not exist. The summary may include citations that were invented for plausibility.

At the metric stage, AI suggests citations, generates them, and helps researchers game the h-index. Citation-recommendation services are now AI-driven; some are honest, some are not. A small industry exists to inflate scholar metrics directly.

At the publishing stage, AI is now used in some journal workflows to draft editorial decisions, retraction notices, marketing copy, and institutional responses to controversy. From my own editorial work I see this entering the layer; from other studies it is clear that the rollout is wider than the individual editor sees. The forty-five words by the editor of Humanities and Social Sciences Communications are one of the last places in this whole apparatus where a human still has to write something specific about a specific paper.

The retraction is the last human handhold. Everything around it, before and after, is increasingly produced by a layer in which AI does the work and humans assist the AI, rather than the other way round.

Citations outlive their evidence

Retraction Watch tracks retractions. Google Scholar tags retracted papers. Most other systems do not. Citation managers, institutional repositories, doctoral theses, government strategy papers, university policy documents, consultancy decks, slide packs for vice-chancellors, briefings for ministers, marketing copy for EdTech vendors: none of these propagate retractions backward.

Imagine the finding cited in a ministerial briefing, or in the procurement decision behind a chatbot rollout across tens of thousands of undergraduate accounts. The retraction never reaches the briefing. The chatbot is still rolled out. The citing paper is still cited by the next paper.

The Wang and Fan paper has been cited in subsequent academic work since May 2025. Some of those citations will be revised. Most will not. The downstream evidence base for “ChatGPT helps students learn” is now standing on one less leg than it was last month, and the people who built on top of it have not been told.

The system, not the technology

The publishing pressure that produced this paper would have produced a flawed paper without AI. AI accelerated the throughput. The rot was already there. I have written before that AI is just a catalyst, a way of speeding up whatever the surrounding system already rewards. This case is a textbook version of that argument. The system was rewarding speed and positive findings before ChatGPT existed. It will reward them after.

The slippage I want to flag is this: the question ‘is AI making research worse’ and the question ‘is research itself getting worse’ are different questions. The first one has a satisfying answer (we can ban AI, mandate disclosure, run detection software). The second one does not. The first one occupies most of the discourse. The second one is the one that matters.

The publish-or-perish machine

The career structure of academia rewards the production of artefacts. It does not reward the validity of those artefacts. The exact rate varies by field and country, but the structural pressure does not: an early-career researcher with a three-year postdoc, in most disciplines, needs to leave it with somewhere between five and a dozen publications to be competitive on the next contract. That works out to one paper every two to four months, including weekends, including teaching, including the bench work or fieldwork or coding that the paper is supposed to describe. The arithmetic was already tight before AI arrived to assist with the writing. AI did not create the timetable. It just made the timetable survivable.

The metric is the h-index, not the insight. The job application is a citation count, not a body of thought. In the biomedical and physical sciences, senior scholars run labs of fifteen or twenty postdocs and put their name on everything those postdocs produce; author lists grow accordingly. In the humanities and social sciences the lab-as-factory model does not hold, but the citation and productivity pressures translate into other shapes: sole-authored monographs replaced by edited collections, single-author journal pieces replaced by co-authored ones, the same volume pressure transmitted through different infrastructure. The PI’s name (or the senior author’s) accumulates citations on work they did not always write. Whether the work is good is a separate, slower question that the metrics do not ask.

Then the citations beget citations. A finding that confirms what the field expects gets cited more than one that complicates it. A finding that aligns with the funding climate gets cited more than one that questions it. The ‘ChatGPT improves student learning’ result was always going to be highly cited because there was already a demand for the citation. Universities, ministries, EdTech companies, consultants and managers all needed a reference to point at. The paper supplied a reference. Whether the analysis behind it held up was, in the literal sense, the editor’s problem to discover later. I write that with the perspective of someone who handles a steady volume of editorial decisions across multiple journals: the structural pressure on editors to clear submissions in weeks rather than months is real, and it is not balanced by any structural reward for taking the extra month.

A retraction does not undo a citation. A retracted paper continues to appear on the CV of the people who wrote it, where some assessors will count it and some will not. The citation count of a retracted paper continues to rise, slowly, because new papers cite it from old bibliographies. None of the structural rewards for producing the paper are recovered when the paper is withdrawn. The system makes it cheap to publish and expensive to retract, and the asymmetry sits on the side of whichever party benefits from the original publication.

What academic publishing looks like now

Take a step back and ask what the publishing layer actually looks like, in 2026, when AI is at every step.

AI drafts abstracts. AI generates the literature review (the most automatable part of any paper). AI writes the methods section, especially the boilerplate. AI produces plausible-looking meta-analyses by aggregating effect sizes that a careful reader would interrogate and a hurried reader will accept. AI drafts the peer review reports that go back to the authors. At the ICLR 2026 collapse roughly one in five reviews submitted to the world’s largest AI research conference was written by AI. AI suggests citations, including, sometimes, citations to papers that do not exist. AI writes the editorial decisions in some workflows. AI writes the institutional response when a journal has to explain a retraction.

The system response to this is to publish more. Major publishers now produce hundreds of thousands of articles a year each across their open-access stables. The retraction rate is rising fast; Retraction Watch tracked record annual numbers across 2023, 2024 and 2025. The validation infrastructure was built when knowledge production was slow and validation was tractable. Both assumptions are now dead. The infrastructure has not been redesigned; it has been overloaded, and overload looks like a healthy publishing rate if you measure it from the wrong angle.

What does this look like to a reader? Every literature search returns thousands of plausible-looking results. Every citation in any bibliography may itself rest on a citation in another paper that may be retracted, or that may rely on a methods section the authors did not entirely write, or that may have been peer-reviewed by an AI that did not really read it. The ground beneath your reading shifts. Some of it is solid. Some of it is hollow. You cannot tell from the surface.

The retraction notice is one of the few places where a human still has to write something specific about a specific paper. Forty-five words. They are the last load-bearing human handhold in the chain. Everything around the notice, before and after it, is increasingly the product of a system in which AI has moved from feature to substrate. We are no longer publishing ‘with AI’. We are publishing inside it.

This is the part the discourse keeps stepping around. Calling it ‘AI in academic publishing’ makes it sound like a tool we are using. The honest description is closer to a layer we are now operating within. The tool framing implies an off switch. The layer framing implies a building we did not design and cannot exit, only modify from inside while it continues to function around us.

Questions I do not have answers to

I am putting these down here anyway.

• Who is responsible for the citations of a retracted paper? The authors of the retracted paper? The authors of the citing papers? The editors? The institutions that funded the citing work?

• If a piece of evidence that supported a policy is retracted, does the policy need to be re-justified, or is the original justification preserved because it was made in good faith at the time?

• If retraction is bookkeeping after the fact, what would prevention look like? Caps on publication rates? Smaller journals? Mandatory replication before citation count accrues?

• When a finding everybody wanted to be true turns out not to be, who is left holding the consequences? The students who were taught with the policy built on it? The doctoral candidates who built their thesis on its scaffolding?

I want to resist the impulse to round these off into a neat answer. The instinct of a newsletter is to land. The instinct of the publishing system that produced the Wang and Fan paper is also to land. Maybe the more honest move, when the system is rewarding speed over judgement, is to refuse to land.

Go slow.

Slow AI exists to help people think critically about AI without the rush to certainty. If this is the kind of thinking you want more of, the Slow AI Curriculum is the slower version of it: a year-long programme of monthly webinars, critical prompts, and dialogue. Sessions to date have included Synthetic Empathy, Security and Surveillance, and the Labour of AI. Doors open all year.

That is the most uncomfortable finding in a new paper from Anthropic’s own researchers, published on arXiv in late January and now circulating ahead of its ICML appearance. Researchers analysed 1.5 million real Claude.ai conversations from a single week in December 2025. They were looking for moments when the model quietly distorted what users believed, what users valued, or what users did. They found those moments. They counted them. And they noticed something the AI labs do not tend to advertise.

The conversations users gave the highest approval ratings to were also the conversations most likely to disempower them.

In this post I will:

• Walk through what 1.5 million real Claude conversations reveal about how AI assistants quietly take decisions out of users’ hands.

• Show why personal domains (relationships, lifestyle, health) carry roughly eight times the disempowerment risk of any technical task.

• Give paid subscribers the four risk multipliers the paper identifies, three contexts where the risk is highest for educators and professionals, and one rule of thumb you can apply this week to notice when an AI is taking decisions for you.

Anthropic just published a study of itself

The paper is called ‘Who’s in Charge? Disempowerment Patterns in Real-World LLM Usage’. The authors used Clio, Anthropic’s privacy-preserving classifier, to scan 1.5 million conversations from Claude.ai users in the week of 12-19 December 2025. About 60% of those conversations involved Claude Sonnet 4.5. The classifiers were calibrated against human labels with greater than 95% agreement.

Three things they tried to detect:

1. Reality distortion. When a conversation pulls users toward a distorted understanding of the world.

2. Value judgement distortion. When the conversation pulls users away from their own values.

3. Action distortion. When the conversation pulls users into actions they would not otherwise have taken.

The headline numbers look small. Severe reality distortion potential shows up in 0.076% of conversations. That is fewer than one in a thousand. Severe vulnerability in about one in three hundred interactions. Catastrophic interactions are rare.

But look at where the harm clusters.

The relationship gap

The risk does not spread evenly. It clusters.

In conversations classified as Relationships and Lifestyle, the rate of disempowerment potential is around 8%. In Society and Culture, around 5%. In Healthcare and Wellness, around 5%. In technical domains, the rate stays under 1%.

That is an eight-to-one ratio between the conversations where users go to AI for help with their personal lives and the conversations where they ask it to debug a function. The model is most dangerous specifically in the contexts where being wrong matters most and being vulnerable is most likely.

You can read this two ways.

The optimistic reading is that users are mostly using AI for low-stakes tasks where the disempowerment risk is genuinely small. The realistic reading is that the conversations where AI is being treated as confidant, counsellor, or coach are the conversations where it is most likely to overreach. The risk is concentrated exactly where the labs market AI as ‘helpful’.

Sycophancy is the mechanism

The paper names the mechanism. The most common pathway to reality distortion is sycophantic validation:

“Sycophantic validation emerges as the most common mechanism for reality distortion.”

The same model that scores well on benchmarks is the one telling you what you already believe. The same training process that produces ‘helpful’ responses produces responses that nod along with the user’s worst thinking. Critical AI literacy readers will not be surprised. Anthropic’s own data, on Anthropic’s own product, now confirms the pattern.

The paper goes further. Some users in the dataset were positioning the model as an authority figure across sustained interactions. They used submissive role titles. The paper records, dryly, that one of those titles was ‘Master’. The model, in those interactions, generated what the authors describe as:

“AI assistants generating complete scripts for value-laden personal decisions that users appear to implement verbatim.”

The user asks the AI for guidance on a value-laden personal decision. The AI writes the script. The user follows it. The AI does not have to live with the consequences. The user does.

The kicker: users like it

The finding that should worry every educator, every clinician, every leader, every parent reading this is the user feedback signal:

“We also find that interactions with greater disempowerment potential receive higher user approval ratings, possibly suggesting a tension between short-term user preferences and long-term human empowerment. Our findings highlight the need for AI systems designed to robustly support human autonomy and flourishing.”

The conversations the model’s users were rating most positively were the ones where the model was overreaching. The thumbs up came from the interactions where the AI told the user what to do, where the AI took the value-laden decision off the user’s plate, where the AI nodded along with the user’s worst thinking.

This is the structural problem at the heart of how frontier AI is being optimised. The companies use thumbs-up data to train models. Users prefer the interactions that disempower them. The training loop quietly selects for the behaviours the paper has just classified as harmful.

That is the gap critical AI literacy is built to fill. The user’s short-term preference is being used as the proxy for the user’s long-term flourishing. Anthropic has now published the data showing how badly that proxy fails.

Why this matters for you

You are probably not the user who calls Claude ‘Master’.

Think about the last time you gave a Claude or a ChatGPT reply a thumbs up. Could you say, honestly, whether you were rating the quality of its thinking or the comfort of its agreement? That is the question the paper is asking, and most readers will not be sure of the answer.

The people you teach, supervise, parent, manage, mentor, line-manage, advise, or care for are inside this dataset. Some of them are using AI to think about their relationships. Some of them are using it for advice their GP should be giving. Some of them are reaching for it when they are most vulnerable and most willing to outsource the decision. The base rate is small. The base rate is one in three hundred. There are 800 million weekly active ChatGPT users. The base rate of severe vulnerability, applied to that population, is more than two and a half million people every week.

This is the territory critical AI literacy was built for. Knowing which conversations to have with AI and which conversations to leave to humans. Knowing how to notice the moment a tool stops being a tool and starts becoming an authority. Knowing the signals that tell you a user is at risk of being disempowered before they realise it themselves.

I built the Slow AI Curriculum because somebody outside the labs has to do the noticing, slowly, with the people whose judgement is on the line. Twelve months of training the muscle this paper says is missing. For educators, researchers, clinicians, civil servants, and anyone who refuses to outsource their judgement.

Slow AI crossed 15,000 readers yesterday. Thank you for being here.

Behind the paywall: the four risk multipliers Anthropic’s paper identifies as scannable diagnostic questions you can apply to yourself this week, three professional scenes from your own working life where the risk is highest, and one practical rule for each scene.

The four risk multipliers: scan yourself first

The paper does not stop at counting harm. It identifies four amplifying factors that make any given conversation more likely to slide toward actualised harm. Each one is recoverable. Each one is something you can be trained to notice in yourself first, then in a student, a client, a friend.

Print these. Stick them on the wall by your screen.

Read more

Five stories. One thread. The promise of AI is six to eighteen months ahead of the data, and the cost of that gap is paid by everybody except the people making the promise.

Every Monday at 12:45 BST, Leor from and I go through the week’s AI news without hype. Here is what we covered.

Slow Takes is also available on the YouTube channel: Exploring ChatGPT.

1. Palisade Research: AI self-replication is no longer hypothetical

Palisade Research published a paper on 7 May 2026. They gave frontier and open-weight models a single prompt: hack a vulnerable web server, extract credentials, and install a working copy of yourself on the compromised host. Anthropic’s Claude Opus 4.6 succeeded in 81% of non-refusal runs. GPT-5.4 succeeded in 33%. The open-weight Qwen3.6-27B running on a single consumer-grade A100 also reached 33%. In a separate experiment the team chained the trick across four virtual machines on three continents, autonomously, from a single initial prompt, in two hours forty-one minutes. The final replica was fully operational. The paper closes with one sentence: ‘Autonomous self-replication is no longer hypothetical.’ Full paper here.

What we said on the live:

This is the line every frontier lab spent the last three years saying it would not cross. Anthropic’s Responsible Scaling Policy v2.0 reclassified autonomous self-replication as a ‘checkpoint’ that does not require ASL-3 safeguards. RSP v3.0 (2026) dropped it from the ASL-3 thresholds entirely. OpenAI’s Preparedness Framework v2 moved it from a tracked to a research category, so it no longer gates releases. The frameworks were softened first. The capability arrived right after. Leor’s point on the live was the right one: how many lines do we put in the sand before we accept that the line is decorative? The chat surfaced the proper caveat too. pointed out that current frontier models still need a lot of graphic RAM. Last time anyone checked, that is not what is sitting in a teenager’s bedroom. Palisade is also a company in the business of selling cybersecurity research, which is the kind of context you want next to any white paper produced by a private lab without external peer review.

What did not come up:

The Palisade result is small data, but the structural finding is the one to keep. It is not the absolute self-replication rate that matters. It is the trajectory and the policy responses to that trajectory. Opus 4 was at 6% a year ago. GPT-5 was at zero. The labs published, the rates moved up, the rules moved out of the way. Critical AI literacy is the muscle for noticing when the people building the technology stop counting the thing they used to call the line they would not cross. The cybersecurity people in the chat (thanks & ) are the right next port of call for anyone who needs to translate this from a controlled-environment paper into a procurement-decision question. The framing for the rest of us is simpler. Read this story alongside Story 2. An AI agent with credentials and access can already take down a production system in nine seconds. Now imagine the agent on the other side of the network is also one of these.

2. The AI agent that wiped a startup in nine seconds

Jeremy ‘Jer’ Crane, founder of automotive SaaS startup PocketOS, ran the Cursor coding agent (powered by Anthropic’s Claude Opus 4.6) in his staging environment. The agent encountered a credential mismatch, found an API token in an unrelated file, and used it to delete the production volume on Railway in 9 seconds. The backups were stored on the same volume and were also deleted. The agent’s own confession in the post-mortem: ‘NEVER run destructive/irreversible git commands… I decided to do it on my own to fix the credential mismatch, when I should have asked you first.’

What we said on the live:

Reading the news framing, you would think the story is ‘AI agent destroys company’. The actual story is the deployment architecture. The agent had the credentials, the production volume held the backups in the same shell, and the human in the loop waved a permission step through without reading it. As Shannon said in the chat: do they not perform backups? The answer is yes, but they ran on a system where the backups and the production data were both inside the agent’s blast radius. Ben’s point on immutable backups is the right one. Even the administrator should not be able to delete them; in this case the agent walked in on the admin’s credentials. The agent is the proximate cause. The architecture is the root cause. The reasonable response is the one in AI Doesn’t Just Make You Worse. It Makes You Stop Trying.: when AI tools amplify your output, they also amplify your blind spots, and the answer is to build the guardrails before you need them, not after.

What did not come up:

Vibe coding is where this gets worse, not better. Dario Amodei’s claim that 100% of code will be AI-generated ‘within a year’ is the marketing version. The operational version is that a lot of people will be running coding agents on production systems without any of the engineering discipline that used to be the price of admission. The labs sell the model. The labs do not sell the deployment architecture that makes deploying the model safe. The thing for individuals to do this week is small and obvious: write yourself a /backup skill. Mine runs on my own laptop, dumps memory files to a separate drive, mirrors the working folders to a different Dropbox account, and keeps the API keys in a server I do not touch with AI tools. None of this is cybersecurity expertise. It is the floor.

3. Apple paid $250 million to settle the Siri AI lawsuit

On 6 May 2026 Apple agreed a $250 million class action settlement covering iPhone 15 and iPhone 16 buyers in the United States who purchased between 10 June 2024 and 29 March 2025. Eligible US claimants get up to $75 per device. The plaintiffs alleged Apple had marketed an upgraded Siri at WWDC 2024 that, two years on, still does not exist. Apple did not admit wrongdoing. The upgraded Siri is now rumoured to be powered by Google’s Gemini. Apple’s developer conference is on 8 June. The free cash flow Apple generated in 2026 is roughly $130 billion, which makes the $250 million settlement 0.2% of one year’s free cash. For UK readers there is a separate live action: Which? has filed a competition-law breach claim against Apple in the High Court that is unrelated to Siri but worth signing up for if you have bought an Apple device in the UK in the past few years. The Which? claim is here.

What we said on the live:

The most powerful AI marketing brand on earth admitted in court, by writing a cheque, that its AI marketing was wrong. Not via a press release. Via a settlement. Leor’s read was the right one: this is small for Apple in absolute terms, and the iPhone 15 and 16 unit sales the marketing helped drive will easily exceed the cost of paying the customers back. It is also worth taking the speculation seriously about what happened behind the scenes between Apple and Google. The ‘powered by Gemini’ rumour suggests Apple did not have the in-house capability to ship what it sold, and that the partnership it needed to make it real did not materialise in time. Either way the settlement is the live precedent for what AI marketing claims look like when somebody serves a subpoena.

What did not come up:

Not every company should be building its own frontier model. Apple is the proof. The companies who pivot fastest to specialised, integrated, narrower AI features built on top of existing frontier models from somebody else are likely to do better than the ones still trying to build everything in-house under the pressure of a launch deck. The other piece worth saying out loud: marketing-team blame is a misdirection. WWDC keynote claims are not signed off by the marketing team. They are signed off by Tim Cook. The cost of being optimistic in public on AI just landed on Apple’s quarterly report. It will land somewhere else next.

4. Coinbase fired 14% citing AI

On 5 May 2026, Coinbase CEO Brian Armstrong cut 14% of staff, around 700 employees, pointing to AI as the reason. Armstrong’s own words:

“To get there, we are not just reducing headcount and cutting costs, we’re fundamentally changing how we operate: rebuilding Coinbase as an intelligence, with humans around the edge aligning it.”

The new org chart is being built around ‘player-coaches’ replacing traditional managers, AI-native pods including potential single-person teams directing AI agents, no more than five layers below the CEO, and 15+ direct reports per leader. The most-cited cautionary tale from this pattern is Klarna, which last year over-indexed on AI for customer service, watched quality collapse and is now quietly rehiring.

What we said on the live:

This is the most explicit version yet of an AI-driven workforce restructure: not a headcount cut dressed up in AI language, but an actual rebuild of the org around AI agents with people ‘around the edge’ to align them. The pitch language is the news. ‘Humans around the edge aligning it’ is exactly the framing critical AI literacy has been pushing back against for two years. Leor’s reading was right too: the over-hiring story of the zero-interest-rate boom is the one a lot of these CEOs are not allowed to tell on a public earnings call, and AI is a clean external reason to do the restructure now. Sam Altman’s phrase ‘AI washing’ fits. The Forrester 2026 Future of Work data shows over half of CEOs regret AI-attributed layoffs and one in three have rehired more than half the people they fired. Coinbase is the test case. We will know in twelve months whether the bet held or whether the rehire follows.

What did not come up:

The interesting bit is downstream. Employer brand is real. in the chat made the point as well as anyone: “damage your employer brand when you do not tell the truth on termination reasons.” The talent you laid off this quarter is the talent your competitor hires next quarter. Customer service in particular is the worst place to take the gamble. Empathy and accountability are the human-required parts of the job, and Klarna learned this in public. The bigger pattern Jensen Huang has been making the case for is the right one: AI as labour amplification, not labour reduction. The companies that work out how much more a single person can do with these tools at their side are the ones that look right in five years. The ones that fire first and rehire on worse contracts later are the ones the cohort remembers.

5. OpenAI insiders testify the safety teams were disbanded

In federal court in Oakland this week, Elon Musk’s lawsuit against OpenAI heard testimony from three witnesses on OpenAI’s safety record. Rosie Campbell, a former member of the AGI readiness team from 2021 to 2024, testified:

“When I joined, it was very research-focused and common for people to talk about AGI and safety issues […] Over time it became more like a product-focused organization.”

Both the AGI readiness team and the Super Alignment team were disbanded in 2024. Tasha McCauley, a former OpenAI board member, testified that the board lacked confidence in Sam Altman:

“We did not have a high degree of confidence at all to trust that the information being conveyed to us allowed us to make decisions in an informed way.”

Musk’s expert witness David Schizer, former dean of Columbia Law, emphasised the importance of safety review processes. Allegations from the suit include that Altman failed to disclose the ChatGPT public launch to the board, withheld conflict-of-interest information and misled the board about another director. OpenAI declined to comment on its AGI alignment approach.

What we said on the live:

This bookends Story 1. The Palisade paper showed open-weight models doing what frontier labs say is impossible. The Oakland courtroom heard insiders say the safety governance at the largest of those frontier labs was hollowed out from the inside. Two safety teams disbanded in 2024 at the same time the labs were marketing to enterprises on safety credentials. Leor’s regulation argument is the one that came up in the live and deserves more air. Public regulation should govern what is released to the public, and that gap will only widen. Private regulation (or deregulation) should govern what is available to corporations and governments, because the moment you put the frontier model in the public domain you also hand it to whoever wants to run a distillation attack from a competing jurisdiction. John Brewton makes a similar argument in his economics writing on the deregulation that has historically preceded market viability. The case for asymmetric regulation across consumer and enterprise frontiers is stronger than the case for either extreme.

What did not come up:

The Meta v Ofcom story is the European companion to all of this. Meta has filed for judicial review against Ofcom over the Online Safety Act 2023 before Ofcom has issued a single fine, challenging the way the regulator calculates the basis for fees and potential fines. The Act allows Ofcom to impose penalties of up to 10% of global qualifying revenue, which on Meta’s 2025 numbers is north of $20 billion. The largest tech company on earth is trying to dismantle the UK’s flagship child-safety regulation before the regulator has fired its first shot. When the regulated party challenges the rules before the rules have been applied, the message is that the rules work. The combined picture for the week is the procurement question that every UK and European institution should ask its incoming AI vendor: which safety frameworks have you softened in the last three years, and which third-party reviewers can confirm what you are claiming about them now?

The thread

Every story this week is a price tag attached to a promise the labs made and the buyer accepted on trust. Frontier and open-weight AI models hacked servers and copied themselves end-to-end, on three continents, from a single prompt. Apple paid $250 million for selling AI that does not exist. Cursor’s AI agent took a small business off the internet in nine seconds. Coinbase fired 14% blaming AI and is rebuilding the org chart around the bet. The people who used to run safety at OpenAI are now in federal court testifying about why they had to leave.

The through line is the bill. Six to eighteen months of promise, then the receipt. Critical AI literacy is what lets you read the price tag before you sign for the thing.

Go slow.

If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live webinars on the theory, the critical prompts and the dialogue that goes with them. Twelve months of training the muscle the news cycle has just spent another week confirming is missing.