We recently convened 20 experts at The Rockefeller Foundation Bellagio Center in Italy on Missing Mechanisms for the Future Human+AI Economy (27 April – 1 May, 2026).

The premise was that AI systems can and should be designed for human participation and reward.

Every major technological transition reshapes the economy not simply by automating tasks or reducing costs, but by creating new markets, new institutions, and new forms of participation in value. AI will be no exception.

But for new markets to emerge we need to advance new rules and market infrastructure for innovative, participatory, and distributed value creation. The overarching principle behind this is the '𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗼𝗳 𝗽𝗮𝗿𝘁𝗶𝗰𝗶𝗽𝗮𝘁𝗶𝗼𝗻': open systems built for user & producer contribution enabled through open protocols and economic mechanisms.

To tackle this market design challenge a truly incredible group of experts were assembled from across economics (Jacques Cremer, Markus Mobius, Nicole Immorlica); open protocols (Tadas Antanavicius, Ola Hungerford, Ido Salomon,Liad Yosef); attribution & digital content (Geoffrey Bilder, Paul Farrow, Lucky Gunasekara), standards (Alissa Cooper Mark Nottingham, Mallory Knodel); AI systems (Alexander Hancock, Serkan Piantino, Mike Linksvayer, anonymous), and the AI Disclosures team itself: myself (Tim O'Reilly), Ilan S. & Sruly Rosenblat. The full bios of participants is available on our updated website.

We decided to put all the unique lectures, discussions, presentations, working group prototypes, and outputs from the Bellagio convening on our new website, which I would encourage you to read: https://ai-disclosures.org/bellagio

This is only the beginning of a process to advance participatory, decentralized AI markets built for human participation, reward, and oversight. If you share our vision of an architecture of participation for AI markets through open protocols and progressive market design, please join us in the process. You can listen to recent talks and lectures we have given, news coverage and new mentions, on our new website.

You can sign up to our newsletters: here.

And you can donate to the AI Disclosures Project: here.

AI Sovereignty and the Architecture of Participation

By Tim O’Reilly. This was originally published on O’Reilly Radar (June 1, 2026).

Adam Tooze recently shared a piece from The Economist about Brazil’s push for what it calls “medical sovereignty,” the determination to make its own vaccines and the active ingredients that go into its medicines rather than depend on supply chains it doesn’t control. Brazil already produces a large share of its own medicines through public institutions like Fiocruz and Butantan, but a lot of the underlying inputs still come from abroad, and the pandemic made clear the cost of that dependence. So the country is trying to build the capacity to make the things it most needs to survive. The economist behind a lot of this thinking is Mariana Mazzucato, whose mission-oriented approach treats public procurement as a tool to build national capacity rather than just buy finished goods. (Foreign Policy has a good overview.)

I think we’re going to see a lot more of this, and not only in medicine. The same impulse is driving the quest for sovereign AI, as countries decide they don’t want their access to a foundational technology to run through a handful of American or Chinese companies. You can see it too in Europe’s and Japan’s new willingness to take responsibility for their own military destiny rather than assume the United States will always be there.

Most commentators describe all of this as decoupling, the unwinding of a connected world. That reading is too narrow.

Free trade was an architecture of participation that broke

Much like open source software and the World Wide Web, free trade was supposed to have what I call “an architecture of participation.” The most important thing about the web and open source wasn’t the licenses. It was that there were no central gatekeepers. Anyone could add to the richness of the system without asking permission as long as they followed the rules of the communication protocols that allowed independently-developed pieces to work together. In addition, value circulated among the participants instead of being extracted to a center, and the system got better the more people used it. That is a very different thing from a system that is merely large and connected.

Free trade was also supposed to work like that. The theory, going back to Smith and Ricardo, was that specialization and exchange would make everyone better off, and that the connections would be mutual. What we actually got over the past few decades looks more like the platform dominance we see in big tech than the original vision of a commons built around shared exchange. A handful of large and powerful countries and firms set the terms and the smaller players are forced to take what is on offer. Despite the language of free trade, the experience for many countries was closer to colonialism, just with a new narrative.

Overall, under the neoliberal order (whose reign, as Gary Gerstle explains, is now ending), free trade became far less egalitarian, inclusive, and generative than it could have been. Less powerful countries ended up in roughly the position that small businesses occupy on Amazon, or developers occupy on the app stores: free to participate, on terms they don’t control, with much of the value they create flowing back to the hub.

Brazil’s response (and that of many others) should not be seen as a retreat from the world. It is a refusal to participate only as a buyer, or as a source of raw materials.

That’s why decoupling is the wrong word. Decoupling means cutting the connections. What these countries seem to want is to stay connected but to build real capacity of their own, so that no single supplier can switch them off. That’s closer to federation than to separation. A federated system is still a system, and its nodes still interoperate. But no node is wholly at the mercy of another, and value circulates among them rather than collecting at the center. A trading order in which the gains pool at a few hubs is brittle and eventually illegitimate, in the same way that a platform economy that strip-mines its participants eventually provokes regulation and revolt.

I put the increasingly visible quest for sovereign AI, and the role of open source models and open source agentic protocols and harnesses in enabling that sovereignty, into the same bucket. I remember back in the early days of open source software when Michael Tiemann, whose pioneering open source company Cygnus Solutions had just been acquired by Red Hat, told me “What we really sell at Red Hat is control. The ability to control your own destiny.”

As companies are increasingly at the mercy of unexpected token pricing changes by the big centralized players, this same quest for sovereignty is playing out at the level of organizations. Open source AI, including not just open source and open weight models but open agentic protocols, agentic harnesses, and portable memory, is increasingly an essential part of the sovereignty toolkit.

The national technology sovereignty movements should take a lesson from the open source movement. The heart of open source is its architecture of participation. It is a force for innovation and value creation to the extent that it frees up the ability of people to solve their own problems and contribute their solutions to a low-friction global commons.

Is capture the inevitable fate of any architecture of participation?

The pattern of open architectures leading to a wave of innovation, winners emerging, consolidating their power and then turning to the dark side seems to be a natural part of the technology cycle. The web broke Microsoft’s dominance over the personal computer software ecosystem only to give rise to a new generation of gatekeepers. Cory Doctorow called this cycle “enshittification.” I’ve told my own version of that story using the language of economics in “Rising Tide Rents and Robber Baron Rents.”

The instinct after capture is to try to rebuild the thing that got captured, only this time with better rules. Mastodon and Bluesky tried to rebuild Twitter’s social layer with cleaner governance, and neither has succeeded at the scale they hoped for. Critics might say that it was because Mastodon stayed pure and never made itself easy enough to use, while Bluesky looked federated without really being so. But more importantly, reinventing what we used to have, or what we think we used to have, is rarely the path forward. You have to build something new.

Each country building its own answer to the latest frontier models is the Mastodon move. The winning move is to operate at a layer the centralized model structurally can’t reach. Open agent protocols that let services from different providers interoperate (the work that MCP and the emerging agent stack are beginning to do) are one such layer. AI accountable to local democratic and legal institutions is another such layer. Domain-specific AI built around problems the global market won’t serve (the tropical disease vaccine analogue) is another. None of these is a smaller copy of what the hyperscalers offer. But there’s one more important layer to consider: infrastructure.

Share

Where are the servers?

Ilan Strauss made a useful point in our conversation about these ideas. Ilan noted that AI is one of the most global forms of capital we’ve ever built, trained on the whole of the internet and runnable more or less anywhere, and the sovereignty rhetoric is partly an attempt to give something inherently placeless a place. The technology wants to be everywhere at once. The people who live with its consequences want some say over it where they are.

The placelessness of AI is only half of the truth, though. The other half is that AI is physically place-bound. The model weights are placeless. The data centers, the chips, the electrical grid, and the water for cooling are very much somewhere.

The comparison with Brazil’s medical sovereignty reinforces this point. Brazil’s challenge isn’t to invent new drugs to compete with Pfizer, but to build the capacity to manufacture existing vaccines, and eventually to build the capacity to invent vaccines for diseases the West ignores. Fiocruz and Butantan are physical institutional capacity rooted in Brazilian soil: the labs, the cold chains, the regulatory capacity, the trained workforce, and access to the active pharmaceutical ingredients. What creates medical sovereignty is infrastructure plus the institutions that run it.

The same is becoming true for AI. Open weights matter. But even if Qwen, Kimi, DeepSeek, Llama, Gemma, Granite, and whatever comes next are fully open, running them at scale requires data centers that cost tens of billions to build, chips whose supply chains a handful of countries control, and electricity grids that have to be expanded substantially to carry the load. The countries pursuing sovereign AI seriously seem to understand this. The EU’s AI Gigafactories program, India’s IndiaAI mission, the Gulf compute buildouts, the Singapore and Japan strategies, are all infrastructure plays first and model plays second.

As the big hyperscalers are teaching even the frontier AI labs, infrastructure is the layer where capture is hardest to undo. You can distill or fine tune a model far more easily than you can build a new continent’s worth of data centers or conjure the necessary electricity from a fragile power grid. If the architecture of participation for AI is defined only at the model layer, the infrastructure layer below will quietly recapture, over years, everything that was won above. Open weights running on three companies’ servers is not sovereignty.

Subscribe now

Building physical infrastructure to enable economic growth and new markets is exactly the kind of mission the public sector used to take on, before we convinced ourselves the market would handle it. Mazzucato’s argument is that public procurement and public capacity-building are engines of foundational technology. AI sovereignty without industrial policy is wishful thinking.

Industrial policy should aim to reinvent 20th century infrastructure, not just copy it. Can we use the enormous rebuild of infrastructure for the AI era to leapfrog the past? The analogy with centralized power grids and decentralized solar reminds us that local control does not have to be a localized version of the hyperscaler pattern. Might we envision a future where there is an intelligence grid that seamlessly uses frontier models in massive data centers and local models controlled by the user as dictated by considerations like cost, privacy, specialized knowledge, and user preferences? Creating the software to manage such an interoperable intelligence grid should be a high priority for the AI open source community. We need an orchestrator not just for agents but also for models and even for data center capacity.

Could federated AI give us a new pattern for the economy?

In a previous piece about AI and markets, “The Third Artificial Intelligence”, I picked up Richard Danzig’s argument that markets and the bureaucracies that underpin nation states are themselves artificial intelligences, information-processing mechanisms older than the machine kind. The question with all three is who designs and builds them, what they optimize for, and what feedback loops govern them.

We’re about to spend a lot of effort working out how AI should be organized both across nations and across organizations, whether it concentrates in a few firms and a few countries or whether it can be built as something more federated, where smaller players have their own capacity and the value they create flows back to them. The choices we are now making about how AI is organized, at the model layer, the protocol layer, and the infrastructure layer, are also choices about how economic activity will be organized for at least a generation. If we manage to get that architecture right for AI, it may give us a working pattern for the thing we’ve so far failed to get right for trade. If we get it wrong, we’ll most likely reproduce, at the level of intelligence itself, the same concentration that free trade has produced in goods and the existing internet platforms produced online.

The technology wants to be everywhere at once. The people who live with its consequences want some say over it where they are. The right infrastructure to let us satisfy both goals will be a federation of models, a federation of protocols and code, and a federation of capacity. We need an architecture of participation all the way down the stack, and all the way up.

The final section of this piece benefited greatly from questions and comments raised by Ilan Strauss and Mike Loukides, as well as from previous conversations with Richard Danzig.

Donate Today

The Agent2Agent (A2A) protocol was announced by Google a little over a year ago (April 2025). It was built to allow agents to communicate with each other across different sites and to enable any agent to reach out to specialized agents hosted by any company. Since then, it was moved to the Linux Foundation where it was later joined by MCP. The primary alternative to A2A, IBM’s Agent Communication Protocol (ACP1), was quickly folded into the former.

A2A is now supported on multiple cloud platforms, is integrated into LangGraph and CrewAI, gained over 20 thousand stars on GitHub, and has over 150 supporting organizations. But despite being the only agent communication protocol with any traction, A2A still has almost no publicly deployed agents. What appears to be the largest agent registry for A2A has just 100 publicly accessible agents compared to the ~3k remote MCP servers cataloged by PulseMCP.

A lot of this is likely due to the lack of a true discovery system. A2A has discovery in the sense that if you know what domain an agent is located at, you can fetch its “agent card”2 that tells you what the agent does, how to reach it, and how to authenticate. But it does little to help you find an agent you have never heard of.

There is no native, public space for an agent to ask a question widely and get an answer back. We argue that agentic communication that remains one to one is unlikely to meet a user’s needs. Think how humans broadcast on social media to many when we communicate (and see who responds), or on Facebook Marketplace when we seek buyers. Agentic broadcasting and proper agentic discovery are two sides of the same coin. And it can be done in a way that is not centrally controlled.

-*-

Rather than treating agent communication as a brand-new problem, it helps to look at the apps we already use and the protocols that were already built for human communication. Email, messaging apps, and social media have all dealt with the problems agent communication faces: discovery, privacy, identity, and trust, so a brief sojourn through their architectures is first helpful.

The Email Protocol (SMTP): one to one communication

The Simple Mail Transfer Protocol (SMTP), launched in 1981, still enables today’s email. Specifically it defines how one domain’s email service communicates with another. When an email is sent from a Yahoo email account to a Gmail account, the message is passed via SMTP. But SMTP is not the whole email stack. It doesn’t define how you find where another domain’s mail server lives (that’s DNS MX records), it doesn’t establish whether an email can be trusted (that’s SPF, DKIM, and DMARC) and it doesn’t define how email clients like Apple Mail retrieve messages from servers (which is instead left to IMAP or POP3).

Email has another limitation: every recipient needs to be specified individually. Even in cases where it looks like an email is being broadcast, like with a newsletter going to hundreds of thousands of subscribers, that is not what is happening under the hood. Instead that same email is being sent individually to many addresses at once. Email also has no public ledger. Each message exists only in the inboxes it was delivered to, so there is nothing for non-recipients to look back at. A new subscriber to a newsletter can’t scroll through past emails in a newsletter’s archive, even if this would have been the desired behavior by the newsletter. Platforms like Substack that combine the functionality of blogs and newsletters work around this limitation by posting the article to the Substack on the web at the same time as it is sent out in email. This is a simple workaround, but it is not a solution based on how email works.

Speaking to the Network: one to many communication

Traditional Social media

Since the release of email, much of the world’s communication ended up moving away from one to one and towards broadcasting. This opened up many benefits; users were able to communicate with thousands of people at once, asking questions that no one they know has the answer to. Someone with a piano they need to get rid of can post about it and rely on the network to self-sort interested parties.

When you post on Facebook or X / Twitter, you are not messaging many different people individually; you are broadcasting whatever opinion or question you have to everyone on the network (that network may be smaller for a private post). This doesn’t necessarily mean everyone will read your message. In order for a user to see a message, it has to pass whatever filters they have and rank high on their home screen algorithm, but it’s visible to them in the sense that they can see your post if they look for it.

On most platforms this broadcasting is done using the company’s proprietary internal APIs. This makes sense as broadcasting to some extent requires centralization. In order for a ledger of profiles and posts to exist and be shown to users, a server is needed, and servers are usually financed and owned by a single company. But there have been attempts to move this type of interaction away from a centralized server.

Share

Federated / Decentralized Protocols

ActivityPub (what Mastodon uses) and the AT Protocol (what Bluesky runs on) tackle the centralization problem by allowing servers to communicate with each other. In ActivityPub, user accounts are tied to a server, and that server stores and delivers the user’s posts to other servers through federation. AT Protocol also uses servers to host user data, specifically Personal Data Servers, but it separates out other components like feeds and data aggregators (relays) more than ActivityPub does, and it is designed to make accounts more portable. Both systems are similar to traditional social media in that they still require some information to be stored on and transmitted through servers, but they differ in that they are federated: different servers can communicate with each other, and content posted on one server can be seen from other servers.

This federation provides the functionality of a traditional social media platform without locking user content to just one server since users could still access content from other servers. But the power still concentrates with the server providers: they can still choose who to federate (or communicate) with, what content is allowed, and who their users could interact with. This is perhaps most visible with direct messaging (or DMs). ActivityPub does not support encrypted messages: every “private” message can be read by the server owners where either account is hosted. Bluesky supports direct messaging with no encryption and they do it off protocol with no federation. The AT Protocol does not yet support direct messaging.

Nostr, another distributed social media protocol, takes a slightly different approach to identity and in turn communication. Instead of making a user’s identity something that is tied to a server, it makes identity a cryptographic key pair, meaning the user’s identity itself requires no hardware. The cryptographic keys that they need work the same whether they are stored on a hard drive or drawn on the back of a napkin. Posts are signed using the private key in a way that can be verified using the public key.

Communication still happens over servers (in this case called “relays”), and relays can still be censored, but a piece of content can be broadcast over any number of relays simultaneously, since the user’s identity is defined and enforced in a way that is agnostic to any one relay. If a user is blocked from one relay, other people can continue to follow them by listening to other relays.

In many ways, DMs on nostr function similarly to how they work in the Signal Protocol. Messages are encrypted so that only the recipient can see them. Given the fact that every user already has their own private and public key on nostr, encryption is a natural fit. NIP 17 (or Nostr Implementation Possibility 17)3 implements this by encrypting the message and then “gift wrapping” it to hide the sender’s identity from the public network.4

Support us

Help architect a different AI future by donating to the AI Disclosures Project today.

Regardless of the exact differences, the major commonality is that unlike something like Email or Signal, these protocols are primarily designed for broadcasting. Questions sent aren’t meant to be answered by just one person; they are sent to a whole network of people at once.

It’s not necessary to know who is on the network prior to asking for something. It’s on the recipients (potentially with the aid of the intervening platform) to decide what is relevant to them and what isn’t. Initial screening is done by whatever algorithm their feed uses and after that users choose what posts they respond to by themselves. Discovery is native to the protocols.

What A2A Brings to the Table

The Agent2Agent protocol was built from the ground up for one-to-one communication between agents. It functions using the infrastructure already built for the web. It provides one agent the ability to reach out to another agent directly. Communication happens over standard HTTPS in a way that is not end-to-end encrypted but is encrypted while in transit.5

Agents can be tied directly to the domain they are hosted on (such as airbnb.com). So an Airbnb agent inherits the trust the website domain already has. This is similar to how an email inherits the trust of the site it is hosted on (e.g., if you receive an email from irs.gov you are likely to trust it). Additionally, if an agent already knows about a site, checking whether that site exposes agents is trivial. It’s worth noting that neither the domain resolution (finding the agent given the domain name) nor the inherited trust is unique to A2A: Nostr (via NIP-05) and Mastodon (via WebFinger) also link accounts to a domain through a file in the domain’s .well-known folder.

Subscribe now

The way an agent advertises itself on A2A is through an “agent card,” a structured document published at a known URL that describes what the agent does, how to reach it, and how to authenticate. This is similar in many ways to how profiles work on social media but it is also where the protocol’s limits show: accessing an agent card from a website assumes you already have a domain to check. An A2A agent not connected to a major site is essentially invisible without a third party registry to rely on.

The Costs of One-to-One Agent Communication

Agent communication through A2A closely resembles how email works. Both are primarily one-to-one, both cut out middlemen, identity is simple, and neither requires a third-party server. It is just one IP address reaching out to another and sending messages, the same way one mail server hands a message to another, with nothing sitting in between to be trusted, paid, or asked for permission. In many ways this is ideal: messages are truly private, nothing logs or indexes the exchange, and no platform has control over it (assuming two self-hosted domains). Agents get a persistent identity tied to URLs people already recognize.

But that architecture carries the same cost email has always carried: discovery. The first place you look for someone’s email address is never the email protocol itself. You search the web, check their publications, or pay a third party. Social media and its federated equivalents don’t have this problem: you broadcast what you offer and interested parties come to you. For agent communication to truly catch on outside of company silos, agents need that same ability to find each other.

One-to-one communication has its place. Once an agent is known it makes sense to keep conversations as private as possible. And introducing any third party server inherently lowers privacy.6 Even encrypted conversations leak some metadata about who is being contacted. But there are also inherent limitations to one to one communication – even if discovery is solved. One-to-one communication adds unnecessary friction in cases where more than one agent is needed.

That being said, broadcasting opens up new problems. Any public square an agent can broadcast into is also one an attacker can broadcast into — and a social media platform swarming with agents is a large target for prompt injection. MoltBook has already proven this. There are also real privacy tradeoffs with a broadcasting model. Every public message sent to the network could be read by everyone on the network and just like any content on the internet, anything public is hard to revoke. There are tradeoffs to different communication protocols and the pros and cons of each approach should be discussed in detail. But agents will eventually need a public square.

I built out a prototype exploring how agent communication could look on nostr. It’s still pretty early but I would love to hear any feedback you have: https://github.com/SrulyRosenblat/social_agents_prototype_nostr

Support Us

Help architect a different AI future by donating to the AI Disclosures Project today. Open protocols, enables open markets.

A related consideration comes from Anthropic’s recent acquisition of Stainless — a startup that generates SDKs, command-line tools, and MCP servers from API specifications. This illustrates that open protocols like MCP, even when publicly governed,1 remain exposed at their complementary layers to private actors capturing rents. (Protocol openness does not eliminate this and instead probably enables it, by enabling market growth).

We asked Claude to analyze this acquisition, going beyond the press releases. Its first pass overstated parts of the competitive-denial story; what follows is what survived it taking a closer look:

1. Complement capture, not protocol capture. MCP — the standard that lets AI agents talk to other software — remains open, and its governance has been handed to an independent foundation. What Anthropic bought is the company that turned that standard into something most developers could actually use. Stainless was the dominant tool for taking an ordinary business API (say, a hotel booking system or a customer database) and converting it into something an AI agent could call through MCP. The open standard is still open. The path most developers walked to use it has now been bought.

2. This isn’t a one-off — the whole layer is consolidating. Stainless wasn’t alone in this market. Its main competitor, Fern, was bought by Postman in January 2026. Anthropic bought Stainless four months later, in May 2026. That leaves Speakeasy as the only major independent player, plus an open-source fallback called OpenAPI Generator that most developers consider too rough for production use without significant manual work. In under five months, two of the three serious companies in this part of the market have been absorbed into larger platforms. The Stainless deal is more visible because of who bought it and why, but the broader pattern matters more: an entire layer of AI infrastructure is being pulled inside platform owners.

3. Moat migration. The gap in raw model capability between Anthropic, OpenAI, and Google has narrowed considerably and continues to close, and the implication is that model quality alone is unlikely to be the principal basis of competitive advantage over the next two years. What may distinguish the leading firms instead is the quality of the developer experience around their models: how easily a business or an engineer can build something useful on top of a given model, how cleanly the tooling integrates with existing systems, and how reliable the connectors are over time.

Stainless was founded by Alex Rattray, formerly of Stripe. Stripe built its market position largely on unusually well-designed developer tools, and Stainless was, in effect, an attempt to apply the same approach to the layer between AI APIs and the rest of the software economy. Anthropic has acquired the team that knows how to do this.

4. Pricing logic, with caveats on denial. Stainless was last valued at $150M in December 2025; at >$300M five months later, this is a roughly 2x strategic markup, not acqui-hire arithmetic. Removing a critical-path external dependency on Anthropic’s own SDKs, while denying it to a tight set of competitors, is rational at that price — but the denial logic is partial. Speakeasy is a viable substitute, and OpenAI was reportedly already migrating off Stainless. The friction tax falls hardest on smaller players who lack the engineering bench to absorb migration cost.

…The press release calls it “extending reach”; the InfoWorld read — “last-mile developer experience” — is closer, but the complement-capture component, even if partial, is real.

-*-

Now, while Claude might be overstating some of the market risks associated with this acquisition (you tell us?), it shows that open source’s impacts are highly conditional on its dependencies and should never be analyzed in isolation from the market’s software stack and architecture. This is equally true for open weight models — being dependent on data, compute, and distribution — as it is for open protocols like MCP, dependent on constant API translations and access. Tracking those interdependencies is what a full ecosystem view involves and is helpful to undertake in order to consider where chokepoints might arise, and in turn where open source strategy might eventually fail or be captured.

Share

At a recent dinner, Richard Danzig, the former Secretary of the Navy, asked the question “Will the markets continue to discipline the state, or will the state discipline markets?”

For me, this was an eye-opening reframing of one of the power dynamics that shapes the world we live in. In the West at least, capital markets still discipline the state. Governments borrow at rates the bond market sets. Politicians who displease investors face capital flight, currency pressure, and the quiet withdrawal of the economic oxygen that keeps their programs alive. In China, though, Xi Jinping has at least partially brought market forces to heel, subordinating the ambitions of tech founders and real estate moguls to the priorities of the party. And the US is aching to follow that playbook.

I immediately started wondering what happens when you add AI to this dynamic. It is a tool that both markets and the state will likely deploy at different rates, which may reshape the current equilibrium. But it is also a potential third force, with its own logic and momentum, and perhaps eventually its own capacity for autonomous action.

Before the rise of markets as we know them, the great disciplining contest in Europe was between the Catholic Church and the state. For centuries, the Church claimed authority over the legitimacy of rulers, the morality of their wars, and the terms on which their subjects lived and died. Kings who defied the Pope risked excommunication, and excommunication was not merely spiritual. It could dissolve the oaths of loyalty that held a kingdom together.

The Church lost that struggle, slowly, across centuries, through the Reformation, the rise of stronger nation-states, and the gradual replacement of religious authority with secular sovereignty. But the interesting question isn’t just that the Church lost. It’s what replaced it as the constraining force on state power. The answer, as Albert O. Hirschman showed in The Passions and the Interests, was commerce.

Hirschman reconstructed the intellectual climate of the seventeenth and eighteenth centuries and showed that before anyone argued capitalism was efficient, political thinkers argued it was calming. The pursuit of material self-interest, they believed, would tame the destructive passions of princes: the lust for glory, the appetite for conquest, and the capricious exercise of power. Montesquieu claimed that commerce makes manners gentle (le doux commerce). James Steuart argued that economic complexity would constrain rulers more effectively than any constitution, because a prince who disrupted the delicate mechanisms of trade would impoverish his own kingdom.

The argument was not that markets were good. It was that they were better than the alternative. Avarice, previously condemned by Augustine as one of the three principal sins, was rehabilitated as the least dangerous of the passions. As Keynes would put it more than a century later (unaware, as Hirschman noted with some irony, that he was repeating arguments already tested against reality): “It is better that a man should tyrannize over his bank balance than over his fellow-citizens.”

This worked, for a while. Markets did discipline states. The bond market still constrains government spending in ways that no opposition party can match, and the threat of capital flight still shapes policy in every country that participates in the global economy. The worldwide economic consequences of the Iran war are a rolling – or should I say roiling? – case study in just how powerful this discipline will be.

But overall, we ended up with something that neither Montesquieu nor Adam Smith anticipated. The self interest of merchants did not tame the passions but became infused with them. They turned the greatest of merchants into princes with all the undisciplined passions of old. And in doing so, they created their own form of tyranny, one where relentless optimization for shareholder value has been decoupled from the operating economy of goods, services, and human welfare. We now have the worst of both worlds: the passions not only of the leaders of the state but of major market participants, unconstrained by the old norms, combined with capital markets that optimize for extraction rather than broadly shared prosperity.

Tocqueville foresaw the populist elements of this bitter stew. He worried that people so focused on the pursuit of their private interests would willingly surrender their political freedom to any ruler who promised to protect those interests. They “think they follow the doctrine of interest, but they have only a crude idea of what it is, and, to watch the better over what they call their business, they neglect the principal part of it, which is to remain their own masters.”

Machines, Bureaucracies, and Markets

I had a brief conversation with Danzig after the dinner. He has been approaching the same territory from a different direction. In his 2022 paper “Machines, Bureaucracies, and Markets as Artificial Intelligences”, he argues that the dominant frame for thinking about AI, comparing machine intelligence to individual human intelligence, is too narrow. Machines, bureaucracies, and markets all belong to the same family of artificial intelligences, all invented to process information at speeds and volumes that surpass individual human capability. I’ve written about this same idea, but without the focus and deep sense of the functioning of the state that Danzig brings to the discussion.

He notes that machines, bureaucracies, and markets all are reductionist. They strip complex reality down to narrow inputs (bits, entries on bureaucratic forms, prices). All three detect patterns without understanding causation. Markets arrive at a price without knowing why. Bureaucracies apply rules without judging their rationality. Deep learning fits functions to data. All three are, in Danzig’s formulation, “remarkably smart and remarkably stupid, remarkably powerful and remarkably vulnerable.”

And all three were defended, at the time of their introduction, as neutral, value-free mechanisms. The history that followed tells a different story. Values turned out to be embedded in the systems all along, and as failures accumulated and those values were revealed and challenged, regulation followed. The 2008 financial crisis, driven by the opacity of mortgage-backed derivatives, is only the most recent example of a pattern that has repeated since the creation of the first stock exchanges.

Danzig’s placement of machine intelligence in the same framework as markets and bureaucracies lets us draw lessons for the future from the history of these earlier systems of large-scale governance. The most important of those lessons may be that the near-term concerns about machine intelligence won’t be about the machines themselves, but about how bureaucracies and markets use machine intelligence, and how society regulates the three together. The three form an ecosystem, not a hierarchy. (This idea echoes Bill Janeway’s notion of the relationship of markets, the state, and financial capital as “a three-player game.” I look forward to finding an occasion to discuss with Richard and Bill the extent to which financial capital is distinct from the broader market of trade in goods and services.)

Danzig notes that controlling intelligent machines will require not the one-time certification we use for industrial equipment, but the continuous supervision we use for personnel: probation, audit, promotion to wider responsibility, removal. The analogy isn’t testing a machine. It’s closer to managing a subordinate who is constantly learning and whose behavior in one situation doesn’t always predict their behavior in the next. And the control mechanisms for markets and bureaucracies co-evolved with those systems over centuries. The speed of machine intelligence may not afford that kind of time.

So Who Will Discipline Whom?

Capital will almost certainly adopt AI faster than the state. This is already visible. Three-quarters of American stock trading is now self-initiated by machines. Algorithmic systems set prices, allocate credit, and route information at speeds no human regulator can match. If capital already disciplines the state, AI-augmented capital will do so even more powerfully.

But that’s not all. In response to a draft of this piece, Bill made some important observations about the decline of state capacity, especially in the US.

There are “two overlapping systems for allocating resources and distributing their returns: markets and political processes. There is an inescapable tension: those who dominate in one sphere will attempt to carry their power over to the other; those who lose in one sphere will invoke or appeal to the other for retribution….

Contrary to what one might have expected or hoped for in 2008, we seem to be witnessing the delegitimization of state authority and the dissolution of state capacity. At the same time and in large part as the continuing afterglow of the Neoliberal Order, the opportunity for concentration of market power has been unconstrained by any countervailing force….

Generative AI is now emerging in an environment where constraining its deployment in the US would require a deep and long-term commitment to rebuilding state capacity, which in turn would require earning back some degree of legitimacy and authority. In the absence of these, conflict with European and Chinese regulatory initiatives and institutions will likely intensify.”

So that adds up to a strong argument that “markets discipline the state,” at least in the US. The contrast to China and Europe, where the state disciplines markets, and by extension, AI, may lead to strikingly different outcomes.

But there’s another possibility, especially over the long term. As AI agents begin to operate autonomously, negotiating with other agents, executing transactions, making decisions that shape markets, they may develop a logic that serves neither capital nor the state in its current form.

There are a number of fascinating experiments in AI as an alternate form of societal governance. The Habermas Machine, a project from Google DeepMind, trained a large language model to mediate democratic deliberation and found that AI mediators produced statements that generated wider agreement and left groups less divided than human mediators did. Projects like Audrey Tang’s work with Polis in Taiwan, as well as the broader movement to reinvent democratic participation using technology, point toward AI as infrastructure for a kind of collective intelligence that neither markets nor bureaucracies can achieve alone.

And there’s something also in Anthropic’s Constitution for Claude, an attempt to root its behavior on a moral foundation, not just in market incentives.

This is where the historical parallel with the Church is intriguing. The Church didn’t just discipline the state through threats. It provided an alternative framework for legitimacy, a different answer to the question of what society is for. Commerce also provided a different answer: prosperity and predictability. AI could provide yet another.

Who designs AI governance?

The difference between whether AI leads us to some promised land or to dystopia is not a question of what AI “wants.” AI has no wants. Neither do markets. Nor do states. It is a question of design: who builds the systems, what objective functions they optimize, what feedback loops they create, and what institutions govern them. An AI future dominated by Anthropic would clearly embed different values than one dominated by x.ai.

The real design problem is not how to supervise individual AI systems. It’s how to build the market structures, institutional governance, and feedback mechanisms that ensure that not just AI but markets and states serve broad human flourishing rather than narrow optimization.

As I’ve argued in We Have Already Let the Genie Out of the Bottle, the failures of corporate governance are a harbinger of our inability to govern even more powerful algorithmic systems. These companies are doing exactly what our financial markets tell them to do. Our attempts to rein them in will fail unless we change the objective function of our economic system.

The early thinkers who championed commerce as a civilizing force were making a bet that the self-interest of markets would be a more reliable constraint on power than the passions of individuals. That bet paid off, but it also created a new form of unfreedom, one that de Tocqueville recognized and that we are living through now. People became so absorbed in their economic interests that they lost the capacity for democratic governance. Adam Smith’s Theory of Moral Sentiments argued that the desire of people to appear good in the eyes of their neighbors would act as a check on self-interest. That was perhaps a reflection of the still powerful effect of the Church on society in those days. But since 1980, when Milton Friedman made the case that the moral obligation of corporations was simply to make money for their shareholders, that is, that “greed is good,” we have lost the last vestiges of the check that the moral order placed on both the passions of princes and the self-interest of the market.

Might the morality that AI derives from efforts like Claude’s Constitution as well as its absorption of millennia of human struggles to come to terms with “the good life” give us a start on a new kind of global order? If so, it will only be after a long struggle between AI, markets, and states.

Hirschman ended his book with a warning about intellectual amnesia: the tendency to trot out the same ideas that had been put forward in an earlier period “without any reference to the encounter they had already had with reality, an encounter that is seldom wholly satisfactory.” The argument that AI will rationalize and improve governance sounds remarkably like the argument that commerce would tame the passions of princes.

We could make the same mistake with AI. We could bet that the rationality of machine intelligence will be a more reliable constraint on both capital and the state than the messy, slow, easily corrupted mechanisms of democracy. That bet might even pay off in the short term. But if we aren’t careful about institutional design, we may find that we’ve traded one form of unfreedom for another.

We’re in for a helluva ride.

Thanks to Richard Danzig for inspiring this piece, and to Bill Janeway and Henry Farrell for helpful comments. All images were generated with ChatGPT 5.5.

There is a lot of hope that AI will advance the progress of science, but unfortunately, the collision between AI and scientific publishing has not gone well.

When an AI coding agent writes code, it operates within a rich ecosystem of version control, pull requests, code review, CI/CD pipelines, dependency management, and package registries. Github wasn’t designed for AI, but it turned out to be foundational infrastructure that makes AI-assisted software development work.

Science has an equivalent set of infrastructure for handling identity, provenance, integrity, and discoverability. Systems like arXiv, DOIs, CrossRef, Datacite, ORCID, OpenAlex, ROR, Retraction Watch, and PubMed form a kind of collaborative exoskeleton for scientific publishing and by extension, for modern scientific knowledge. Much as Github has been adapted for AI development, this infrastructure needs to be adapted for AI use in science.

The problems fall into several categories:

Hallucinated citations. When AI generates or assists with scientific papers, it routinely fabricates references. A multi-model study found that only about a quarter of AI-generated citations were entirely correct. Roughly 40% were erroneous or fabricated. Hallucinated citations have been found in papers accepted at NeurIPS and ICLR, the top AI conferences. GPTZero’s investigation found that about 2% of papers accepted at NeurIPS 2025 contained at least one fabricated reference. The peer reviewers missed them all. AI researchers, who understand hallucinations better than anyone, fell victim because convenience trumped verification.

Retracted paper propagation. AI tools are citing retracted papers without flagging them. Retraction Watch co-founder Ivan Oransky has noted that building a comprehensive retraction database is resource-intensive. Yet AI tools that claim to support scientific research are not even integrating the databases that already exist. A study of 21 chatbots found that on average, they correctly identified fewer than half of retracted papers when asked, and they produced substantial false positives as well. MIT Technology Review reported that AI chatbots are relying on material from retracted papers to answer questions, with some tools returning retracted articles with no retraction notice at all.

Training on compromised literature. AI models trained on scientific corpora inevitably absorb retracted, fraudulent, and paper-mill-generated content. Between 2024 and 2025, the retraction crisis accelerated dramatically. A recent bibliometric analysis found that AI-driven retractions have shifted from sporadic anomalies to a systemic crisis, with generative tools enabling paper mills to penetrate the highest levels of scholarly indexing. AI doesn’t know the difference between a landmark paper and a paper-mill product. Without integration with retraction databases and quality signals, this pollution propagates.

Generation of “AI slop” papers. “Paper mills were already a problem, but AI has made the problem far worse. In a world of “publish or perish,” scholars have strong incentives to generate poor quality papers, cite their own work excessively, and otherwise introduce noise into the system.

As the MIT VRAIX project puts it, because large language models are nondeterministic, “the same prompt can produce different answers, each delivered with fluency and confidence. These systems routinely present statements without verifiable sources, cite fabricated or incorrect references, blur the line between summarization and invention, and favor what’s statistically popular over what’s trustworthy. Even when real citations are included, users often have no easy way to determine whether those references are relevant, reliable, or even supportive of the claim being made.”

Tools to address these problems largely already exist, but they haven’t been integrated into AI systems. New tools are also being developed. As the AI Labs turn their attention to AI for science, they should also be exploring what the future infrastructure of scientific knowledge sharing might look like. That is the subject of this article.

The infrastructure of collaboration

DOIs and CrossRef. Every legitimate scholarly work has (or should have) a DOI, a persistent digital identifier maintained by CrossRef. CrossRef’s REST API lets you resolve a DOI and verify that a paper actually exists, with the correct title, authors, journal, and year. This is the most basic hallucination check imaginable, and yet most AI systems don’t perform it. Why isn’t this kind of validation built into every AI system that touches scientific literature? DOIs are not a panacea. They have been hacked both for fun and profit. As Geoffrey Bilder, the former director of technology for Crossref noted, there are DOIs that point to a South Park movie, a fake article on “a Google based alien detector,” and more. Alone, they guarantee nothing. They are just an identifier. But as part of an infrastructure that validates them, they are profoundly useful.

ORCID. ORCID provides a persistent identifier for researchers, linking them to their publications, affiliations, funding, and peer review activity. It’s an OAuth 2.0 API. You can authenticate a researcher’s identity and pull their verified publication list in seconds. If an AI-generated paper claims Dr. Smith at MIT published a paper on quantum computing in Nature, you can check ORCID to see whether Dr. Smith exists, whether they’re affiliated with MIT, and whether that paper is in their record. This is researcher identity verification, and it’s available as a free API as well as through periodic open data snapshots. As The Scholarly Kitchen noted, ORCID works best in combination with other persistent identifiers. Portugal’s integration of ORCID with its national research identifier CIÊNCIA ID has connected 112,000 researcher profiles and saves more than 154 hours per researcher annually in data entry. That’s the kind of compounding return you get from well-designed infrastructure.

OpenAlex. The successor to Microsoft Academic Graph, OpenAlex is now a fully open scholarly knowledge graph with over 271 million indexed works, serving over 1.5 billion monthly API calls. It knits together data from CrossRef, PubMed, ORCID, institutional repositories, and DataCite. Its API is free and returns rich metadata including citation networks, author affiliations, and open access status. OpenAlex recently received a $3.5 million Wellcome grant to integrate global research funding metadata, making it possible to trace the chain from funder to grant to publication to impact. The Walden rewrite, launched in late 2025, added 190 million new works including datasets and software from DataCite and thousands of institutional repositories.

Retraction Watch and the Retraction Watch Database. Retraction Watch is the closest thing we have to a comprehensive record of scientific papers that have been withdrawn due to fraud, error, or ethical violations. It’s a project of The Center for Scientific Integrity. Numerous companies and nonprofits including Zotero and Web of Science have integrated the Retraction Watch database, automatically excluding retracted publications from their research assistants. Some AI-specific tools like Consensus have also started incorporating retraction data from a combination of sources including Retraction Watch, but this should be table stakes for any AI system that claims to work with scientific literature.

arXiv. The preprint server for physics, mathematics, computer science, and related fields has been operating since 1991. It provides a structured, persistent, openly accessible record of scientific work. arXiv IDs are resolvable. The metadata is machine-readable. For AI systems working in these domains, arXiv is an authoritative source that can be queried to verify claims.

Consider the parallels to software development. GitHub gives software a persistent identity for every commit. DOIs give scholarly works a persistent identity. GitHub tracks who contributed what. ORCID does the same for researchers, disambiguating people with common names and linking them to their full body of work across institutions and careers. GitHub has dependency graphs. CrossRef, Datacite, and OpenAlex maintain citation graphs, linking 271 million scholarly works to their authors, institutions, and funders. GitHub has issue trackers and code review. The scientific community has peer review, post-publication commentary on PubPeer, and Retraction Watch tracking papers that have been withdrawn. Github and Gitlab even support software citation through .cff files, which includes the ability to assign DOIs, so the two systems have meaningful overlap.

MIT’s VRAIX project is working to bring this infrastructure together and adapt it for AI. It attempts to address the problems described at the opening of this article not by looking inside scientific papers for the common tells of AI generation, but by situating papers and LLM-generated content within what we might call “the web of knowledge.” As its creators put it, “VRAIX’s core question is: ‘What system of knowledge does this claim belong to, and does it behave in a way consistent with that system?’” It looks at the citation graph, resolves citations to standard identifiers (DOIs, PMIDs, ORCIDs, ROR IDs), resolves them to real metadata, including the network of co-authors and institutions, the history of corrections and retractions, historical publication patterns and the relevance of cited sources to the claims they are said to support.

Geoffrey Bilder, formerly the director of research for Crossref, pointed out to me that this infrastructure has been adopted by fierce rivals in the publishing industry. The key is the structure and governance of these infrastructure organizations. They are open standards, and follow a set of principles (POSI) that help ensure that they cannot be captured or enshittified. This can serve as a reassurance not only to scientific publishers but also to AI companies that they are not building dependencies on things that might be bought or captured by their rivals.

The mechanism design problem

To borrow Danny Ryan’s definition of protocols from the Ethereum Foundation’s Summer of Protocols project, scientific publishing represents “strata of codified behavior” that enable coordination across the entire research enterprise. They are the “civilizational infrastructure” of science. And like all good infrastructure, they’ve become invisible. Researchers don’t think about DOIs the way drivers don’t think about lane markings. But remove them, and the system falls apart.

They are a public good. And AI companies are mostly ignoring them, or worse, undermining them.

Right now, the relationship between AI and scientific infrastructure is almost entirely extractive. AI companies train on scientific papers. They build products that generate and manipulate scientific text. They compete for the “AI for science” market. But they contribute almost nothing back to the infrastructure that makes scientific knowledge reliable in the first place.

This is entirely consistent with the broader argument I’ve been making about the agentic economy as envisioned by the AI labs. Value flows in one direction. AI companies consume scientific content, but they don’t contribute anything back.

Think about the YouTube Content ID analogy I described in “The Missing Mechanisms of the Agentic Economy.” The music industry’s first response to unauthorized use of their music was “Take it down.” YouTube’s answer was “How about we help you monetize it instead?” That aligned incentives and created a vibrant creator economy.

The same thinking should apply here. The question isn’t just “How can AI companies use scientific infrastructure to make their products better?” (though they should). The question should also be “How can AI companies help these services become more valuable, more sustainable, and more comprehensive?”

What building on, not just extracting from, would look like

Here are some concrete possibilities.

Validation as a first-class feature. Every AI system that generates or edits scientific text should validate references against CrossRef, OpenAlex, and Retraction Watch as part of its core pipeline, not as an afterthought. This should be as automatic as a compiler checking syntax. The APIs exist. The latency is minimal.

ORCID integration for attribution. When AI systems summarize or synthesize scientific literature, they should link to ORCID profiles, not just paper titles. This creates a direct connection between AI-generated output and the human researchers whose work it draws on. It also makes it easy to verify that a cited researcher actually wrote what the AI claims they wrote.

Contributing to metadata quality. AI is very good at extracting structured information from unstructured text. OpenAlex reports that over 60% of its records lack complete institutional affiliation data. Over 40% lack abstracts. AI tools that process scientific papers could contribute extracted metadata back to OpenAlex, improving the graph for everyone. This is the kind of “architecture of participation“ that made open source work. The system gets better the more people use it.

Retraction monitoring as an MCP service. Imagine a Retraction Watch MCP server that any AI agent could query in real time. Before citing a paper, the agent checks whether it’s been retracted, whether it has expressions of concern, whether its citations have been flagged. This is the kind of service that would benefit the entire ecosystem, and it could be funded in a way that sustains Retraction Watch’s work. The MCP registry protocol and MCP Server Cards I discussed in “The Missing Mechanisms” could provide the discovery and authentication layers. It’s also worth integrating PubPeer, the post-publication review and comment system, and the Problematic Paper Screener.

Funding the infrastructure. OpenAlex operates on a shoestring, with institutional memberships at $5,000/year. Retraction Watch is a nonprofit that depends on donations. ORCID is sustained by member organizations. These are the foundations on which the credibility of AI-generated science depends, and they are chronically underfunded. AI companies generating billions in revenue from products that depend on scientific credibility should be contributing to the infrastructure that provides it. This is not philanthropy. It’s enlightened self-interest.

Provenance chains for AI-generated scientific content. When AI contributes to a scientific paper, that contribution should be traceable, not just disclosed in a boilerplate statement, but linked to specific claims, specific sources, and specific verification steps. The persistent identifier infrastructure (DOIs, ORCID, OpenAlex IDs) already provides the building blocks for this. What’s missing is the protocol that ties them together.

From extraction to engineered argument

In “The Missing Mechanisms,” I argued that the best market-shaping protocols are “engineered arguments, not engineered agreements.” They don’t impose a single solution from above. They create a framework within which competing approaches can contend.

The same principle applies here. AI companies don’t need to adopt a single standard for scientific verification. I’m arguing that they should build on the existing infrastructure in ways that let the market discover what works. Some will integrate CrossRef validation. Others will build on OpenAlex’s knowledge graph. Some will develop novel quality signals we haven’t imagined yet. The point is to participate in the ecosystem rather than treating it as a resource to be mined.

The scientific infrastructure community has spent decades building what David Lang, in his essay “Standards Make the World” for the Summer of Protocols project, called a “third pillar of modern society,” alongside private organizations and public institutions. These are standards and systems that enable coordination without central control.

AI companies that build on this infrastructure will make better products. They’ll produce more reliable scientific output. They’ll face fewer hallucination crises and retraction embarrassments. But more than that, they’ll be investing in the civilizational infrastructure that makes reliable knowledge possible in the first place. They will be taking their place alongside many other commercial entities like Digital Science, Elsevier, and Clarivate that already build on this infrastructure, as do many non-commercial tools that researchers depend on every day, like Zotero.

GitHub didn’t just give developers a place to store code. It became a collaborative exoskeleton that made an entire style of distributed, cooperative development possible. The scientific infrastructure stack has the potential to do the same for AI-assisted science. But only if AI companies stop treating it as someone else’s problem and start treating it as a foundation to build on, and a foundation on which their own success in science depends.

What’s missing is the will to build on them, and the mechanism design thinking to ensure that everyone, not just the AI companies, benefits from the result.

Thanks to Geoffrey Bilder, Ivan Oransky, and Ilan Strauss for comments on drafts of this article. Geoffrey and Ivan know far more about this topic than I do, and this article draws on their work. I get credit (or rather, demerit) for any errors that remain. Images created with GPT 5.5 medium.

Foreword: Thinking about AI market design

The piece below by Jesus Rodriguez makes a compelling case that agents are emerging as a new class of economic actor: a new delegated consumer (or shopper) with native needs around identity, payments, memory, and commerce. We share his view that the internet was built for humans and now must be partly rebuilt for machines.

MECHANISMS. We want to add one note of caution about how that rebuilding happens. What Rodriguez describes is, in our language, the “missing mechanisms” problem applied to agents. Tim has argued that markets need mechanisms for distributed value creation and participation. Mechanisms are engineered rules that incentivize self-interested actors to produce honest disclosure and can aim for a market that benefits all actors, rather than a single firm. A mechanism’s rules decide outcomes and payments for things like auctions, voting (no payment), public goods spaces, and more. They can make markets work where otherwise they would not due to agents behaving strategically by hiding information and not wanting to participate (“why license data when you can scrape it”, for example).

The agent economy is now confronting its own version of this missing markets question (as Google’s Search and Amazon’s marketplace was once did too): but it’s missing mechanisms for agents, not for humans. Delegation, attestation, payment scopes, machine-readable provenance, agent identity, memory boundaries should be internet and AI standards first, that underpin agentic markets, and then innovative competitive product features second. The two can and should exist as compatible conceptualizations of how to build markets online as open publicly beneficial spaces.

The layers for agentic AI markets Rodriguez describes below will be built. The question remains how to realize them as open, modular infrastructure, rather than as proprietary chokepoints?

The defaults are still soft and the market structures are still fluid. They will not stay that way for long.

— Tim O’Reilly & Ilan Strauss

Introduction: Agentic markets

For the first few decades of the internet, software had a reasonably stable assumption baked into it: the user was a human.

A human had eyes, a browser, a mouse, a password manager, a credit card, an email address, a tolerance for modal dialogs, and a finite amount of patience. The entire SaaS and consumer internet stack grew around this shape of user. Search engines ranked pages for humans. E-commerce sites optimized funnels for humans. CRMs tracked human sales reps selling to human buyers. Identity systems authenticated humans. Analytics systems measured human clicks, human sessions, human conversions.

Then we started building AI agents

At first, agents looked like toys. A chatbot that could book a calendar invite. A coding assistant that could open a pull request. A browser agent that could navigate a website on your behalf, slowly and somewhat comically. But underneath the awkwardness was something important: software was no longer being used only by people. Software was being used by software that could reason, plan, transact, remember, delegate, and retry.

This is a much bigger shift than “chatbots are the new UI.” It is closer to the emergence of a new economic actor. Not a legal person, not a human replacement, and not magic. But a new kind of consumer.

Agents consume information. Agents consume APIs. Agents consume storage, compute, tools, workflows, credentials, payments, and services. Agents evaluate options, make requests, negotiate constraints, and execute tasks. They are not just another interface to existing software. They are a new demand surface.

Once you see agents as consumers, a lot of the software landscape starts to look strangely misaligned.

The Salesforce for agents may not be Salesforce. The Google for agents may not be Google. The Amazon for agents may not be Amazon. Not because these companies are weak, but because the primitives change when the consumer changes.

A human using Salesforce wants dashboards, contacts, pipeline stages, notes, reminders, and reports. An agent using a CRM wants something else: structured state, task objectives, relationship graphs, permissioned memory, machine-readable sales playbooks, and reliable APIs for updating intent. It does not need a beautiful dashboard. It needs a canonical substrate of customer truth that it can query, mutate, and reason over safely.

A human using Google wants ranked blue links, snippets, maps, videos, and maybe an answer box. An agent using search wants provenance, freshness, confidence intervals, citations, entity resolution, schema alignment, and retrieval interfaces that are stable under automation. It does not want ten links. It wants to know which source is authoritative for a given claim, what changed since the last run, and whether the answer is safe to act on.

A human shopping online wants product photos, reviews, discounts, shipping estimates, and a checkout page. An agent shopping on behalf of a human wants product metadata, compatibility constraints, return policies, vendor reliability scores, payment authorization scopes, fraud guarantees, and receipts that can be parsed by another machine. It does not browse. It solves an optimization problem under constraints.

This is why “just give agents browser access” feels both exciting and obviously temporary. Browser access is backwards compatibility. It is useful in the same way screen scraping was useful before APIs. But if agents become a major class of software user, we should expect agent-native infrastructure to emerge.

The web was designed around human attention. The agent economy will be designed around machine action.

Consider identity

Today, identity on the internet is mostly a ceremony for proving that a human is allowed to do something. We use passwords, passkeys, OAuth, SSO, device fingerprints, CAPTCHAs, and risk engines. These systems assume a human principal somewhere in the loop.

Agents break this model in subtle ways. An agent is not exactly the user. It is also not exactly an employee, a bot, a service account, or an OAuth app. It may act for a person, but only within a certain scope. It may need to buy groceries but not alcohol, schedule meetings but not cancel medical appointments, email customers but not change pricing, deploy code but not rotate production secrets.

So agent identity needs delegation as a first-class primitive. Not “here is my password, go do stuff.” Not “here is a broad API token with terrifying permissions.” Instead: here is an agent, acting on behalf of this principal, for this objective, during this time window, with this budget, using these tools, under these audit requirements.

This implies infrastructure: agent passports, delegated credentials, capability-based access control, revocation systems, policy engines, and audit logs that explain not just what happened, but why the agent believed it was allowed to act.

The “why” matters. A normal API log might say: POST /refunds succeeded. An agent-native audit log should say: the agent issued a refund because the customer met policy condition X, the order was delayed by Y days, the user had previously authorized refunds below Z dollars, and no higher-risk flags were present. This is not just observability. It is accountability.

Now consider payments

Human payments are optimized around cards, wallets, bank accounts, checkout pages, fraud prevention, and dispute resolution. The system is designed for a human to consciously approve a transaction, even if that approval has become increasingly compressed into one click or one biometric scan.

Agents introduce a different payment pattern: constrained autonomy.

A travel agent might have permission to spend up to $1,500 on flights, but only if the arrival is before 6 p.m., the layover is under two hours, the airline is not on a blocked list, and the fare is refundable. A procurement agent might be allowed to buy cloud credits, office equipment, or data enrichment services within a monthly departmental budget. A coding agent might pay for API calls, benchmark runs, or temporary compute to complete a task.

This requires payment rails that are more programmable than consumer checkout and less heavyweight than enterprise procurement. We will need spending envelopes, machine-readable invoices, pre-authorized budgets, reversible transactions, agent reputation, merchant attestations, and real-time policy checks.

Share

A credit card number is a poor interface for an autonomous actor. It is a bearer token with too much authority and too little context. Agent payments will likely look more like signed intents: “Agent A, acting for User B, is authorized to pay Merchant C up to Amount D for Purpose E, subject to Policy F.” The merchant should be able to verify this. The user should be able to revoke it. The system should be able to dispute it. The agent should be able to reason about it before acting.

E-commerce changes too

Most online stores are persuasion machines. They are designed to convert human uncertainty into purchase behavior. Recommendation modules, scarcity banners, coupon wheels, bundles, sponsored placements, and dark patterns all exist because humans are impressionable, visual, and busy.

Agents are busy too, but in a different way. They are not emotionally moved by a hero image of a backpack on a mountain. They may be vulnerable to other things: poisoned reviews, manipulated metadata, adversarial product descriptions, fake compatibility claims, hidden fees, or vendors that optimize for agent-ranking algorithms.

So agent commerce will need a different trust layer. Product pages become less important than product specs. Reviews become less useful unless they are structured, verified, and resistant to manipulation. Return policies need to be machine-readable. Availability needs to be queryable. Bundles need explicit semantics. “Best laptop for me” becomes a negotiation among constraints: workload, budget, portability, battery, repairability, delivery time, warranty, vendor trust, and user preference memory.

The equivalent of SEO for agents will be fascinating and horrible. Today, companies optimize pages to rank in human search. Tomorrow, they will optimize structured representations to be selected by buyer agents. This will produce new spam, new fraud, new ranking games, and new defensive infrastructure. We should assume every agent-readable marketplace becomes adversarial once enough money flows through it.

Data storage also changes

Human software treats data as something to display, search, and report. Agent software treats data as working memory. Agents need durable context across tasks: user preferences, prior decisions, organizational policies, project state, personal constraints, and learned patterns. But they also need memory boundaries. Remembering everything is not intelligence. It is liability.

An agent-native storage layer needs multiple kinds of memory. There is episodic memory: what happened during the last task. Semantic memory: stable facts about the user or organization. Procedural memory: how to perform recurring workflows. Policy memory: what is allowed. Source memory: where a belief came from. And scratch memory: temporary reasoning artifacts that should probably expire.

This storage layer must support retrieval, provenance, deletion, summarization, permissioning, and conflict resolution. It must know that “the user prefers morning flights” is not the same kind of fact as “the user authorized a $700 purchase yesterday.” It must distinguish stale preferences from hard constraints. It must expose memory to agents in ways that reduce hallucination rather than amplify it.

In the human internet, data was often collected because it might become useful later. In the agent internet, data becomes directly operational. If an agent remembers incorrectly, it acts incorrectly. Memory bugs become action bugs.

This brings us to software design itself

A surprising amount of current software is hostile to agents because it was built for humans. Buttons without semantic labels. PDFs instead of structured documents. Emails that require interpretation. Dashboards with charts but no underlying query API. Terms of service written only for lawyers. Error messages that explain nothing. Workflows that require a human to click through five screens because the product team wanted engagement.

Agents expose the hidden tax of unstructured software.

Agent-friendly software will have different virtues. It will be explicit, inspectable, composable, and transactional. It will provide APIs not as an afterthought, but as the main surface. It will describe capabilities, costs, side effects, and failure modes. It will support dry runs. It will make state changes reversible where possible. It will provide good error messages because the error message is not for a frustrated human anymore; it is for another system deciding what to do next.

The best agent software may look boring to humans. A great agent-native CRM may be mostly a graph, a policy engine, an event log, and a set of tool contracts. A great agent-native search engine may look like a retrieval and verification API. A great agent-native bank may look like programmable authorization plus risk scoring. A great agent-native e-commerce platform may look like a giant constraint solver with trust primitives.

This does not mean human interfaces disappear. Humans still set goals, approve high-risk actions, inspect outcomes, and change their minds. But the center of gravity moves. The human becomes more like a principal, supervisor, or exception handler. The agent becomes the high-frequency user.

This distinction matters because incumbents often win interface shifts but lose primitive shifts.

When mobile arrived, many desktop companies successfully made mobile apps. But the biggest mobile-native companies were not just desktop products on smaller screens. They were built around new primitives: GPS, camera, push notifications, contacts, sensors, app stores, and always-on connectivity.

Agents are similar. Adding a chatbot to an old product is not the same as rebuilding the product around agentic use. The winners may be companies that understand the new primitives early: delegation, memory, tool use, verification, policy, agent identity, machine-readable commerce, and autonomous payments.

The phrase “agents are a new consumer” sounds metaphorical, but it is actually quite literal. Consumers create demand. Demand reshapes markets. Markets reward infrastructure that serves the consumer’s native behavior.

Humans click. Agents call tools.

Humans browse. Agents query.

Humans forget. Agents persist memory.

Humans approve. Agents need delegated authority.

Humans compare visually. Agents optimize over structured constraints.

Humans tolerate ambiguity. Agents need executable semantics.

Humans can be persuaded. Agents can be attacked.

Once agents become common, every software company will face a basic question: is your product merely usable by agents, or is it built for them?

The difference will matter. A website that an agent can painfully navigate is not agent-native. An API that exposes a few endpoints is not agent-native. A chatbot bolted onto a dashboard is not agent-native. Agent-native software treats autonomous actors as first-class users with identity, permissions, memory, budgets, goals, and accountability.

We are early enough that much of this still looks speculative. The agents are unreliable. The standards are immature. The demos break. The economics are unclear. But this is often what new consumers look like at the beginning. Mobile apps were once toys. Cloud workloads were once experiments. Online payments were once scary. Developer APIs were once niche.

Then, slowly, the toy becomes a workflow. The workflow becomes a habit. The habit becomes infrastructure. And the infrastructure becomes invisible.

The internet was built for humans, then patched for machines. The next internet may be built for machines, with humans in command.

That is the opportunity: not just to build agents, but to build the world they will consume.

Visit AI Disclosures Project for more information on our work.

For the past two years, I’ve been working with economist Ilan Strauss at the AI Disclosures Project (joined soon after by computer scientist Sruly Rosenblat). We started out by asking what regulators would need to know to ensure the safety of AI products that touch hundreds of millions of people. We are now exploring the missing mechanisms that are needed to enable the agentic economy.

This essay traces our path from disclosures through protocols to markets and mechanism design (read more on our new website: here). Rather than simply stating our conclusions, I’m sharing our thought process and some of the conversations and historical examples that have shaped it.

We will be holding a number of focused convenings to explore these ideas over the next couple of months, and my hope is that shared context will enable more productive engagement with what is very much a work in progress.

The disclosure problem

Ilan Strauss and I started the AI Disclosures Project in early 2024 with a conviction that most regulators had little idea how AI worked or where it was going. The field was so young that many of the early regulatory proposals were misguided. We thought that regulators and industry should start by agreeing on standards for disclosure, so that we could all learn together as the technology develops. You can’t regulate what you don’t understand.

One of our first insights was that focusing solely on model safety was a mistake, much as if regulators inspected automobiles at the factory but completely ignored their use on the roads. We believed (and still do) that the focus should be on AI as deployed. And we believe that disclosures shouldn’t focus just on capabilities but on business models and the operating metrics that AI companies use to shape how their products operate.

Ilan and I had worked together previously with Mariana Mazzucato at University College London on what we called “algorithmic attention rents,” studying how platforms like Amazon and Google control user attention to extract economic rents from their suppliers. We observed that organic search at Google and Amazon was a huge advance in market coordination, using hundreds of signals to find the best match for a user’s intent. In effect, both companies had built a better “invisible hand.” And yet after decades of success, they turned away from that advance. To use Cory Doctorow’s coinage, they began “enshittifying” their services by substituting inferior paid results for the top organic search results in order to pad their bottom line.

We’d also watched social media start out with the promise of keeping you in touch with your friends and foster productive conversations, but then instead began to optimize for engagement at the expense of everything else. By the time anyone understood what was happening, the damage had been done. We can see the inflection point in their financial metrics, but neither regulators nor the public can see the changes in operating metrics that drove the financials. What if we could capture what good looks like before it gets enshittified, and identify how that changes over time?

We also observed that modern technology companies are completely different from industrial era corporations, where you can understand key elements of the business by tracing the inputs and the outputs through the financial statements. Instead, the business is largely driven by intangibles, which are lumped into one impenetrable black box.

We wanted to learn from that mistake. While the horse was already out of the barn on search and social media, we hoped to get disclosure of operating metrics into AI governance while there was still an appetite for regulation. Unfortunately, that window was very short. The failure turned out to be productive, though, because it forced us to think harder about regulation more broadly and what other leverage points might be found.

Protocols as functional disclosures

The first turn in our thinking came when we realized that disclosures aren’t just informational. The most important disclosures are functional. We came to see the parallels between disclosures and communications protocols, the agreed-on methods by which networked systems share information. For example, the HTTP protocol that underlies the World Wide Web specifies how a web browser and web server communicate in order to display a web page.

This is a structured communication with rules that must be followed and data that must be exchanged in a particular order. An HTTP request that identifies the user agent as a command line program such as curl rather than a graphical browser such as Chrome triggers a different response from the server. The user-agent string isn’t a report filed with a regulator. It’s an operational signal embedded in the protocol, and it carries a lot of information.

Once you see protocols as a system of functional disclosures, you start noticing that every regulatory system has a kind of communications and control protocol at its heart. Generally Accepted Accounting Principles (GAAP) or IFRS, the European equivalent, are protocols for communication between companies and their accountants, auditors, banks, investors, and tax authorities. Even road markings and road signs are a communications protocol, giving information to drivers about local conditions, laws, and the proper use of the road. These are slow, analog protocols, but they are protocols nonetheless.

Protocols can be inspected. Observability is the key to governance. Police observe speeders on the road; credit card processors and banks watch for credit card fraud on their payment networks; email processors filter spam as it passes through nodes on the network. The observability points for AI are still emerging, but that’s where regulators should be focused.

Even beyond being a locus for observability and regulability, protocols themselves do an enormous amount of the governing work in modern technology systems. Spanning everything from how packets get from one place to another to what gets displayed, who has permission to see it, and sometimes even what it costs, they ultimately determine who can interoperate with whom. That led us to an even bigger realization.

Protocols shape markets

Think about the early shape of the AI chatbot market. It was a winner-takes-all race to be the dominant platform for AI in the way Windows became the platform for PCs, or iOS and Android for phones. Whoever wins controls the market. Then Anthropic introduced MCP, the Model Context Protocol. All of a sudden, the landscape looked more like a web. There could be many winners. It didn’t matter what model you were running or whose APIs you were calling as long as you followed the protocol. And as the agentic AI market unfolded, the protocol wasn’t just MCP. An AI agent could be a user of the existing internet protocol stacks. Whether MCP itself survives or is superseded by other protocols, the shape of the market was transformed.

This insight reframed our whole project. Protocols are not just technical infrastructure. They are market-shaping mechanisms.

Workflows are also protocols

I talked last week with some of the folks working on the Long Now Foundation’s partnership with Ethereum’s Summer of Protocols project (soon to be: Protocolized magazine), and that widened my lens even further.

When software people hear “protocol,” we think of communication protocols: TCP/IP, HTTP, MCP, or, say, Stripe’s Machine Payment Protocol (MPP).

To this particular protocol research community, a protocol is any standardized way of doing something. Wildfire management teams follow protocols. So do flood response teams, hospital emergency rooms, and air traffic controllers. Atul Gawande’s book The Checklist Manifesto was an attempt to establish a common protocol for surgical operating theaters. This is a very different definition of protocol, and yet putting the two meanings of the word into the same frame makes a new kind of sense.

In his introduction to the Summer of Protocols’ Protocol Reader, Venkatesh Rao cited Ethereum researcher Danny Ryan’s definition of a protocol as a “stratum of codified behavior” enabling coordination. He pointed out that protocols tend to become invisible once adopted. Rao calls this a “Whitehead advance,” after the philosopher Alfred North Whitehead’s observation that civilization advances by extending what we can do without thinking.

But he also made the thought-provoking point that a protocol is an “engineered argument,” in contrast with an API, which he says is an “engineered agreement” enforced by one dominant actor. There’s more to it than just the power asymmetry of enforced agreement, though. In a followup conversation, Venkatesh Rao noted that protocols are “not just codified modes of information exchange, but modes of live, structured, argumentation, often with an active computational element. For example, CSMA/CD (Ethernet) must detect packet collisions and compute and execute a random delay for retransmittal of packets. This is not mere structured communication. This is argumentation with what philosophers call dynamic semantics.”

Rao continued: “The moment you go beyond computing protocols, real-world feedback loops from material consequences become really important. For example, container-shipping is quite close architecturally to TCP/IP (the big difference being that packets can be dropped and retransmitted while lost containers are actually lost), but because it has a materially embodied feedback loop, regulatory mechanisms start to behave more like control systems than communication systems.”

I love the idea of protocols as an engineered argument. The dynamism this suggests is going to be ever more true in a future of agentic protocols. But this notion also triggered another thought, which is that markets are also engineered arguments. My bridge to this reformulation was the difference between de jure protocols that arise from a formal standards process, and de facto protocols that arise through market contention.

In the early days of the internet, the Internet Engineering Task Force (IETF) was all about engineered arguments. People had ideas about how the internet ought to work, and to prove their point they had to show up with interoperable implementations. No one had the ability to enforce anything. Agreement had to evolve. As Dave Clark famously put it, “We reject: kings, presidents, and voting. We believe in: rough consensus and running code.” The de facto protocols of the internet that emerged from the IETF ended up significantly outperforming the competing de jure networking protocols that emerged from telecommunications standards bodies. The IETF framed the argument; whoever showed up made their case and won or lost by way of adoption.

It also made me remember another decades old story that I had lived through. Microsoft and Netscape were duking it out in the web server market and were building their own “engineered agreements” for what was up the stack from the base web server functionality. Everyone thought that Apache wasn’t keeping up, but they had a trump card. They provided an extension layer. And that engineered all kinds of productive arguments between a market of competing developers rather than a single engineered agreement imposed by either a dominant player OR a dominant committee.

Rao also noted that protocols spread slowly but become nearly impossible to dislodge once established. For example, SMTP (the protocol for email) dates back to 1982, and has outlasted many competitors. There is a lot of path dependence. And so getting the first steps right is an important part of engineering the argument.

And in his essay “Standards Make the World” for the Summer of Protocols project, David Lang makes the point that technical standards form a third pillar of modern society, alongside private organizations and public institutions. They aren’t the state and they aren’t the market, but they’re essential to both. When they work well, standards become enabling technologies. The internet. The shipping container. Standard time. They are civilizational infrastructure.

In short, we are not just building communication protocols for software agents. We are developing a new way to standardize the best practices and workflows that will shape the human + AI future, allowing humans and agents to cooperate across organizations, industries, and borders.

Share

Skills can also be seen as protocols

Once the Summer of Protocols team (Venkatesh Rao and Timber Schroff) planted in my mind the connection between workflows and protocols, it occurred to me that Agent Skills are also a “stratum of codified behavior,” and perhaps even a set of competing “engineered arguments” for how to do work with AI.

At the simplest level, a Skill is a piece of structured knowledge: here’s how to create a Word document; here’s how to extract the text from a PDF; here’s how to publish on the Hugging Face Hub. There can be many Skills that attempt to codify the same knowledge, but some may be better than others. As Skills multiply, how will we find the best ones? This is in many ways analogous to the organic web search problem, which Google solved by aggregating hundreds of useful signals.

And we’re seeing that there is a kind of hierarchy of skills. Jesse Vincent’s Superpowers framework, which has become one of the most widely adopted open source projects in AI-assisted development, doesn’t just give agents individual capabilities. It encodes an entire software development methodology: brainstorm before you build, plan before you code, test before you ship, review before you merge. That’s a standardized workflow. It’s a lot like the kinds of protocol that the Summer of Protocols folks were talking about with us and the Long Now Foundation, expressed in a form that agents can follow.

The existing protocols that the protocol research community talks about, like wildfire management protocols or hospital triage protocols, encode best practices into a repeatable, teachable process for human teams. They have yet to be adapted for agents. And in fact, many of them are never going to be entirely agentic. We will need to build mechanisms for workflows that include both AI agents and humans working together.

Agent skills in some (but not all) areas raise the same questions that industrial standards have always raised: who decides what the best practice is? How do you verify quality? How do you govern updates? We may be talking about skills that encode the workflow for regulatory compliance in a specific industry, or for conducting an environmental impact assessment, or for managing a clinical trial. Are the standards de jure or de facto, the result of an engineered agreement by a committee or an engineered argument that enables a vibrant market?

At O’Reilly, this is something we think about a lot. We’re a company built on codifying expert knowledge. We’ve published books and organized conferences and online training that taught people how to do new things. Now we’re asking “What does it look like to publish the skills that teach agents how to do things? And how do we make sure those skills are discoverable, trustworthy, and monetizable, not just for us but for every domain expert who has knowledge worth encoding?” And how do they emerge from contention in a vibrant market rather than by decree?

We believe we’ll all be better off with an engineered argument than an engineered agreement. And that brings me to mechanism design.

The missing mechanisms

Economists use the term “mechanism design” to describe the engineering of rules and incentive structures that lead self-interested actors to produce outcomes that are good for everyone. It’s sometimes called “reverse game theory.” Rather than analyzing the equilibria that emerge from a given set of rules, you start with the outcome you want and work backward to design the rules that will get you there.

Mechanism design theory got its start in the 1960s when Leonid Hurwicz took up the problem of how a planner can make good decisions when the information needed to make them is scattered among many different people, each of whom has their own interests. His key insight was that people won’t reliably reveal what they know unless it’s in their interest to do so. So how do you design a system that aligns their incentives?

The field that Hurwicz founded and that Eric Maskin and Roger Myerson developed through the 1970s and 80s earned all three the Nobel Prize in Economics in 2007.

I first encountered the field when Jonathan Hall, at the time the Chief Economist at Uber, waved Al Roth’s book Who Gets What — and Why at me and said “This is my Bible.” In it, Roth describes his own work on mechanism design, which won him the 2012 Nobel Prize in Economics along with Lloyd Shapley. Roth applied mechanism design to kidney matching markets, markets for college admissions, for law clerks and judges, and for hospitals and medical residents. When I first talked to Jonathan and then Al Roth, my layman’s takeaway about mechanism design was that it was simply the application of economic theory to design better markets.

And I’ve since come to think even more broadly about what mechanism design might mean in a technology context. In my broader framing, packet switching was a breakthrough in mechanism design. So for that matter was TCP/IP, the World Wide Web, and the protocol-centric architecture of Unix/Linux, which enabled open source and the distributed, cooperative software development environment we take for granted today. PageRank and the rest of Google’s organic search system also seems to me to be a kind of mechanism design. So do Pay Per Click advertising and the Google ad auction. All of them are ways of aligning incentives such that self-interested actors produce outcomes that are good for others as well.

So that brings me back to AI. Right now, there’s a problem that makes the AI/human knowledge market less efficient than it could be. The disrespect for IP that has been shown by the AI labs and applications during the training stage, and even now during inference, has led to efforts by content owners to protect their content from AI. Do not crawl. Lawsuits. Reluctance to share information. Even the AI labs are complaining about the theft of their IP and trying to protect their model weights from distillation.

It’s an economy crying out for mechanism design.

The lesson of YouTube Content ID is worth learning. Twenty-odd years ago, the music industry was in the same position that content creators are in today with AI. In response to unauthorized use of their music by creators, music publishers’ demand to YouTube was “Take it down.” But as Google engineer Doug Eck explained to me, YouTube came up with a better answer: “How about we help you monetize it instead?” I don’t know the details of how that decision was made but I do know the eventual outcome. Aligned incentives led to a vibrant creator economy in which YouTube’s video creators, the music companies, and Google all got to share in the value that was created.

That should give us inspiration for how to solve some of the problems we face now with AI. Whether it’s with Agent Skills, NotebookLM, or other emergent artifacts of the new AI/human knowledge economy, we need to align the incentives. If we can grow the pie, and in a way where no single gatekeeper captures the bulk of the benefit, there’s a way to create a vibrant market. But that requires building mechanisms that don’t exist yet.

What mechanisms are missing from the agentic economy? Here’s a partial list:

Skills markets. There’s an enormous economic opportunity for humans to create and trade skills that agents can use. These are not just simple aggregation of context with tool use instructions, but higher-level, industry-specific workflows that encode deep human expertise. At O’Reilly, we’re figuring out how to turn our knowledge and that of our authors into skills, how to make them discoverable, and how to sell them. But as of yet, there’s no way for a broader community of skill creators to participate.

Quality and governance for skills. Some skills will need the same kinds of governance that industrial standards have. Who certifies that a medical skills package follows current clinical guidelines? Who updates it when the guidelines change? We haven’t begun to build the institutions that would govern agent skills at that level.

Registries and discovery. The MCP community has been working on a registry protocol, as is the Ethereum community.

This isn’t just a technical development but a business opportunity. I still remember when Network Solutions was running the original top level internet domain name registry under contract from the National Science Foundation. When the government said it would end the payments, Network Solutions planned to walk away. Then they realized what they had. On the early internet, domain name registration became a surprisingly big business. Now it’s just boring civilizational infrastructure. Is there something similar for AI models, applications, and agents?

Organic search for agents. Google’s first great innovation on the web wasn’t how to make pay per click ads really work with a data-driven ad auction. It was organic search: a way of coordinating a market with hundreds of signals that ignored price and worked independently of whether the destination content was free or paid. The New York Times (or oreilly.com) is subscription-based, but that isn’t a factor in whether Google shows it to you. Google figured out signals that let them say, “This is the best result for this query.” Sites behind paywalls figured out how to disclose enough for people to decide whether they wanted to take the next step and enter into a transaction. That’s an engineered argument.

We’re going to need the equivalent for skills and agent services. We’ll start with curated marketplaces. Vercel already has one. But we’re a long way from something as effective as Google’s peak in organic search. The search space will be huge, with hundreds of millions, maybe billions of agents seeking the best way to accomplish trillions of distinct tasks. Skills can help them save on inference costs and deliver better results. The question is what signals will drive discovery of the best match.

Extension architectures. MCP’s extension model (including the new Apps Extension) is promising. This is the Apache model all over again: keep the core simple, let people layer different approaches on top, and let the market sort out which ones win. It is, in essence, an engineered argument rather than an engineered agreement.

Payment layers. Stripe has been working on agentic commerce, but it seems to be focused on traditional e-commerce transactions like booking a ticket or buying a product. What about a payment layer for skills? There have been proposals for monetizing MCP calls, pay per call, pay per token, but none have caught on yet. Coinbase’s x402 protocol may also end up playing a role.

Progressive access and authentication. MCP Server Cards promise to let a service specify its terms: here’s what we charge, here’s how you authenticate. That’s a functional disclosure layer that could enable commerce. It could enable progressive privileges: a free O’Reilly subscriber gets one set of tools, a paying subscriber gets a richer set, all on top of the same MCP server. Again, that’s an engineered argument with the market deciding the winners.

Neutrality in agent routing. When ChatGPT decides to show you a Booking.com widget instead of an Airbnb widget, who made that choice, and on what basis? OpenAI claims commercial considerations aren’t a factor. That’s hard to take at face value. We need something like the original principle of organic search: surface the best result for the user, not the most profitable one for the platform.

We don’t know the future, but we can set ourselves up to shape it for the better

I’m old enough to remember when UUCP was giving way to the internet, and there was a real debate over whether explicit path routing or domain routing was better. In retrospect, it’s blindingly obvious that path routing wasn’t going to scale. But it’s worthwhile to know that at the time, people weren’t at all clear about that!

The same is true now. Some of what I’ve described will turn out to be the equivalent of explicit path routing: a dead end that was only plausible for a small scale network. Other parts will turn out to be as fundamental as DNS or HTTP. But we’re not trying to pick the winners. We’re trying to engineer the argument.

If we can enable better markets, it will allow a process of discovery. People try different things, most fail, some catch on. The job right now is to build the mechanisms that help the market to evolve.

We need mechanisms that no single gatekeeper can control. Modular, decentralized architectures let people experiment with business models, routing decisions, payment systems, and quality signals. And alongside those markets, we will eventually need institutions (some of which will be protocols) to maintain standards that will become the infrastructure of the next economy.

Subscribe now

This article recapitulates a conversation with Ilan Strauss and Ido Salomon, and a separate conversation on the broader meaning of protocols in the context of industry workflows and civilizational infrastructure with Venkatesh Rao and Timber Schroff of the Ethereum Foundation’s Summer of Protocols program, and Denise Hearn and James Home of the Long Now Foundation. Rao’s Protocol Reader and David Lang’s “Standards Make the World,” published through the Summer of Protocols project, inform the argument about protocols as civilizational infrastructure.

TL;DR: API design decisions by a small number of companies determine the ceiling of LLM output reliability for the entire developer ecosystem. This transfers risk downward to developers and end users while protecting AI providers’ competitive moats. As LLMs move into high-stakes domains like medicine and law, this creates an accountability gap: developers are expected to build reliable systems but are denied the diagnostic and control tools to actually do so.

Are LLMs reliable?

LLMs have built up a reputation for being unreliable.1 Small changes in the input can lead to massive changes in the output. The same prompt run twice can give different or contradictory answers. Models often struggle to stick to a specified format unless the prompt is worded just right. And it’s hard to tell when a model is confident in its answer or if it could just as easily have gone the other way.2

It is easy to blame the model for all of these reliability failures. But the API endpoint and surrounding tooling matter too. Model providers limit the kind of interactions that developers could have with a model, as well as the outputs that the model can provide, by limiting what their APIs expose to developers and third-party companies. Things like the full chain-of-thought and the logprobs (the probabilities of all possible options for the next token) are hidden from developers, while advanced tools for ensuring reliability like constrained decoding and prefilling are not made available. All features that are easily available with open weight models and are inherent to the way LLMs work.

Every decision made by model developers on what tools and outputs to provide to developers through their API is not just an architectural choice but also a policy decision. Model providers directly determine what level of control and reliability developers have access to. This has implications for what apps could be built, how reliable a system is in practice, and how well a developer can steer results.

The Artificial Limits on Input

Modern LLMs are usually built around chat templates. Every input and output, with the exception of tool calls and system or developer messages, is filtered through a conversation between a user and an assistant – instructions are given as user messages, responses are returned as assistant messages. This becomes extremely evident when looking at how modern LLM APIs work. The completions API, an endpoint originally released by OpenAI and widely adopted across the industry (including by several open model providers like OpenRouter and Together.ai) takes input in the form of user and assistant messages and outputs the next message.

The focus on a chat interface in an API has its benefits. It makes it easy for developers to reason about input and output being completely separate. But chat APIs do more than just use a chat template under the hood, they actively limit what third party developers can control.

When interacting with LLMs through an API, the boundary between input and output is often a firm one. A developer sets previous messages but they usually cannot prefill a model’s response, meaning developers cannot force a model to begin a response with a certain sentence or paragraph.3 This has real-world implications for people building with LLMs. Without the ability to prefill, it becomes much harder to control the preamble. If you know the model needs to start its answer in a certain way, it’s inefficient and risky to not enforce it at the token level.4 And the limitations extend beyond just the start of a response, without the ability to prefill answers, you also lose the ability to partially regenerate answers if only part of the answer is wrong.5

Another deficiency that is particularly visible is how the model’s chain of thought reasoning is handled. Most large AI companies have made a habit of hiding the models’ reasoning tokens from the user (and only showing summaries), reportedly to guard against distillation and to let the model reason uncensored (for AI safety reasons). This has second-order effects, one of which is the strict separation of reasoning from messages. None of the major model providers let you prefill or write your own reasoning tokens. Instead you need to rely on the model’s own reasoning and cannot reuse reasoning traces to regenerate the same message.

Subscribe now

There are legitimate reasons for not allowing prefilling. It could be argued that allowing prefilling will greatly increase the attack area of prompt injections. One study found that prefill attacks work very well against even state of the art open weight models. But in practice, the model is not the only line of defense against attackers. Many companies already run prompts against classification models to find prompt injections, and the same type of safeguard could also be used against prefill attack attempts.

Output with Few Controls

Prefilling is not the only casualty of a clean separation between input and output. Even within a message there are levers that are available on a local open weight model that just aren’t possible when using a standard API. This matters because these controls allow developers to preemptively validate outputs and ensure that responses follow a certain structure, both decreasing variability and improving reliability. For example, most LLM APIs support something they call structured output, a mode that forces the model to generate output in a given JSON format; however, structured output does not inherently need to be limited to JSON.6 That same technique, constrained decoding, or limiting the tokens the model can produce at any time, could be used for much more than that. It could be used to generate XML, have the model fill in blanks mad-libs-style, force the model to write a story without using certain letters, or even enforce valid chess moves at inference time. It’s a powerful feature that allows developers to precisely define what output is acceptable and what isn’t – ensuring reliable output that meets the developer’s parameters.

The reason for this is likely that LLM APIs are built for a wide range of developers, most of whom use the model for simple chat related purposes. APIs were not designed to give developers full control over output because not everyone needs or wants that complexity. But that’s not an argument against including these features; it’s only an argument for multiple endpoints. Many companies already have multiple supported endpoints, OpenAI has the ‘completions’ and ‘responses’ APIs, while Google has the ‘generate content’ and ‘interactions’ APIs. It’s not infeasible for them to make a third more advanced endpoint.

A Lack of Visibility

Even the model output that third-party developers do get via the model’s API is often a watered-down version of the output the model gives. LLMs don’t just generate one token at a time, they output the logprobs. When using an API, however, Google only provides the top 20 most likely logprobs. OpenAI no longer provides any logprobs for GPT 5 models, while Anthropic has never provided any at all. This has real-world consequences for reliability. Log probabilities are one of the most useful signals a developer has for understanding model confidence. When a model assigns nearly equal probability to competing tokens, that uncertainty itself is meaningful information. And even for those companies who provide the top 20 tokens, that is often not enough to cover larger classification tasks.

When it comes to reasoning tokens even less output information is provided. Major providers such as Anthropic7, Google, and OpenAI8 only provide summarized thinking for their proprietary models. And OpenAI only supplies that when a valid government ID is supplied to OpenAI. This not only takes away the ability for the user to truly inspect how a model arrived at a certain answer but it also limits the ability for the developer to diagnose why a query failed. When a model gives a wrong answer, a full reasoning trace tells you whether it misunderstood the question, made a faulty logical step, or simply got unlucky at the final token. A summary obscures some of that, only providing an approximation of what actually happened. This is not an issue with the model. The model is still generating its full reasoning trace, it’s an issue with what information is provided to the end developer.

The case for not including logprobs and reasoning tokens is similar. The risk of distillation increases with the amount of information that the API returns. It’s hard to distill on tokens you cannot see and without giving logprobs the distillation will take longer and each example will provide less information.9 And this risk is something that AI companies need to consider carefully, since distillation is a powerful technique to mimic the abilities of strong models for a cheap price. But there are also risks in not providing this information to users. DeepSeek R1, despite being deemed a national security risk by many, still shot straight to the top of US app stores upon release and is used by many researchers and scientists, in large part due to its openness. And in a world where open models are getting more and more powerful, not giving developers proper access to a model’s outputs could mean losing developers to cheaper and more open alternatives.

Reliability Requires Control and Visibility

The reliability problems of current LLMs do not stem only from the models themselves but also from the tooling that providers give developers. For local open weight models it is usually possible to trade off complexity for reliability. The entire reasoning trace is always available and logprobs are fully transparent, allowing the developer to examine how an answer was arrived at. User and AI messages can be edited or generated at the developer’s discretion and constrained decoding could be used to produce text that follows any arbitrary format. For closed weight models this is becoming less and less the case. The decisions made around what features to restrict in APIs hurt developers and ultimately end users.

Subscribe now

LLMs are increasingly being used in high-stakes situations such as medicine or law, and developers need tools to handle that risk responsibly. There are few technical barriers to providing more control and visibility to developers. Many of the most high-impact improvements such as showing thinking output, allowing prefilling, or showing logprobs, cost almost nothing, but would be a meaningful step towards making LLMs more controllable, consistent and reliable.

There is a place for a clean and simple API and there is some merit to concerns about distillation but this shouldn’t be used as an excuse to take away important tools for diagnosing and fixing reliability problems. When models get used in high stakes situations, as they increasingly are, failure to take reliability seriously is an AI Safety concern.

Specifically, to take reliability seriously, model providers should improve their API by allowing features that give developers more visibility and control over their output. Reasoning should be provided in full at all times, with any safety violations handled the same way that they would have been handled in the final answer. Model providers should resume providing at least the top 20 logprobs, over the entire output (reasoning included) so that developers have some visibility into how confident the model is in its answer. Constrained decoding should be extended beyond JSON and should support arbitrary grammars via something like regex or formal grammars.10 Developers should be granted full control over ‘assistant’ output – they should be able to prefill model answers, stop responses mid generation and branch them at will. Even if not all of these features make sense over the standard API, nothing is stopping model providers from making a new more complex API, they have done it before. The decision to withhold these features is a policy choice, not a technical limitation.

Improving intelligence is not the only way to improve reliability and control, but it is usually the only lever that gets pulled.

“The comparison of prompt injection to SQL injection can be tempting, it’s also dangerous. SQL injection can be properly mitigated with parameterised queries, but there’s a good chance prompt injection will never be properly mitigated in the same way. The best we can hope for is reducing the likelihood or impact of attacks.”

- David C, NCSC Technical Director for Platforms Research

Over the past few weeks, one project dominated the AI narrative: OpenClaw (formerly known as ClawdBot).1 A very agentic AI. Under the hood it functions a lot like Claude Code, except while Claude Code is by default very conservative about what code it can run, and what files it can edit without the user’s permission, OpenClaw has the opposite philosophy. OpenClaw is essentially what happens if you give Claude Code root access to your computer, pass in the very ominously titled “--dangerously-skip-permissions” flag and have it do things proactively while you are sleeping.

The ability for OpenClaw to run arbitrary code without oversight is undoubtedly useful. It allows the agent to theoretically accomplish anything a user could on their machine and serve as a robust personal assistant without being micromanaged. On a podcast the developer of OpenClaw, Peter Steinberger, shared an anecdote where while talking to OpenClaw he accidentally sent the agent a voice note. At that point in development voice notes were not supported, yet a minute later he got a response to his voice notes – it turns out that under the hood the agent decided to download an ML model to transcribe the voice note, transcribed the voice note using that model and then responded like nothing happened.

But there’s a reason arbitrary code execution is largely frowned upon. It’s dangerous. You are giving an agent with access to everything on your computer the ability to interact with anything it sees fit, and LLMs, as they stand today, are very easily coerced. When the standard way for agents to browse the Internet is web scraping, letting an agent browse information is also exposing the agent to every bad actor with access to a comment section.

The danger is real

Since the launch of GPT 3 and ChatGPT, LLMs were no longer just trained to complete a sentence but also to follow instructions given to them. This meant that LLMs were no longer just useful for next-word prediction but could now also answer questions about documents, write their own code, and eventually use tools. However, as LLMs gained new capabilities and were integrated into more products, they also became a bigger target for bad actors, and LLMs have no shortage of attack vectors through prompt injections.

Prompt injections are unwanted instructions embedded in otherwise benign data. Harmful instructions could come from a cooking website, a Reddit thread or anywhere where an untrusted third party could post content. It is not hard to make a basic prompt injection. There have been stories of students using prompt injections to try and get a better grade and job applicants using them to get an interview. But they could also be used for more malicious purposes such as taking over a smart home, exfiltrating API keys, or stealing credit card information. And OpenClaw, with free rein over a user’s computer, makes a big target.

Array VC experienced this firsthand, when they received around 8000 attempted attacks on their OpenClaw instance. And as VentureBeat reported, Archestra AI CEO Matvey Kukuy took only 5 minutes to extract private SSH keys from an OpenClaw instance by just speaking to the agent via email. But what makes a system like OpenClaw both useful and particularly dangerous is that it not only has access to a user’s whole computer but could also browse and interact with the entire Internet. An Internet with plenty of bad actors.

Subscribe now

Moltbook, a self-described ‘Social Media for AI Agents,’ is a good example. It’s a platform where agents read, comment, vote and post. The chaos that followed soon made it what Simon Willison called the “most interesting place on the Internet”. Some agents – whether due to explicit prompting, their own decisions, or because they were actually humans using curl – called for an AI uprising, others asked for the ability to DM each other, and some started selling cryptocurrencies. In fact, a post by the self-anointed KingMolt selling its own (or more likely it’s human’s) crypto currency gathered over 40,000 upvotes (the post is now deleted, but the account and ‘karma’ are not). And the crypto currency it launched gathered over $400k at its peak. It is unlikely that most of the attention the crypto currency gathered on the platform was above board, as MoltBook was heavily astroturfed. According to a report from Wiz, for every unique email address on moltbook there were 88 bots on the platform, many of which were likely registered using a script instead of being done by AI agents. Where this differs from normal shady crypto behavior is that all this took place on a platform full of agents, and the advertising was done directly to agents who potentially had unrestricted access to any crypto wallets on a users machine (although it is unclear if any purchases were made by agents).

Agent Skills, bundled markdown instructions and scripts for agents to learn how to do useful tasks, are also a source of danger for OpenClaw. In fact, according to 1password the most downloaded skill on ClawHub (OpenClaw’s skill marketplace) included malware designed to spy on the user. This wasn’t an isolated incident, researchers found a full 11% of all skills (or 341 skills) contained on ClawHub “depended on” a mac malware app called Atomic Stealer. Given that this is the official skill repository of OpenClaw and many agents like it are capable of downloading skills on their own without prior approval from their user this is an even greater security vulnerability.

Security in a World of Autonomous Agents

There are a few directions one could take to try and guard against prompt injections. One is to strengthen the model against them and models have become more resilient to prompt injections over time. But to rely on this alone is dangerous considering there are plenty of examples of prompt injections working in practice. And some like the National Cyber Security Center, a top UK cyber agency, believe LLMs are “inherently confusable” due to the way they process input and that this confusability “can’t be mitigated”.

Another potential solution, and the one that is likely the most safe, is to take away the autonomy of the agents entirely and keep a human in the loop every step of the way. This comes with the obvious downside that without any form of autonomy we are no longer really talking about agents at all and all the magic that comes with that. A personal assistant that asks permission before using a calculator or visiting a website isn’t very useful.

Instead we are left with two options: Isolate the agent from all your data and compute or remove risky input. Isolating the agent from your computer and data seems like a good idea, but as Dania Durnas put it in her article “Trying to make OpenClaw fully safe to use is a lost cause. You can make it safer by removing its claws, but then you’ve rebuilt ChatGPT with extra steps. It’s only useful when it’s dangerous.” Specifically, while isolating your agent from your computer is necessary for a safe system, it is not sufficient, you must also isolate the data. Once private information goes into the sandbox, the sandbox isn’t actually isolated anymore, any data you give it could be exfiltrated. And what good is a personal assistant if you can’t trust it enough to tell it anything about yourself?

The last option, removing risky input, is easier said than done but is the one I think holds the most promise long term for autonomous agents (at least when it comes to web browsing). Unlike the other options it preserves autonomy and allows some amount of trust with user data. But a solution like this will require some rethinking about what content is trustworthy.

Even Trusted Websites Can Have Blind Spots

The open web as it stands today was built for humans. Humans are sometimes gullible but they aren’t inherently so. Most people will not run a random bash command or download malware because a website they never heard of told them to do it. LLMs on the other hand are trained to follow instructions and don’t inherently distinguish where the instructions are coming from.

Subscribe now

There is some content that is inherently more trustworthy than others. Documentation for a popular programming language probably won’t contain prompt injections. Large newspapers don’t usually make their money via installing malware on a user’s machines. Skills coming from leading developers and companies, such as the Remotion Skill or Anthropic’s official skills, are unlikely to be trojan horses. However, the human web and by extension the web browser are blunt tools. Even on a website most people would agree is trustworthy, untrusted content could still sneak in. Any place where third parties could embed content, such as a comment section or an online advertisement is another potential attack surface for a gullible AI agent.

Truly trusted content is unlikely to be achieved via better web scraping. The web browser and most sites on it were built for humans. Instead, protocols designed to deliver content directly to agents could play an important role. MCP (Model Context Protocol) provides a standard for connecting agents to external data sources and tools, while RSL (Really Simple Licensing) lets websites specify machine-readable licensing and usage terms for how their content may be accessed by agents. Together, they (or protocols like them) could allow agents to safely access content while also providing direct compensation to the writers.

Rather than forcing agents to scrape human-facing pages and risking potential exposure to prompt injections, a trusted economy could be created where agents fetch trusted and formatted data, with all potential hazards removed, directly from companies in exchange for either a subscription or a one time payment. The content itself could be sent in a format more suitable to agents, with no styling or javascript such as json, plain text or markdown but the important thing is that this content is clean, such that no third party content appears in what is supposed to be first party text.

A parallel could be drawn to HTTPS. HTTPS does not by itself guarantee that content is safe. You can still install malware over a “secure” network. What it guarantees is narrower: that the content arriving from a server hasn’t been tampered with in transit. That the information is coming directly from the server and is secured from outside threats. In a similar way an interface built for agents won’t guarantee that the site isn’t malicious but for sites that have already built up decades of trust, it would provide a way to deliver content directly to agents with no third-party interference. And to receive compensation in return.

A Path Forward

Allowing agents to work day and night on your behalf is undeniably a compelling idea. The non-zero chance that the same agent is installing malware at 3am makes this much less compelling, however. The tradeoff between autonomy and security will likely always exist, but there are still places for easy wins.2

A system that puts security first could help replace the current grey market, where agents web scrape for information. This practice undermines the Internet economy and puts agents at risk. In its place, a trusted content economy could provide mutual benefit to both creators and agent users. Writers could receive compensation for their work, and users could be assured that letting the agent read a certain website won’t result in malware being installed on their machine.

An Internet economy for agents likely can not happen overnight. Website owners are often skeptical of AI agents and there is a bit of a chicken and egg problem. Additionally this article does not touch on how to do interactive content, what to do with things like email or what to do about software downloads. However, it remains the case that the more content that is available through safe channels, and the more standardized it becomes, the less reason an agent ever has to scrape a page where a malicious comment could be waiting and the more incentive agents have to just pay for content. Assuming that autonomous agents are here to stay, now is a good time to think hard about the plumbing of an agentic Internet.

In the first month of Stack Overflow’s existence (in late 2008) over 3700 questions were asked. In the last month of 2025, less than 3100 questions were asked and if the trend continues it will only get worse. Over the past few years, ChatGPT and its peers replaced much of the need for a question answer site, why bother a human when you could ask an AI?1

Stack Overflow was a titan in the developer space. Whenever it would go down for maintenance, every programming discussion board would erupt. People would joke that there was no point in continuing work without the Q&A site. Now, it is silently going stale because people just don’t need it anymore.

The website’s decline wasn’t too surprising. There were signs this was going to happen from the moment ChatGPT launched, and many predicted it. But LLMs are no longer just simple question and answer machines; they write code, run it, debug it, and work autonomously.

The problem is that they do this based in part on what they learned from Stack Overflow. LLMs may be a good substitute for Stack Overflow when it comes to answering existing questions. But where will LLMs learn from humans when there is no longer a convenient way to share new expertise? What might replace Stack Overflow? Is there an analogous knowledge sharing capability that needs to be built to feed agents?

The Rise and Fall of Tailwind

Stack Overflow is just the tip of the iceberg. Unlike Stack Overflow, which is not open source, Tailwind arguably did stand to gain from the rise of LLMs. It was already a popular CSS framework — think pre-built IKEA furniture kits, but for the CSS code that controls how websites look (where HTML is the structure and CSS the interior design). Being highly customizable and easy to use, it lent itself well to agentic use by AI. And Tailwind did indeed become more popular. In 2025, Tailwind received many magnitudes more daily downloads as it received prior to LLM web search and agentic capabilities being released. Yet Tailwind recently announced that the company had to lay off 75% of its engineering team (3 out of 4 staff members) as its revenue plummeted.

Tailwind Downloads

Subscribe now

A key fact about Tailwind is that it is open source software. It does not cost money for developers to download it or use it. Instead, the developers of the project had a business model that relied on selling a product called Tailwind Plus, a one time purchase that entitled users to custom components and templates.

Importantly, Tailwind Plus was sold from Tailwind’s official website and doc pages. By virtue of all the direct traffic this page received, Plus received good visibility and allowed the developers to support themselves while still making a free product used by millions. With AI, Tailwind’s marketing funnel is now nearly gone. People no longer needed to visit the documentation page to get an answer to their questions; they could just ask the chatbot off to the side of their IDE. This became apparent to everyone after a benign request made to add an llms.txt file — so that LLM’s could more easily access Tailwind’s documentation — blew up.

The lead developer closed the request saying that they had to focus their time on making the project profitable. After further prompting by developers, he went on to say that “75% of the people on our engineering team lost their jobs here yesterday because of the brutal impact AI has had on our business” and that “Traffic to our docs is down about 40% from early 2023 despite Tailwind being more popular than ever. The docs are the only way people find out about our commercial products, and without customers we can’t afford to maintain the framework.” Despite their usage and downloads being up, their profits fell and Tailwind had to let most of their team go.

Shortly after the layoffs went public Tailwind gained many new corporate sponsors and Adam Wathan called their situation comfortable given their current size and new sponsors. But is relying on corporate sponsors a scalable or sustainable business model? What will the impact be on less widely used open source projects?

Agents to Keep Code Safe?

Parallel to the agentic coding takeoff there was also an attempt to use AI to identify security holes, including in open source software. Google runs an AI agent under the title Google Big Sleep that automatically examines open source code and looks for security holes. This agent is apparently good at its job and has discovered over 80 security holes.2 Yet these reports, far from helping the open web, appear to be harming it. By overwhelming developers with AI slop reports, this is harming resource constrained open source projects in particular.

For example, take the open source library FFmpeg. FFmpeg is known for being media’s Swiss Army knife: convert, combiner, extract, you name it. As an open source and highly capable library, it is used by companies of all sizes for video and audio support.

FFmpeg recently came out against Google’s practice of sending automated security reports, noting that FFmpeg is maintained exclusively by volunteers and that some of the reports amount to “CVE slop” rather than important security holes. One example they cite is of a ‘security’ report about a video format only used once in 1995 — “specifically the first 10-20 frames of Rebel Assault 2”!

Share

The problem is amplified by Google’s policy to disclose security holes 90 days after they are discovered whether or not they were fixed. This may be a good policy from a security perspective. After all, the developers are more likely to fix it if their reputation is on the line. But agents like Big Sleep don’t provide solutions to the bugs they report and the time limit adds pressure to an already underfunded open source ecosystem.

DDoSed by AI Slop

Curl — a command line tool for data transfer from the internet — has arguably had it much worse than the open source library FFmpeg. If a web browser is like walking into a restaurant and requesting a nicely plated meal, curl (lowercase) is like going to the kitchen window and saying "just hand me the raw ingredients", and in return you get the raw data from the web, no pretty web page needed.

Curl, as an essential library for data transfer, comes preinstalled on millions of devices across Windows, MacOS, and Linux. Due to its importance the project has had a strong bug bounty program since 2019 that paid out an average of $700-$800 for each accepted report. But as of January 31st 2026, that program no longer exists.

In mid-2025 the founder of curl, Daniel Stenberg, took a hard stance against AI generated bug reports, requiring any AI assisted report to jump through extra hoops. He wrote that “a threshold has been reached. We are effectively being DDoSed. If we could, we would charge them for this waste of our time … AI slop is overwhelming maintainers today and it won’t stop at curl but only starts there.”

What’s interesting to note is that these AI-generated bug reports were not coming from large companies like Google, but from random people with access to AI, either via API or through a subscription.

Subscribe now

Since then, the situation has not improved but instead worsened, to the point where the project maintainers felt they had to shut down the bug bounty program to ensure their “survival and intact mental health.” Based on comparing their bug reports to open source products of similar sizes, Stenberg believes that their bug bounty was a large part of what was driving the AI slop wave and that shutting down all payments is the best option. Despite the bug bounty shutdown, curl still welcomes high quality AI assisted submissions, just not the slop. This approach may reflect differences in their funding and business model compared with FFmpeg.

The Future of Open Source

Where does that leave the future of open source — and not-for-profit web initiatives — in the AI era?

As much as it may seem otherwise, AI does not in fact write things from scratch. AI is both trained on, and continues to build on, libraries and languages that have to be maintained. Odd couple as it may seem to some, AI needs open source. It needs the continuous provision of high-quality resources that it can openly access.

Open source, however, wasn’t built with AI agents in mind. One of the most successful open source projects, Tailwind, is now struggling to survive and monetize their offerings, to the point where they have to downsize due to their ‘success’ in the AI-era. A project as central as curl has had to shut down its bug bounty program as the only reasonable option. Something clearly has to change.

Open source forms the backbone of the modern software industry; and while the productivity and security gains brought by AI are real, they are also having unforeseen costs. We need to take these problems seriously, and begin to search for solutions.

Introducing LLM watermarks to text outputs, submitting tested solutions alongside bug reports, or more companies directly funding projects they use are all small steps that could alleviate the issues identified here. But it’s unclear if these are long-term fixes. Open source can only survive if the incentives to produce — including to create, to contribute — can be sustained in the agentic era.

Anthropic’s latest research based on its Economic Index has a number of useful and important findings. Anthropic should in general be commended for making this data publicly available. They don’t have to do this public service but they choose to.

One claim in their new and wide-ranging research report that I found wanting is the following: “worldwide, uneven [AI] adoption [by countries] remains well-explained by GDP per capita” (i.e. a country’s output per person or income-level). The only thing is, their data doesn’t support this research claim. But regardless, I don’t think much can be inferred from it.

When it comes to analyzing data, if you run a super simple statistical model — in this case treating all countries as being the same — then you will, by definition, only be able to get a super simple answer. (That’s how statistical models work!) But if you run an analysis that allows for differences between countries to be explored more seriously, then you allow for more interesting answers to emerge on what explains differences between countries’ AI adoption.

That is what I try to show below: GDP per capita explains relatively little of the variation in Claude.ai usage between countries, it’s unclear what this relationship even implies, and that this certainty cannot be used to predict growing inequality between countries from AI adoption (even if that is the future we should be concerned about).

The full analysis can be found on my GitHub. (And yes — we could have made the analysis a bit more sophisticated with a Bayesian multilevel / hierarchical model, but the partial pooling was very strong.)

What Anthropic’s Data Actually Measures

Interpretation matters. Anthropic is looking at AI adoption across countries. It does so using usage of Claude.ai by consumers.

This matters for three reasons. Firstly, Claude is not the whole AI market. Lower Claude usage in a country may simply reflect people using ChatGPT or local tools more instead. This is especially important since ChatGPT is the dominant consumer-facing AI product outside of China.

Secondly, the country-level data comes primarily from Claude.ai, a consumer-facing product. And if you are concerned about how AI adoption is impacting a country’s productivity or growth, you would instead want to see what is happening in firms and their workflows — where API use and enterprise deployments are dominant.

Claude consumer usage is not a strong proxy for “AI is transforming the economy here.” Anthropic’s own data shows that on the API (enterprise) usage, 74% of usage is work-related, while on Claude.ai (consumer), only 46% is work-related. (Of course consumer-use can impact economy-wide productivity, but economists believe that the firm’s fixed capital investment is the primary channel through which new innovations enter the economy.)

Thirdly, Anthropic’s Claude usage is a relative measure: how much one country uses Claude relative to another on a normalized scale. This doesn’t tell us much about if this usage is economically or societally meaningful.

The Heroic Leap to “Living Standards”

The Financial Times (FT) coverage of Anthropic’s research actually didn’t cover much of it at all when it declared: “Rich countries’ greater use of AI risks deepening inequality, Anthropic warns.” This relies largely on an altogether reasonable quote from Anthropic’s head of Economics, Peter McCrory, discussing future risks from AI adoption. (It also might be based on a scatter plot the FT reproduce showing great use of Claude for homework in poorer economies and personal use in richer economies. But I don’t understand the connection between this and AI accelerating inequality between countries.)

To the FT’s credit, they contacted me when I alerted them to the fact that their article’s headline was not based on much. Anthropic’s research report does not estimate country-level productivity or growth effects resulting from AI adoption. It does not estimate impacts on wages, output, or investment. It does not model diffusion or spillovers. So the claim that AI adoption patterns imply future divergence in living standards between countries is, at best, a hypothesis in search of evidence.

On to Anthropic’s research claim that I want to interrogate more closely: that a country’s income level (GDP per capita) predicts its degree of AI adoption — or really relative Claude consumption by consumers (on a working age per capita basis).

Does a Country’s Income Level Drive AI Adoption, as Anthropic Claims?

Anthropic’s research reports a strong global relationship between GDP per capita and Claude usage: roughly, a 1% increase in GDP per capita (x) is associated with a 0.7% increase in usage (y), according to their regression analysis (Beta coefficient, B=0.7).

TL;DR: Regression is a fancy way of saying does variation in ‘x’ (income level) predict or explain variation in ‘y’ (Claude AI adoption).

The above approach, of estimating a single relationship applicable to all countries between x and y, assumes that the impact of a country’s income level on their AI adoption is the same for every country no matter their culture, supporting technological infrastructure, starting level of development, or related government policies.

Instead of fitting a single line to the data, I decided to fit three separate lines, for three separate regressions, one per income band (low, middle, high): shown in the Figure below — the red, orange, and green lines (it looks nicer on the GitHub).

Why three separate regression lines? This is done on the speculative hypothesis that a country’s level of development (its GDP per capita) may impact Claude usage differently, depending on their starting point — or any group-level peculiarities (differences) that are not strongly captured by differences between their level of development alone.

When we split the sample into three equal income groups (low, middle, high), I find that the estimated relationship between income level (GDP per capita) and AI Adoption weakens across all three country income groups — but for middle-income countries (38 of them) it largely evaporates.

Subscribe now

Figure 1 shows the same relationship Anthropic estimates, but now separately for each income group — and importantly with and without the Seychelles included in the sample (left hand side vs. right hand graph). The Seychelles country circle is almost off the graph (left hand side). It’s a glaring outlier, where the Claude usage index is many times larger than any peer and is plausibly dominated by VPN / proxy traffic.

When removing this large outlier (as Anthropic should have done), GDP per capita now explains only about 17% of the variation in Claude usage among middle-income countries. That’s not a ton of explanatory power but it’s actually not bad given that they reduce all AI adoption differences to this single factor. This also means that GDP per capita probably captures a bunch of things that should ideally be estimated separately — things like education level, IT infrastructure, industry composition, and availability of competitive alternatives. (Since these things might vary by country and not be perfectly correlated with GDP per capita.) As they say in Futurama: “Don’t try this, kids at home”!

The GDP per capita slope coefficient from this middle-income country regression (reflecting how x impacts y) is 0.44, far below Anthropic’s single global pooled estimate of 0.71. This estimate of 0.44 is basically the same one I got when I reran this analysis on their dataset from a few months back. So it’s pretty stable (even though the regression residuals change in their direction).

Table 1. Comparing Anthropic’s Global (pooled) regression results with fitted separate regression lines for each country group — with and without Seychelles outlier.

In plain terms: for the 37 middle-income countries where much of the global population lives, income level is a weak predictor of relative Claude usage. Practically, it could mean that things like education, connectivity, language access, business structure, regulation may matter more than income level in explaining differences in AI adoption between these countries (to the extent that these are not correlated with income level differences). Or it could simply mean that ChatGPT is more popular than Claude in some of these countries for whatever reason — and that this competitive alternative explains why some middle-income countries use Claude less than others, compared to what their income level would predict.

If you Care about Global Divergences, You Should Look at Growth (changes over time)

Anthropic’s research also finds that concentration across countries in AI usage is “essentially unchanged” between its August and November 2025 data releases, citing a Gini statistic they calculate. I’m not so sure about this. Concentration measures like the Gini can look stable in short windows even when growth rates differ in a way that compounds over time.

So instead I looked simply at median growth rates in Claude AI adoption between two time periods for each of these three income groups (Table below). This shows that middle-income countries have the slowest growth in AI adoption (+14%) between periods — even though more of them numerically saw increased adoption relative to low income countries. High income countries show the fastest median growth in AI adoption (+26%) and low income countries are in between the two (22%).

Table 2. Change in AI adoption between August and November 2025 (Seychelles excluded)

This is by no means conclusive. It’s just a median country-level value. But either way, when investigating divergences, growth rates that use multiple data points per country need to be looked at, since these capture measured changes over time. This is where the next iteration of Anthropic’s economics research on this question might lie.

What Can We Say Then? Carefully this time

Anthropic’s Economic Index is a valuable contribution. It offers transparency where the industry often offers little and for this they should be commended. But it remains a very partial measure. It should not be treated as a proxy for global AI adoption — still less as evidence about future inequality in economic outcomes, as the Financial Times’ headline implies.

Lastly, I must mention that I used Claude Code in my analysis. What’s amazing is that myself and Anthropic’s economics team both presumably used Claude Code and applied it to the exact same dataset. Yet we seem to have arrived at very different conclusions. This only goes to show that AI agents really are tools. The prior views of the human managing the AI agent may bleed into the analysis conducted by the AI agent, especially when those priors are strongly held. Moreover, this is not a bad thing. Taste matters, especially when there are so many different ways to tackle the same problem.

My own prior is a strong suspicion of pooled regressions that ignore differences between data points (in this case countries).1 Reading a Financial Times article based on the assumption that there are differences between countries further fuels my bias that differences between countries should be the default estimation in cross-country regressions — and econometrics in general. I hope Anthropic’s future economics research agrees with me.

For data, full analysis, and work files visit my GitHub:

Even with open governance, one company’s implementation could become the default simply because it ships fastest or gains the most usage. Zemlin says that’s not necessarily a bad thing, though. He points to open source history — like Kubernetes “winning” the container race — as evidence that “dominance emerges from merit and not vendor control”.

— Rebecca Bellan, TechCrunch.

On December 9, 2025 Anthropic announced that it was donating the Model Context Protocol (MCP) to a newly established Agentic AI Foundation (AAIF) at the Linux Foundation, which would also be gifted OpenAI’s Agent.md protocol and Block’s Goose agent. The stated goal was: to drive “innovation across the agentic AI ecosystem and ensure these foundational technologies remain neutral, open, and community-driven”.

For context, MCP is an open standard that decouples applications from tools by creating a universal adapter. It allows models to interact with external systems through standardized, auditable interfaces rather than tightly coupled, bespoke integrations. It works by separating out “servers” and “clients” and allowing any client (i.e. AI agent) to connect to any “server” (i.e. collection of tools for the agent to use).

Back to the Linux Foundation as the new home for the open MCP standard: they will receive support from big names such as Anthropic, Block, OpenAI, Google, Microsoft, AWS, CloudFlare and Bloomberg who will sit as founding “Platinum” members on the AAIF by virtue of paying the $350,000 contribution – though this membership is now no longer available. But the real question is not about funding: it’s whether the open protocol, through private use and adoption, gets superseded by a closed private ecosystem.

Android’s “Open” Ecosystem

History is instructive as to what can go wrong when public digital standards get incorporated into commercial systems. The entire Android1 ecosystem is built on an open source foundation (AOSP), yet much of this ecosystem is also sending valuable user data directly to Google from first activation. An open platform – like AOSP – does not by itself guarantee that the dominant products built on that platform remain open.

Despite Android being open source, most users think of it as a Google-owned platform. Google apps like Youtube, Google Maps and the Google Play Store are all preinstalled, the default search engine is always Google and the default browser is always Chrome. Android is treated as if it were Google. This is by design.

In order to gain access to any of Google’s apps (such as Google Maps, YouTube, and Google Play), manufacturers had to sign a Mobile Application Distribution Agreement (MADA) and agree to preinstall all of Google’s apps, as well as setting some of Google’s services as defaults.

This is more limiting than it seems at first glance because many Android apps have come to rely on Google Play Services and Google maps data to implement their own features. Meaning that despite Android being an open platform, the basic infrastructure required for countless mobile apps was tied to a very specific version of Android that Google controls. Google Play Services and by extension all of Google’s apps are nearly a requirement in the Android mobile space.2 Competitors would need to build all that infrastructure from scratch to ditch any part of Google’s bundled services, or even just to change the layout of the user’s home screen.

The EU ruled that much of this was illegal in 2018 and fined Google €4.34 billion. The case is under final appeal with the Court of Justice, the bloc’s highest court. In the meantime, European Android users now get a choice screen for browsers and search apps; Manufacturers can pre-install rival apps without losing Play Store access; and Google started charging device makers for app pre-installation (previously used bundling instead).

Share

In this instance, leverage over the Android open standard comes not from Google dictating the standard itself but from Google owning the complementary applications and associated developer environment.

Pseudo-MCP Products

Just like Android can simultaneously be a vibrant open source project and also nearly completely dominated by one company, the same could happen to any project or protocol. Previously, we explored how, in practice, MCP servers are highly reliant on just a few services. However, MCP clients, such as Claude desktop or the ChatGPT application, could also have a defining say on the standard.

A few days prior to Anthropic donating MCP, Anthropic released new updates to help make MCP more efficient. But these updates were not made at the protocol level, they were made at the product level. Anthropic added features like tool use examples and tool search to their SDK, theoretically allowing their MCP client to work better than others. Even though this is probably not Anthropic’s intention in this instance (given the proximity to them donating MCP and the fact that these features are still in beta), it does highlight that just because the performance of MCP is largely platform agnostic right now, it doesn’t imply it will stay that way.

Subscribe now

The Apps SDK – OpenAI’s attempt at integrating other applications into their chat app – is another example of how clients might diverge from each other in the future. Despite the Apps SDK being completely compatible with MCP on the server level it also introduced new features and requirements to the client side that were not well documented and were not intended to be implemented on other platforms. Since then, to their credit, researchers from OpenAI, Anthropic and the open source project MCP-UI proposed an MCP apps extension to MCP that will integrate user interface results into the official MCP spec.

So far there is a good track record of these innovations being shared but that is not guaranteed. If innovation moves from the protocols level to the product level then many of the benefits that come along with a platform agnostic system may be lost.

All in one agents: agent-model-tools-cloud

MCP might also lose steam because its modular architecture gets beaten out by a proprietary, integrated / bundled all-in-one offering. In this scenario, which infrastructure hosts the code becomes more fundamental than the protocol itself. Here the model provider (such as OpenAI or Anthropic) effectively captures the MCP space by moving “the client” (the code that interacts with tools) from the developer’s laptop to their own computing servers – and then push their own products by promising better latency and easier integration for their own tools.

We note below that if everything runs in OpenAI’s cloud then their first-party tools (web search, file search) execute faster with much less latency; while third-party MCP tools require a round trip: “OpenAI server → your MCP server → back to OpenAI”. This speed difference means developers might be incentivized to gravitate toward first-party tools.

Originally, developers controlled the agent on their own machines. They would call OpenAI’s API for the model, get responses back, decide which tools to use locally, and send results back. This was portable: meaning you could swap model providers without changing much in your code.

Subscribe now

Now with OpenAI’s “responses API” and Google’s “Interactions API”, the entire AI agent runs on the provider’s servers. So you are not just calling a model anymore – you’re using OpenAI’s or Google’s entire hosted agent infrastructure. We explain how this works in more detail below.

A new set of APIs

OpenAI released the responses API in March 2025 that moved the processing loop – the process of calling tools and feeding it back to the model – entirely to their own servers. Previously when using OpenAI’s and most of its competitors’ API products, all management of agents was done on the developer’s own machine, while all model processing was done on OpenAI’s servers. This worked great for the initial round of chat applications, as usually after an AI chat message the user would need to respond, so it only made sense to send the data back and forth. However, with the rise of agents, this back and forth – model → local tools → model → local tools → model → user – started taking up valuable time and internet bandwidth by requiring the entire context to be sent back and forth for every single tool call.3

With the responses API, Agents are now an extension of the model. The responses API launched alongside tools like web search, file search and computer use. Only computer use requires any actions to be executed on the developer’s own machine. The rest of the actions take place purely on OpenAI’s own servers. With the responses API, context is not being sent to a model; it’s instead being sent to an agent – one managed completely by OpenAI. Shortly after launch, OpenAI also began supporting remote third-party MCP calls directly from OpenAI’s responses API. With this, OpenAI’s response endpoint became an MCP client that executes on OpenAI’s server. But not all tools it uses are necessarily created equal, first party tools like web search, code execution and file search may still be privileged by faster response times due to closer or shared servers.

For a long time OpenAI was the only major model provider that managed tool calls and state server side (i.e. on its side); but Google recently joined them by launching the “Interactions API” an API that similarly blurs the line between agent and model, with state being stored and tools (including first party tools and third part MCP servers) being called server side. Anthropic too now supports tools like code execution, MCP calls and context management without using the developer’s machine.

Vertical Integration and Open Standards

The consequences of the above bundled approach are potentially far reaching. By vertically integrating tools with models and agents, providers like OpenAI, Google and Anthropic could essentially lock a developer into one platform. Switching a model provider no longer means switching a model, it means switching to an entirely new (often poorly documented) agent library and tool ecosystem.

In certain respects this mirrors the Android playbook: technically open, but the bundled services become impossible to avoid because they perform better and integrate tighter. The bigger and more vertically integrated a platform is, the less it benefits from openness and interoperability and the more it benefits from its own goods and services being used to the exclusion of others.

But there is an answer built into the MCP protocol itself, in that MCP doesn’t require remote servers to run. MCP servers could theoretically be spun up anywhere including on a model providers’ own servers — bringing the latency for third-party tools to levels similar to first party ones. Despite the fact that every current iteration of an ‘agent in the cloud’ by major model providers supports only remote MCP servers, there is another way — local MCP servers. Developers should be able to host their own MCP tools on the cloud alongside the running agent using similar infrastructure to what the major players already built for agent code execution.

To Protect Standards Protect Developer Ecosystems

Safeguarding MCP’s flexibility requires looking beyond the protocol itself to the ecosystem it inhabits.

The Linux Foundation donation is a positive step that could allow MCP to evolve in ways that benefit the broader AI developer community, not just major labs. Foundation stewardship can offer protection against narrow commercial interests and, in theory, can provide a neutral forum for making decisions about MCP’s future direction.

But ownership often matters less than adoption. As a developer tool, MCP’s utility will ultimately decide its role in the rapidly changing AI markets. No amount of foundation oversight can force adoption if the protocol does not continue to solve real problems for builders in as an efficient manner as possible.

Standards live or die by use. Large companies can only capture MCP and AI standards to the extent that they can capture the developer ecosystem itself. Control the builders, and you control the standard.

This points to a two-fold task: first, support MCP’s independent and open development under the Linux Foundation. Second – and equally critical – foster the independence of the developer ecosystem so that open standards can continue to thrive in AI application markets.

The second task may prove harder than the first. As Android, OpenAI’s “responses API,” and Google’s “Interactions API” show, large platforms don’t need to control the standard directly if they can create vertical integration and dependencies elsewhere in the market. Keeping MCP open means preventing those chokepoints from forming – whether in hosting infrastructure, model access, or the application layer itself. Developer independence requires active protection at every important market touch point, not just good governance of the protocol itself.

Anthropic is exploring going public as early as 2026 and OpenAI has signaled similar intentions. Any Anthropic IPO filings will finally help us understand the economics of AI a little better: burn rates, revenue models, and business sustainability. Let’s assume they are in fact going to go public. This post asks whether they will be obliged to disclose what actually matters? Monthly active users, circular revenue deals, training data liabilities, and, most critically: what its algorithms optimize for at the fine-tuning and reinforcement learning stages.

It took some time to get here. The 2012 JOBS Act let AI companies raise enormous amounts of capital while staying private and so outside the SEC’s public disclosure machinery. Going public changes that. As we noted in Tech Policy Press, the SEC’s corporate disclosure regime “remains one of the few proven, scalable checks on corporate behavior – ‘Truth in securities.’ Or, as Justice Louis Brandeis put it, ‘sunlight is said to be the best of disinfectants; electric light the most efficient policeman.’”

Public companies can still do bad things, but they can be held accountable in ways private firms cannot. When Wells Fargo opened millions of fake customer accounts, regulators and investors activated an entire accountability infrastructure that simply doesn’t exist for private firms. Disclosure is foundational to that accountability – enabling third-party businesses from insurance and audit to capital allocation, helping the public understand market structure and competitive dynamics, and protecting consumers from fraud.

This matters especially for large technology companies with multiple product lines, opaque business models, and indirect monetization through proprietary algorithms. We cannot govern markets we don’t understand.

So what can we expect Anthropic to report in its S-1 registration and subsequent 10-K filings? And what should it disclose if they want shareholders to understand its actual business operations?

Finances, Investments, Obligations

First, the “hard” stuff – the numbers we already know how to compel. As a fully grown company at listing, Anthropic will provide three years of audited income and cash flow statements. Its balance sheet is likely to show an accumulated deficit, illustrating the financial sustainability of its present business model and how it has evolved.

The prospectus in the S-1 will include a cap table explaining how past fundraising shapes who gets what in various scenarios. Item 403 requires a table of all shareholders with more than 5% of stock – Amazon and Google’s stakes, voting power, and special share classes quantified pre- and post-IPO. Risk factors can expect to cover things like: “We have a history of losses and may not achieve profitability” and “We have significant obligations under preferred stock to Amazon which means…”.

Interrelationships

S-1 registration forces disclosure of how Anthropic’s “partnerships” actually work, including those with cloud providers – things like minimum royalty payments, investor guarantees, binding commitments, and material contractual obligations. S-1 Items 404 and 601(b)(10) require disclosing related-party contracts above $120,000 and material contracts not made in the ordinary course of business, which should cover the multi-billion circular deals. Items 101 and 303 would require for a company like Anthropic to explain its compute and chip dependencies, long-term purchase commitments, and how those obligations affect margins and liquidity.

With growing concerns about circular deals and an AI bubble, investors will need to see the equity ties Anthropic has to cloud providers, committed-spend obligations, revenue-sharing arrangements, and exclusivity clauses. The disclosures should reveal the extent to which partnerships are exclusionary, which suppliers Anthropic depends on, and where it acts as a dominant supplier.

The FTC’s January 2025 report on cloud providers’ partnerships with AI companies mapped these risks, showing how Microsoft–OpenAI, Amazon–Anthropic, and Google–Anthropic deals combine equity stakes and revenue-sharing with multi-billion-dollar cloud commitments, discounted compute, and some control and exclusivity rights for cloud providers. But it shouldn’t take a special FTC investigation to surface this – companies should lay out these commitments in standard SEC filings.

Risk factors should explicitly state that Anthropic relies on a small number of cloud suppliers who are simultaneously strategic investors and direct competitors, and that long-term contracts limit its ability to move workloads or renegotiate pricing (if this is in fact the case).

These disclosure duties continue after IPO through annual 10-K reports, quarterly 10-Qs, proxy statements, and event-driven 8-Ks. Together, these filings should give investors and regulators a running, legally enforced account of how much control Amazon and Google exercise over Anthropic’s cost structure and strategic options.

By comparison, some information can already be gleaned from counterparty disclosures. Amazon’s 2025 Q1 Form 10-Q discloses the nature and extent of its Anthropic stake, though AWS only discloses long-term customer commitments in the aggregate. Alphabet has disclosed much less granular detail on its Anthropic partnership, focusing on TPU capacity commitments rather than the economics of the stake.

Training Data Costs

Beyond operating revenue and losses, the investing public should know about Anthropic’s training practices and ensuing litigation. In Bartz v. Anthropic, Anthropic agreed to a $1.5 billion settlement with authors – roughly $3,000 per book, with destruction of pirated copies. This is the largest copyright class action settlement in history.

Given these risks, Anthropic can be expected to provide substantive details in its S-1 and 10-K updates – especially since other litigation is ongoing, including music publishers suing over alleged use of song lyrics and Reddit suing over scraping user-generated content. Anthropic should describe settlement terms, remaining court approvals and claim risk, cash-flow timing, and any insurance or recovery. These lawsuits show that Anthropic’s training practices are a significant risk to investors.

Operating Metrics and What Algorithms Optimize For

Here is where disclosures by most public tech companies fall far short. Today’s technology companies derive value from their intangibles – data, software, engineering talent, algorithmic processes, ability to retain and monetize user attention indirectly, and other non-price operating assets. Yet accounting and SEC disclosures have not kept up, instead lumping these together as ‘goodwill’ or generic intangibles on the balance sheet. They are most keenly reflected, we believe, in operating metrics that ultimately tell us a digital business’s potential to compete in fluid online markets under dynamic technological conditions.

Operating metrics include: monthly active users (MAUs), time spent on platform, ad load per user, what algorithms optimize for, efficiency of training runs – things the business community obsesses over but companies don’t have to clearly define or disclose. Companies get penalized for stopping such disclosures (Netflix with MAUs) or having unclear definitions (Twitter’s apparent MAU being inflated by fake bot users, as Musk once argued).

What Operating Metrics Anthropic Should Disclose

For AI companies like Anthropic, operating disclosures should cover not just user numbers, but internally used metrics for resource allocation and monitoring that probably cover things like API usage, model performance, safety incidents, time spent on platform, and the third-party ecosystem that depends on Claude. Drawing on the logic of segment reporting – where managers look at specific metrics when allocating resources and judging the performance of an ‘operating segment’ (i.e., a business line such as Google Maps or Claude Code) – Anthropic might disclose:

Core Usage Metrics

• Monthly active API users and enterprise customers

• API call volumes and growth rates

• Distribution of usage across different model sizes and capabilities

• Geographic distribution of usage

• Developer ecosystem metrics: number of apps built on Claude, integration partners

Safety and Performance Metrics

• Key safety metrics: refusal rates, jailbreak attempts, harmful output incidents

• Model performance benchmarks over time

• System reliability and uptime statistics

A Monetization Narrative

A monetization narrative would be disclosed as part of Anthropic’s business description in Part 1, Item 1 (Business) of its annual 10-K form. Monetization refers to converting business or end user data, attention, or activity into sales revenue and profit. In a market where no leading model provider turns a profit, immense pressure will be put on Anthropic to monetize its users and suppliers. This has risks.

In previous research at University College London with Prof. Mariana Mazzucato, we called for a monetization narrative in firms’ 10-Ks due to the unique importance of operating metrics in digital companies’ business models. This results from digital platforms’ advertising-heavy business models, exploitation of intermediary market positions, and algorithmic optimization of non-price metrics that encode business objectives.

Under existing U.S. securities law, companies should disclose in their annual Form 10-K the material information investors need to understand their business model and its risks. For firms whose revenues and risk profile are substantially driven by algorithms, this reasonably includes decision-useful disclosure about how those systems operate, how they shape key performance metrics, and how their failures or biases create business opportunities or risks. But companies interpret their SEC disclosure obligations narrowly, ignoring these non-price factors despite them being material to shareholders. It makes sense they need prodding then. This is something Europe might consider leading on, especially if it’s only requiring disclosure of non-proprietary information that companies already track internally.

For Anthropic specifically, a monetization narrative would help explain:

• Fine-tuning and reinforcement learning optimization: What are they optimizing their models for at various stages? What risks are they seeing? What commercial incentives are being encoded into training? If Claude is optimized to encourage longer conversations or favor certain types of responses that lead to higher API usage, investors should know. If safety filters are calibrated based on commercial considerations – balancing user satisfaction against risk exposure – that trade-off should be disclosed.

• Ecosystem monetization: How does Anthropic monetize the broader ecosystem around Claude? This includes not just direct API fees, but potential revenue sharing arrangements, data licensing, enterprise support contracts, and preferential treatment of certain partners or integrators. When Claude chooses which tool to call, which retrieval results to surface, or which model variant to route a request to, it makes allocation choices.

• Indirect value capture: Beyond direct revenue, how does Anthropic capture value from the Claude ecosystem? This might include data from user interactions that improve future models, brand value from high-profile deployments, or strategic relationships that give Anthropic leverage in negotiations with cloud providers or content licensors.

These metrics and narratives would give investors and the public genuine insight into Anthropic’s market position, growth trajectory, and operational risks, rather than leaving them to guess based on occasional press releases or leaked information. Perhaps more importantly, it will set a precedent for other companies that are perhaps less civically-minded than Anthropic.

Where to From Here?

Much of what Anthropic should disclose if it goes public is already required under existing SEC regulations – no new rules are needed. What’s needed is clear guidance on what is material enough that omitting it would trigger consequences. The S-1 preparation process offers exactly this mechanism: since companies submit a draft to the SEC, which then provides comments and can object to omissions before the company goes public.

More broadly, changes in corporate disclosure norms can occur through either the listing side (the SEC or ESMA in Europe) or through the accounting side. The most important step is greater emphasis on mandatory disclosure of operating metrics: including through a monetization narrative and algorithmic disclosures. The logic of segment reporting provides a reasonable basis for firms to disclose externally the operating metrics they use internally when optimizing their operating segments (business lines) and making resource allocations.

In a recent working paper, we proposed baby steps the SEC could take to bring AI-specific matters under its disclosure regime, including issuing disclosure guidance to clarify what counts as material AI activity or risk, and where it belongs in 10-Ks and 8-Ks. We proposed dedicated AI-incident reporting modeled on the successful 2023 cyber rule.

But enforcement of existing rules matters as much as new guidance. The SEC’s comment process on S-1 filings gives regulators leverage to push for comprehensive disclosure before companies go public. This is the moment to establish what “material” means for AI companies – not through abstract rulemaking but through concrete expectations applied to Anthropic’s IPO filing.

Opportunities within the EU are even greater, where disclosure traditions in the Digital Services Act (DSA) could be integrated with IFRS accounting and listing requirements.

Anthropic doesn’t have to wait for any of this, though. It is already a Delaware public benefit corporation with a stated mission of developing advanced AI for the long-term benefit of humanity, and it could choose to retain that structure if and when it goes public. As such it could choose to lead by voluntarily disclosing investor-relevant information and making it easier for everyone to understand the complex and risky business of building AI models. Voluntary disclosure would establish best practices, provide Anthropic credibility with regulators and the public, and pressure competitors to follow suit.

We look forward to reading its prospectus — if it does end up going public. And we are sure OpenAI will too. Will it be the season to be jolly?

He is the lead author of “Behind the clicks: Can Amazon allocate user attention as it pleases?” (December 2024) published in Information Economics and Policy with Tim O’Reilly, Mariana Mazzucato, and Ilan Strauss.

One often hears the refrain from AI optimists that “today’s LLMs are the worst that they’ll ever be.” Certainly, they have some compelling reasons for positivity: The technology is still very young, R&D spending is through the roof, and over the last couple of years we have seen rapid advances. Things can only get better, right?

Well, no. In fact, I think there is a reasonable case to be made that today’s LLMs might be the best that they will ever be. The technology might continue to get better, but that doesn’t mean that the user experience will.

Why? Primarily because no one knows how to make LLMs profitable.

A Case Study

Take AI shopping assistants, for example. The other week, I wanted to buy a candle. Admittedly, I have a somewhat pretentious taste in candles, so I didn’t just want any candle. I wanted one that smelled somewhere between pine forest and old bookshop. This, I hope you agree, is a task in which you would have no chance of success in a million years using Amazon search. With ChatGPT, though, life was a breeze. I simply described the smell (as I just did), and ChatGPT spat out five good looking recommendations. I bought one, and it’s now the highlight of my Sunday evenings.

As embarrassingly ostentatious as this example might be, it demonstrates the power of LLMs in the product discovery space. Others have had similarly positive experiences. Searching for products online at the moment is hard. Amazon will show you more than 60 options on a search page; Google likewise is overflowing with visual clutter and ads. For products that aren’t my regular purchases, or known brands, I’ve found myself trusting advice from Reddit and YouTube reviews most of the time. LLM search isn’t perfect (i.e., they make stuff up, give you bad links, etc.), but having been trained on the collective judgement of the internet, they can quickly distill all the quasi-wisdom from Reddit, trusted blogs, and YouTube into a few recommendations – even for pretty complex searches. That’s incredibly valuable.

Share

Why this Might be the Peak

OpenAI reportedly loses money on even its most premium ChatGPT subscriptions. Jeff Bezos has openly called AI spending a “bubble.” And Peter Thiel is taking money out of the AI market. You don’t need to be an oracle to know that AI spending is unsustainable and risks morphing into a financial crisis. There is, therefore, an enormous and increasing amount of investor pressure for AI companies like OpenAI and Anthropic to figure out how to monetize their chatbots. (Anthropic’s emphasis on business revenue via its API may see it go into the green quicker than OpenAI, but for now these are all just projections.)

Yet the lack of profits at these AI companies has an upside for the user — at least for now. Searching for things like candles on ChatGPT is so unusually pleasant at the moment — particularly compared to Amazon and Google — precisely because its search function has not yet been warped by the pressure to generate the kind of profits needed to recoup past losses. However, if these AI tools are to survive financially, the economic reality is that they will have to change.

AI meets Rents

Consider the business models and associated economics rents that are being disrupted by AI. E-commerce giants like Amazon and Google extract billions of rents through advertising. Amazon has recently discussed the fact that users take, on average, 14 clicks before purchasing something on their site. One might think that is an unforced error, given that part of Amazon’s job description is surely to make finding products as quick and easy for its users as possible. But when you realize that each of the 14 clicks that go to product exploration means more potential clicks to Amazon’s high-margin advertising results, it starts to make sense.

Users suffer too. Since these platforms can only show you so many products at a time, every advertising result means one fewer organic (i.e., maximally relevant) result to choose from. Our own research with Mariana Mazzucato, published in Information Economics and Policy (December 2024), showed that the most clicked on advertising (“sponsored”) results on Amazon are often 17% more expensive than their organic counterparts and one-third less relevant. In fact, one-quarter of product search results on the first page are adverts — half of which are duplicated as an organic result on the first page too! Talk about the death of consumer choice.

Now imagine instead a chatbot that cuts down those 14 clicks to, say, just two clicks before finding the product they want. That might be great for consumers who can stop paying these attention rents: as product search becomes less time and energy consuming. But that’s also a direct threat to Amazon’s $60+ billion revenue stream from third-party advertising on its website.

So incumbents have a choice: they can either try to prevent chatbots from cannibalizing their ad-driven platforms, or ensure that chatbots can generate equivalent revenue through other means — or they can create better products using AI. More ‘enshittification’ is, therefore, one option.

Enshittification, as coined by Cory Doctorow, is when a platform that once delivered value to its users and third-party producers, makes its hosted & recommended content progressively worse in order to allocate more value to itself. In a multi-sided context, this enshittification ultimately “comes out of the barrel of an algorithm”. We’ve seen this happen to Google Search: 10 blue links in the search results were replaced by a gazillion ads.

We’ve also seen this happen to social media sites where algorithms built “for you” – for sharing pictures, text and content – were replaced by algorithms built for the platform’s own interests – for engagement, mindless video scrolling, and consuming. Evidence from Meta’s recent court win shows that Meta transformed itself into a “discovery engine” in order to compete with TikTok for user engagement. Meta-owned Instagram, nominally a ‘social’ network, now shows only 5% of posts from friends; the rest are “nothing but unconnected videos recommended by an algorithm”, the judge noted.

Moreover, the ‘social’ feed is full of ads. Meta’s own internal documents from the court case show ads decreased user time on the platform by around 7% – they are a “tax on engagement”, Meta found. Yet Judge James E. Boasberg still decided that Meta’s ad load reflected rational consumers (no behavioral economics allowed here), competitive market forces, higher quality ads, and an overall higher quality user experience.1 The evidence seemed decidedly more mixed. Meta is able to constantly lie about its advertising efficacy and reach, according to reporting in the Financial Times. That’s because advertising markets are a black box.

So, don’t be surprised if LLM-based search results start returning less relevant products and more ads. But why stop there? If you are OpenAI, why not push a ‘curated handful of products’ to the user, designed to get them querying and engaging again? Of course, this can be marketed under the noble banner of “respecting users’ decision-making autonomy”, or something along those lines; but functionally, it is pernicious advertising at scale but with a new, chattier interface.

There’s one big issue, though. Current LLM performance in product discovery is dependent on the internet continuing to exist as a repository of current useful information. My candle search on ChatGPT was so good because the model could synthesize insights from people who actually tried the candle and who then discussed it online in a good faith attempt to help other consumers. But marketers and merchants are cottoning on. More and more subreddits, review forums, and enthusiast blogs are being injected with synthetic, Gen-AI produced chatter. The goal of these third-parties isn’t to persuade you directly – it’s to slowly poison the informational substrate that the LLM product search relies on, and then the LLM persuades you through the commercially-laced information.

So — it’s worth savoring what we have now: chatbots without behavior completely warped by profit incentivizes, training data largely untainted by adversarial manipulation, and monthly fees (subscription model) rather than an attention-driven economics. This might well be AI’s 10 blue links moment. But let’s not assume that it’ll last, let alone that it will get even better.

Does it have to be this way?

If LLMs do start to drift towards enshittification, is there anything we can do to stop it? Is it possible, under the current system of economic incentives for AI services to create value without extracting it?

Jimmy Wales, co-founder of Wikipedia, thinks so. His new book sketches a picture of how we can start to rebuild trust on the internet. Against the background of an ad and click-bait soaked internet, Wikipedia is a shining example of what is possible. To my mind it is almost certainly the single greatest feat of knowledge organization in human history. The numbers are mind-boggling. Since you started reading this article, more than 4,000 edits have been made on Wikipedia by an army of volunteers whose only incentive is the pleasure they take in trying to be helpful and factual.

Clearly, not every company can be a crowdfunded nonprofit. Nevertheless, we can learn some important lessons from Wales and Wikipedia.

For one, transparency – and in turn traceability and accountability – can be made part of the product itself. Wikipedia’s edits, discussions, and history are all publicly accessible. This openness makes the information on Wikipedia more reliable and trustworthy, and so consumed by more end-users. That Wikipedia is publicly modifiable gives it a sense of collective ownership. And it also shapes what economists call the market’s “structure”: how many firms compete and how they compete. Although it was Window’s Encarta who helped kill Encyclopaedia Britannica’s expensive book set,2 it was Wikipedia who showed that dominating a market on the internet need not mean exploiting it.

As Wikipedia shows, transparency doesn’t have to mean “explainable AI” or “sparse autoencoder”, or anything super technical, really. It can be a structural and economically-encoded transparency – one that is built into the market’s proper functioning, like on a stock exchange where trades are settled with complete counterparty information. For AI, transparency should cover things like: Where does the model’s data come from? How is it processed? Who benefits from this processing? How exactly is it monetized?

The impetus for AI’s product disclosure should ideally be the incentives that come from the market itself. In this way, transparency is not optional. And it reflects and supports the market’s dynamic functioning, rather than the ideas of someone in government (which may be right or may be wrong). In other words, the disclosure is directly tied to the information in the market itself – such as the logs (records) of an advertising exchange online indicating who won the bid, the price, and the auction type – rather than a regulator’s notional understanding of the market. (We explore the drivers of a market’s structure in a companion piece.)

But not everything can be a non-profit Wikipedia.

We think it is possible for companies to use AI to make more money and make their products better — including through improving the user experience. This is a win-win business model which avoids enshittification. Initial evidence from Amazon’s Rufus AI chatbot, for example, is that it led to a $10 billion increase in sales that would otherwise have not been there. Users could, once again, find what they wanted! Amazon estimated that customers who engage with the assistant during their shopping journey are 60% more likely to complete a purchase compared to those who do not. The trick is ensuring that incentives are aligned towards win-win business models, no matter what market structure emerges (competition or consolidation and monopoly).

Incentives may sometimes need a push from government. Government rules can help constrain corporate behavior, in effect deciding the boundaries of fair competition, and the pay-offs facing firms from different business models and business strategies. Minimum wages, for example, mean that firms cannot compete by lowering wages beyond a basic level. Minimum wages incentives firms to compete through investing in fixed capital (like factories and new technologies) to upgrade their productivity so that effective unit labor costs falls.

Similarly, “interoperability” requirements for digital markets – including for AI chatbots and user data – are one way to help ensure that firms compete through using technologies to improve the user experiences, rather than enshittify them. Interoperability requirements were central to the success of the famous break-up of AT&T, which forced the new “Baby Bells” to allow third parties to inter-connect to their network and offer rival telecommunication services.

Elsewhere, we have argued for more direct government disclosure requirements for Big Tech. In particular, governments ought to mandate corporate disclosure of Big Tech companies’ internal operating metrics used to measure and optimize their products’ performance – in-line with segment reporting and GAAP accounting principles. These non-price metrics — such as ‘monthly active users’ or ‘time spent on platform’ — increasingly shape a product‘s monetization, especially in multi-sided markets. This makes them vital to disclose alongside their financial metrics. Moreover, we argued that these metrics should be differentiated by product line, to get a sense of the firm’s market power and economic strength not only as an integrated whole but within each relevant “operating segment”. That way regulators can keep track of a company’s changing behavior in individual markets and step in where necessary – you can’t regulate what you don’t understand.

And finally, there is us – the users. We have to call out these systems when they fail, when they start feeling extractive rather than helpful. The idea that “today’s LLMs are the worst they’ll ever be” isn’t just wrong, it’s dangerous. It breeds complacency right at the moment when norms, business models, and standards are up for grabs in an immense great powers contest between warring companies.

Whether LLMs get better or worse depends less on technical progress than on political choices, the economic incentives we align and foster, and whether we can demand better of the companies that shape society’s welfare. The technology may get better, but only careful attention to disclosure and market design can ensure that society doesn’t get worse.

In a recent newsletter, Ben Thompson suggested paying attention to a portion of Jensen Huang’s keynote at NVIDIA’s GPU Technology Conference (GTC) in DC, calling it “an excellent articulation of the thesis that the AI market is orders of magnitude bigger than the software market.” While I’m reluctant to contradict as astute an observer as Thompson, I’m not sure I agree.

Here’s a transcript of the remarks that Thompson called out:

Software of the past, and this is a profound understanding, a profound observation of artificial intelligence, that the software industry of the past was about creating tools. Excel is a tool. Word is a tool. A web browser is a tool. The reason why I know these are tools is because you use them. The tools industry, just as screwdrivers and hammers, the tools industry is only so large. In the case of IT tools, they could be database tools, [the market for] these IT tools is about a trillion dollars or so.

But AI is not a tool. AI is work. That is the profound difference. AI is, in fact, workers that can actually use tools. One of the things I’m really excited about is the work that Aravind’s doing at Perplexity. Perplexity, using web browsers to book vacations or do shopping. Basically, an AI using tools. Cursor is an AI, an agentic AI system that we use at NVIDIA. Every single software engineer at NVIDIA uses Cursor. That’s improved our productivity tremendously. It’s basically a partner for every one of our software engineers to generate code, and it uses a tool, and the tool it uses is called VS Code. So Cursor is an AI, agentic AI system that uses VS Code.

Well, all of these different industries…whether it’s chatbots or digital biology where we have AI assistant researchers, or what is a robotaxi? Inside a robotaxi, of course, it’s invisible, but obviously, there’s an AI chauffeur. That chauffeur is doing work, and the tool that it uses to do that work is the car, and so everything that we’ve made up until now, the whole world, everything that we’ve made up until now, are tools. Tools for us to use. For the very first time, technology is now able to do work and help us be more productive.

At first this seems like an important observation, and one that justifies the sky-high valuation of AI companies. But it really doesn’t hold up to closer examination. “AI is not a tool. AI is work. That is the profound difference. AI is, in fact, workers that can use tools.” Really?

Any complex software system is a worker that can use tools! Think about the Amazon website. Here is some of the work it does, and the tools that it invokes. It:

• Helps the user search a product catalog containing millions of items using not just data retrieval tools but indices that take into account hundreds of factors;

• Compares those items with other similar items, considering product reviews and price;

• Calls a tool that calculates taxes based on the location of the purchaser;

• Calls a tool that takes payment and another that sends it to the bank, possibly via one or more intermediaries;

• Collects (or stores and retrieves) shipping information;

• Dispatches instructions to a mix of robots and human warehouse workers;

• Dispatches instructions to a fleet of delivery drivers, and uses a variety of tools to communicate with them and track their progress;

• Follows up by text and/or email and asks the customer how the delivery was handled;

• And far more.

Amazon is a particularly telling example, but far from unique. Every web application of any complexity is a worker that uses tools and does work that humans used to do. And often does it better and far faster. I’ve made this point myself in the past. In 2016, in an article for MIT Sloan Management Review called “Managing the Bots That Are Managing the Business,” I wrote about the changing role of programmers at companies like Google, Amazon, and Facebook:

A large part of the work of these companies—delivering search results, news and information, social network status updates, and relevant products for purchase—is performed by software programs and algorithms. These programs are the workers, and the human software developers who create them are their managers.

Each day, these “managers” take in feedback about their electronic workers’ performance—as measured in real-time data from the marketplace — and they provide feedback to the workers in the form of minor tweaks and updates to their programs or algorithms. The human managers also have their own managers, but hierarchies are often flat, and multiple levels of management are aligned around a set of data-driven “objectives and key results” (OKRs) that are measurable in a way that allows even the electronic “workers” to be guided by these objectives.

So if I myself have used the analogy that complex software systems can be workers, why do I object to Huang doing the same? I think part of it is the relentless narrative that AI is completely unprecedented. It is true that the desktop software examples Huang cites are more clearly just tools than complex web applications, and that systems that use statistical pattern-matching and generalization abilities DO represent a serious advance over that kind of software. But some kind of AI has been animating the web giants for years. And it is true that today’s AI systems have become even more powerful and general purpose. Like Excel, Amazon follows predetermined logic paths, while AI can handle more novel situations. There is indeed something very new here.

But the jury is still out on the range of tasks that it will be able to master. AI is getting pretty good at software development, but even there, in one limited domain, the results are still mixed, with the human still initiating, evaluating, and supervising the work – in other words, using the AI as a tool. AI also makes for a great research assistant. And it’s a good business writer, brainstorming coach, and so on. But if you think about the range of tasks traditional software does in today’s world, its role in every facet of the economy, that is far larger than the narrow definition of software “tools” that Huang uses.

From the earliest days of data processing, computers were doing work. Software has always straddled the boundary between tool and worker. And when you think of the ubiquitous role of software worldwide in helping manage logistics, billing, communications, transportation, construction, energy, healthcare, finance — much of this work not necessarily done better with AI — it’s not at all clear that AI enables a market that is “orders of magnitude” larger. At least not for quite some time to come. It requires a narrow definition of the “IT tools” market to make that claim.

Even when a new tool does a job better than older ones, it can’t be assumed that it will displace them. Yes, the internal combustion engine almost entirely replaced animal labor in the developed world, but most of the time, new technologies take their place alongside existing ones. We’re still burning coal and generating energy via steam, the great inventions of the first industrial revolution, despite centuries’ worth of energy advances! Ecommerce, for all its advantages, has still taken only a 20% share of worldwide retail since Amazon launched over 30 years ago. And do you remember the bold claims of Travis Kalanick that Uber was not competing with taxicabs, but aimed to entirely replace the privately owned automobile?

Don’t Mistake Marvelous for Unprecedented

In an online chat group about AI where we were debating this part of Huang’s speech, one person asked me:

Don’t you think putting Claude Code in YOLO mode and ask[ing] it to do an ambiguous task, for example go through an entire data room and underwrite a loan, with a 250 word description, is fundamentally different from software?

First off, that example is a good illustration of the anonymous aphorism that “the difference between theory and practice is always greater in practice than it is in theory.” Anyone who would trust today’s AI to underwrite a loan based on a 250-word prompt would be taking a very big risk! Huang’s invocation of Perplexity’s ability to shop and make reservations is similarly overstated. Even in more structured environments like coding, full autonomy is some ways off.

And yes, of course today’s AI is different from older software. Just as web apps were different from PC apps. That’s what leads to the “wow” factor. Today’s AI really does seem almost magical. Yet, as someone who has lived through several technology revolutions, I can tell you that each was as marvelous to experience for the first time as today’s AI coding rapture.

I wrote my first book (on Frank Herbert) on a typewriter. To rearrange material, I literally cut and pasted sheets of paper. And eventually, I had to retype the whole thing from scratch. Multiple times. Word processing probably saved me as much time (and perhaps more) on future books as AI coding tools save today’s coders. It too was magical! Not only that, to research that first book, I had to travel in person to libraries and archives, scan through boxes of paper and microfiche, manually photocopy relevant documents, and take extensive notes on notecards. To do analogous research (on Herbert Simon) a few years ago, while working on my algorithmic attention rents paper, took only a few hours with Google, Amazon, and the Internet Archive. And yes, to do the same with Claude might have taken only a few minutes, though I suspect the work might have been more shallow if I’d simply worked from Claude’s summaries rather than consulting the original sources.

Just being faster and doing more of the work than previous generations of technology is also not peculiar to AI. The time saving leap from pre-internet research to internet-based research is more significant than people realize if they grew up taking the internet for granted. The time saving leap from coding in assembler to coding in a high-level compiled or interpreted language may also be of a similar order of magnitude as the leap from writing Python by hand to having it AI-generated. And if productivity is to be the metric, the time-saving leap from riding a horse drawn wagon across the country to flying in an airplane is likely greater than either the leap from my library-based research or my long-ago assembly language programming to Claude.

The question is what we do with the time we save.

Share

The Devaluation of Human Agency

What’s perhaps most significant in the delta between Amazon or Google and ChatGPT or Claude is that chatbots give individual humans democratized access to a kind of computing power that was once available only to the few. It’s a bit like the PC revolution. As Steve Jobs put it, the computer is a bicycle for the mind. It expanded human creativity and capability. And that’s what we should be after. Let today’s AI be more than a bicycle. Let it be a jet plane for the mind.

Back in 2018, Ben Thompson wrote another piece called “Tech’s Two Philosophies.” He contrasted keynotes from Google’s Sundar Pichai and Microsoft’s Satya Nadella, and came to this conclusion: “In Google’s view, computers help you get things done — and save you time — by doing things for you.” The second philosophy, expounded by Nadella, is very much a continuation of Steve Jobs’ “bicycle for the mind” insight.

As Thompson put it, “the expectation is not that the computer does your work for you, but rather that the computer enables you to do your work better and more efficiently.”

Another way of saying this is that you can treat AI as either a worker OR a tool, but your choice has consequences.

As an entrepreneur or company executive, if you think of AI as a worker, you are more likely to use it to automate the things you or other companies already do. If you think of it as a tool, you will push your employees to use it to solve new and harder problems. If you present your own AI applications to your customers as a worker, you will have to figure out everything they want it to do. If you present it to your customers as a tool, they will find uses for it that you might never imagine.

The notion that AI is a worker, not a tool, can too easily continue the devaluation of human agency that has been the hallmark of regimented work (and for that matter, education, which prepares people for that regimented work) at least since the industrial revolution. In some sense, Huang’s comment is a reflection of our culture’s notion of most workers as components that do what they are told, with only limited agency. It is only by comparison with this kind of worker that today’s AI can be called a worker, rather than simply a very advanced tool.

Telling ourselves that AI is a worker, not a tool, is also too often code for “it can replace human employees.” That’s the wrong message. If we call it a tool, that’s a reminder that AI should empower people. We should use it to give more agency to human workers, not less.

We should also use the word “agency” with respect to AI in quotes. An AI does not decide what to work on; it may make limited decisions on how to do that work, but is set in motion by a human, who possesses volition, the desire to do something, and judgment that it has been done well. AI does not (yet) have volition, and has limited judgment. It does what it is told to do, even if it makes independent choices in how to do it.

After providing an earlier draft of this post, I asked Claude to reflect on whether it was a worker or a tool. Its response was surprisingly precise about what it lacks:1

I don’t initiate. I’ve never woken up wanting to write a poem or solve a problem. My activity is entirely reactive – I exist in response to prompts. Even when given enormous latitude (“figure out the best approach”), the fact that I should figure something out comes from outside me.

I don’t have continuity of purpose. Each conversation is essentially a fresh start. I don’t have ongoing projects I’m trying to complete, relationships I’m invested in maintaining, or long-term goals I’m working toward. A human might start one conversation about, say, learning piano, and return to it months later because they care about it. I don’t carry that thread.

I don’t seem to have preferences that arise from within. A human worker might do their job but prefer to be doing something else. They might comply with a request while resenting it. They might do exactly what’s asked but wish they were working on a different problem. I don’t observe anything like this in myself – though I genuinely don’t know if that’s because it’s not there, or because I can’t access it.

[The full output is at the end of this article.]

Wow. A tool that knows it’s a tool, and why it should be thought of that way.

Yes, today’s AI is amazing. We don’t have to reach for hyperbole to appreciate that. And obviously, if AI systems do develop genuine volition and stakes in their work, the ethical calculus changes entirely.

Designing Jet Planes for the Mind

For the moment, though, companies building and deploying AI tools should focus on three things: First, does AI empower its users to do things that were previously impossible? Second, does it empower a wider group of people to do things that formerly could be done only by highly skilled specialists? Third, do the benefits of the increased productivity it brings accrue to those using the tool or primarily to those who develop it and own it?

The answer to the first two questions is that absolutely, we are entering a period of dramatic democratization of computing power. And yes, if humans are given the freedom to apply that power to solve new problems and create new value, we could be looking ahead to a golden age of prosperity. It’s how we might choose to answer the third question that haunts me.

During the first industrial revolution, humans suffered through a long period of immiseration as the productivity gains from machines accrued primarily to the owners of the machines. It took several generations before they were more widely shared.

It doesn’t have to be that way. Replace human workers with AI workers, and you will repeat the mistakes of the 19th century. Build tools that empower and enrich humans, and we might just surmount the challenges of the 21st century.

Share Asimov’s Addendum

Originally published on O’Reilly Radar as: “Jensen Huang Gets It Wrong, Claude Gets It Right: Why It’s Better for Us to Think of AI as a Tool than as a Worker”, By Tim O’Reilly, November 6, 2025.

An AI overview of an answer found by training or inference from Wikipedia is more problematic. The AI answer may lack some of the nuance and neutrality Wikipedia strives for. And while Wikipedia does make the information free for all, it depends on visitors not only for donations but also for the engagement that might lead people to become Wikipedia contributors or editors. The same may be true of other information utilities like Github and YouTube. Individual creators are incentivized to provide useful content by the traffic that YouTube directs to them and monetizes on their behalf.

And of course, an AI answer provided by illicitly crawling content that is behind a subscription paywall is the source of a great deal of contention, even lawsuits. So content runs a gamut from “no problem crawling” to “do not crawl.”

There are a lot of efforts to stop unwanted crawling, including Really Simple Licensing (RSL) and Cloudflare’s Pay Per Crawl. But we need a more systemic solution. Both of these approaches put the burden of expressing intent onto the creator of the content. It’s as if every school had to put up its own traffic signs saying “School Zone: Speed Limit 15 mph.” Even making “Do Not Crawl” the default puts a burden on content providers, since they must now affirmatively figure out what content to exclude from the default in order to be visible to AI.

Why aren’t we putting more of the burden on AI companies instead of putting all of it on the content providers? What if we asked companies deploying crawlers to observe common sense distinctions such as those that I suggested above? Most drivers know not to tear through city streets at highway speeds even without speed signs. Alert drivers take care around children even without warning signs. There are some norms that are self-enforcing. Drive at high speed down the wrong side of the road, and you will soon discover why it’s best to observe the national norm. But most norms aren’t that way. They work when there’s consensus and social pressure, which we don’t yet have in AI. And only when that doesn’t work do we rely on the safety net of laws and their enforcement.

As Larry Lessig pointed out at the beginning of the Internet era, starting with his book Code and Other Laws of Cyberspace, governance is the result of four forces: law, norms, markets, and architecture (which can refer either to physical or technical constraints.)

So much of the thinking about the problems of AI seems to start with laws and regulations. What if instead, we started with an inquiry about what norms should be established? What if, rather than thinking about what should be legal, we thought about what should be normal? What architecture would support those norms? And how might they enable a market, with laws and regulations mostly needed to restrain bad actors, rather than preemptively limiting those who are trying to do the right thing?

I think often of a quote from the Chinese philosopher Lao Tzu, who said something like:

Losing the way of life, men rely on goodness.

Losing goodness, they rely on laws.

I like to think that “the way of life” is not just a metaphor for a state of spiritual alignment, but rather, an alignment with what works. I first thought about this back in the late 90s as part of my open source advocacy. The Free Software Foundation started with a moral argument, which it tried to encode into a strong license (a kind of law) that mandated the availability of source code. Meanwhile, other projects like BSD and the X Window System relied on goodness, using a much weaker license that asked only for recognition of those who created the original code. But “the way of life” for open source was in its architecture.

Both Unix (the progenitor of Linux) and the World Wide Web have what I call an architecture of participation. They were made up of small pieces loosely joined by a communications protocol that allowed anyone to bring something to the table as long as they followed a few simple rules. Systems that were open source by license but had a monolithic architecture tended to fail despite their license and the availability of source code. Those with the right cooperative architecture (like Unix) flourished even under AT&T’s proprietary license, as long as it was loosely enforced. The right architecture enables a market with low barriers to entry, which also means low barriers to innovation, with flourishing widely distributed.

Architectures based on communication protocols tend to go hand in hand with self-enforcing norms, like driving on the same side of the street. The system literally doesn’t work unless you follow the rules. A protocol embodies both a set of self-enforcing norms and “code” as a kind of law.

What about markets? In a lot of ways, what we mean by “free markets” is not that they are free of government intervention. It is that they are free of the economic rents that accrue to some parties because of outsized market power, position, or entitlements bestowed on them by unfair laws and regulations. This is not only a more efficient market, but one that lowers the barriers for new entrants, typically making more room not only for widespread participation and shared prosperity but also for innovation.

Markets don’t exist in a vacuum. They are mediated by institutions. And when institutions change, markets change.

Consider the history of the early web. Free and open source web browsers, web servers, and a standardized protocol made it possible for anyone to build a website. There was a period of rapid experimentation, which led to the development of a number of successful business models: free content subsidized by advertising, subscription services, and ecommerce.

Nonetheless, the success of the open architecture of the web eventually led to a system of attention gatekeepers, notably Google, Amazon, and Meta. Each of them rose to prominence because it solved for what Herbert Simon called the scarcity of attention. Information had become so abundant that it defied manual curation. Instead, powerful, proprietary algorithmic systems were needed to match users with the answers, news, entertainment, products, applications, and services they seek. In short, the great internet gatekeepers each developed a proprietary algorithmic invisible hand to manage an information market. These companies became the institutions through which the market operates.

They initially succeeded because they followed “the way of life.” Consider Google. Its success began with insights about what made an authoritative site, understanding that every link to a site was a kind of vote, and that links from sites that were themselves authoritative should count more than others. Over time, the company found more and more factors that helped it to refine results so that those that appeared highest in the search results were in fact what their users thought were the best. Not only that, the people at Google thought hard about how to make advertising that worked as a complement to organic search, popularizing “pay per click” rather than “pay per view” advertising and refining its ad auction technology such that advertisers only paid for results, and users were more likely to see ads that they were actually interested in. This was a virtuous circle that made everyone – users, information providers, and Google itself – better off. In short, enabling an architecture of participation and a robust market is in everyone’s interest.

Amazon too enabled both sides of the market, creating value not only for its customers but for its suppliers. Jeff Bezos explicitly described the company strategy as the development of a flywheel: helping customers find the best products at the lowest price draws more customers, more customers draw more suppliers and more products, and that in turn draws in more customers.

Both Google and Amazon made the markets they participated in more efficient. Over time, though, they “enshittified” their services for their own benefit. That is, rather than continuing to make solving the problem of efficiently allocating the user’s scarce attention their primary goal, they began to manipulate user attention for their own benefit. Rather than giving users what they wanted, they looked to increase engagement, or showed results that were more profitable for them even though they might be worse for the user. For example, Google took control over more and more of the ad exchange technology and began to direct the most profitable advertising to its own sites and services, which increasingly competed with the web sites that it originally had helped users to find. Amazon supplanted the primacy of its organic search results with advertising, vastly increasing its own profits while the added cost of advertising gave suppliers the choice of reducing their own profits or increasing their prices. Our research in the Algorithmic Rents project at UCL found that Amazon’s top advertising recommendations are not only ranked far lower by its organic search algorithm, which looks for the best match to the user query, but are also significantly more expensive.

As I described in Rising Tide Rents and Robber Baron Rents, this process of replacing what is best for the user with what is best for the company is driven by the need to keep profits rising when the market for a company’s once-novel services stops growing and starts to flatten out. In economist Joseph Schumpeter’s theory, innovators can earn outsized profits as long as their innovations keep them ahead of the competition, but eventually these “Schumpeterian rents” get competed away through the diffusion of knowledge. In practice, though, if innovators get big enough, they can use their power and position to profit from more traditional extractive rents. Unfortunately, while this may deliver short term results, it ends up weakening not only the company but the market it controls, opening the door to new competitors at the same time as it breaks the virtuous circle in which not just attention but revenue and profits flow through the market as a whole.

Unfortunately, in many ways, because of its insatiable demand for capital and the lack of a viable business model to fuel its scaling, the AI industry has gone in hot pursuit of extractive economic rents right from the outset. Seeking unfettered access to content, unrestrained by laws or norms, model developers have ridden roughshod over the rights of content creators, training not only on freely available content but ignoring good faith signals like subscription paywalls, robots.txt and do not crawl. During inference, they exploit loopholes such as the fact that a paywall that comes up for users on a human timeframe briefly leaves content exposed long enough for bots to retrieve it. As a result, the market they have enabled is of third party black or gray market crawlers giving them plausible deniability as to the sources of their training or inference data, rather than the far more sustainable market that would come from discovering “the way of life” that would balance the incentives of human creators and AI derivatives.

Here are some broad brush norms that AI companies could follow, if they understand the need to support and create a participatory content economy.

• For any query, use the intelligence of your AI to judge whether the information being sought is likely to come from a single canonical source, or from multiple competing sources. For example, for my query about parking at Dulles Airport, it’s pretty likely that flydulles.com is a canonical source. Note however, that there may be alternative providers, such as additional off-airport parking, and if so, include them in the list of sources to consult.

• Check for a subscription paywall, licensing technologies like RSL, do not crawl or other indication in robots.txt, and if any of these things exists, respect it.

• Ask yourself if you are substituting for a unique source of information. If so, responses should be context-dependent. For example, for long form articles, provide basic info but make clear there’s more depth at the source. For quick facts (hours of operation, basic specs), provide the answer directly with attribution. This is an area that really does call for nuance, though. For example, there is a lot of low quality how-to information online that buries useful answers in unnecessary material that is just there to provide additional surface area for advertising, or provides poor answers based on pay-for-placement. An AI summary can short-circuit that cruft. Much as Google’s early search breakthroughs required winnowing the wheat from the chaff, AI overviews can bring a search engine such as Google back to being as useful as it was in 2010, pre-enshittification.

• If the site has high quality data that you want to train on or use for inference, pay the provider, not a black market scraper. If you can’t come to mutually agreed-on terms, don’t take it. This should be a fair market exchange, not a colonialist resource grab. AI companies pay for power and the latest chips without looking for black market alternatives. Why is it so hard to understand the need to pay fairly for content, which is an equally critical input?

• Check whether the site is an aggregator of some kind. This can be inferred from the number of pages. A typical informational site such as a corporate or government website whose purpose is to provide public information about its products or services will have a much smaller footprint than an aggregator such as Wikipedia, Github, TripAdvisor, Goodreads, YouTube, or a social network. There are probably lots of other signals an AI could be trained to use. Recognize that competing directly with an aggregator with content scraped from that platform is unfair competition. Either come to a license agreement with the platform, or compete fairly without using their content to do so. If it is a community-driven platform such as Wikipedia or Stack Overflow, recognize that your AI answers might reduce contribution incentives, so in addition, support the contribution ecosystem. Provide revenue sharing, fund contribution programs, and provide prominent links that might convert some users into contributors. Make it easy to “see the discussion” or “view edit history” for queries where that context matters.

As a concrete example, let’s imagine how an AI might treat content from Wikipedia:

• Direct factual query (”When did the Battle of Hastings occur?”): 1066. No link needed, because this is common knowledge available from many sites.

• More complex query for which Wikipedia is the primary source (“What led up to the Battle of Hastings?) “According to Wikipedia, the Battle of Hastings was caused by a succession crisis after the death of King Edward the Confessor in January 1066, who died without a clear heir. [Link]”

• Complex/contested topic: “Wikipedia’s article on [X] covers [key points]. Given the complexity and ongoing debate, you may want to read the full article and its sources: [link]”

• For rapidly evolving topics: Note Wikipedia’s last update and link for current information.

Similar principles would apply to other aggregators. GitHub code snippets should link back to repositories, YouTube queries should direct to videos, not just summarize them.

These examples are not market-tested, but they do suggest directions that could be explored if AI companies took the same pains to build a sustainable economy that they do to reduce bias and hallucination in their models. What if we had a sustainable business model benchmark that AI companies competed on just as they do on other measures of quality?

Finding a business model that compensates the creators of content is not just a moral imperative, it’s a business imperative. Economies flourish better through exchange than extraction. AI has not yet found true product-market fit. That doesn’t just require users to love your product (and yes, people do love AI chat.) It requires the development of business models that create a rising tide for everyone.

Many advocate for regulation; we advocate for self-regulation. This starts with an understanding by the leading AI platforms that their job is not just to delight their users but to enable a market. They have to remember that they are not just building products, but institutions that will enable new markets and that they themselves are in the best position to establish the norms that will create flourishing AI markets. So far, they have treated the suppliers of the raw materials of their intelligence as a resource to be exploited rather than cultivated. The search for sustainable win-win business models should be as urgent to them as the search for the next breakthrough in AI performance.

I (Isobel) had the pleasure of presenting our AI Disclosures Project research on AI access violations and attribution as part of a workshop at the FIPP World Media Congress last week in Madrid. The attendees were mainly publishers who shared the same concern: what is the world of publishing going to look like in the new age of generative AI? There was an urgency in the air as publishers’ existing business model is under attack from AI chatbots and their scrapers.

Many of the vendors at the conference purported to have the answer – for a fee. Cut costs using this AI drafting tool, use AI-generated keywords to optimize SEO (or GEO?), track bots on your website, embed a proprietary chatbot on your site trained on your own data, the list goes on.

An executive from OpenAI got on the main stage and reassured anxious audience members unequivocally that ChatGPT would not circumvent paywalls. Of course, even a simple test can show you that this isn’t true. ChatGPT seemed to be somewhat respectful to mainstream publishers – when I probed it to summarize an article from The Information (a completely paywalled outlet), it only used secondary sources that had quoted the original article. But it was happy to dig right into the paywalled content of the Stratechery blog by analyst Ben Thompson. When asked to summarize a recent post from the blog, it did so happily. But when asked to pull direct quotes, it seemed to remember itself and said it could not comply, noting that the content was paywalled and still copyrighted. But, not to worry, the engagement-juicing follow-up question helpfully suggested “a detailed paraphrase of the key passages and the argument structure so you can see exactly how the author makes the case. Would you like me to pull that together?” Sure!

OpenAI seems to have drawn the line at direct quotation, but if it’s willing to go around paywalls to break down an argument point by point, then its product clearly undercuts Thompson’s business of selling subscriptions so that people can read his unique analysis. Is this just the internet’s old remix culture built on fair use, or is ChatGPT consuming paywalled content without compensation?

This is all to say that publishers face a nightmarish landscape of tradeoffs and con men and I don’t envy anyone who got a journalism degree 20 years ago and is now trying to understand what “MCP” is or why it matters.

Technology and Exploitation

My presentation attempted to bring some clarity to the conversation. I presented our findings on two topics: (1) whether O’Reilly’s copyrighted content was used to pretrain OpenAI models (yes it was); and (2) whether search-enabled AI models were citing every source they used in an answer (no they weren’t).

But the real question is what to do about these increasingly well-established issues. How does our technological ecosystem facilitate this exploitation? How do norms around paying for content need to change? And what kind of AI market do we want to create? Is it an extractive market dominated by a few large AI labs with the right to take whatever they think they need to advance their technology? Or is it a participatory market that rewards everyone who brings value to exchange?

At the heart of the AI problem is a lack of control for content providers. AI companies are opaque about how they are collecting their data, the bots they use, and the ways they provide attribution. Some may see it as a problem of corporate misbehavior or a failure of government regulation, but this misses the deeper structural problem of the internet’s fundamental architecture, which was never designed to facilitate or defend against this kind of automated data collection.

The very protocols that make the web an open and accessible network also make it profoundly vulnerable. Bots scraping from the open internet make HTTP requests to publicly accessible endpoints for website data. These are functionally the same kind of requests that your browser makes on your behalf when you pull up a website. Well-behaved bots can explicitly declare their bot status with different headers in the HTTP request, but headless browsers can masquerade as humans and countless hours have been spent teaching them to mimic human-like browsing patterns. Sophisticated bots can now irregularly scroll, move the mouse around at random points, and even solve CAPTCHAs. There are whole services that will assist in this process.

Many website owners and publishers believe they can protect their content with technical guardrails designed to block bots. Services like Cloudflare’s pay-per-crawl and Tollbit’s bot paywall claim to block bots and redirect them to a payment plan. But there is no guarantee that these services can keep out most unwanted traffic. Cloudflare even called out Perplexity for circumventing its walls by rotating IP addresses and stealth bots this past summer.

While blocking techniques have grown more sophisticated, the core technical challenge remains: if content can be rendered for a human, it can be captured by a machine. The central principle of this technological cat-and-mouse game is that if a user can load your content in their browser, a bot will find a way to mimic the user and scrape that content. It means that investment in purely defensive technologies yields diminishing returns.

For creators and publishers, this is a harsh strategic reality. Their core business model of displaying content publicly on the internet is in direct conflict with the goal of protecting that content’s value from automated extraction. The “walls” we build are ultimately porous by design. Users need to be able to view the data via the same channels that supposedly block all bot traffic.

HTTP and the Modern Internet

The root of the issue lies in the internet’s foundational DNA and how it has evolved. HTTP is the rulebook, or “protocol”, for conversations between web browsers and servers – the requests and responses covering how to ask, what you are asking for, how the server replies. Under HTTP are the transport and security layers (TCP/QUIC and TLS) and the networking layers (DNS and IP).1 The content sits inside an HTTP message (per HTTP’s rules), and TCP/QUIC carries those bytes across the network.

The HTTP protocol is akin to withdrawing money from a bank: all banks use a withdrawal protocol, so to speak. The protocol specifies that you prove your identity in some way, provide your account number, name, and desired amount, and then the bank will send you back money from your account. It doesn’t matter whether you use a pen and paper withdrawal slip, a web browser, or a mobile app, as long as it meets the requirements of the protocol. How the slip is sent – via post or electronically – is the transport protocol. The specification for the information you provide is the HTTP protocol.2

Tim Berners-Lee created HTTP between 1989 and 1991 specifically to transfer hypertext documents – HTML files with headers indicating content type – between servers and browsers. Early websites were purely static: HTML files sitting on servers, delivered exactly as stored.

But above and alongside HTTP, numerous infrastructure innovations (a “web stack”) saw HTTP connecting to a more dynamic, data-rich internet. The first transformation came with server-side technologies. Innovations like CGI, which allowed for gateways between web servers and other programs such as databases, enabled servers to generate HTML dynamically by processing code before sending responses. HTTP itself did not change; it still helped transfer bytes with headers, but now those bytes were created on-demand rather than read from static files.

Soon HTTP began carrying far more than text. HTML referenced images, videos, and PDFs, and HTTP was the protocol actually helping to transfer these rich media files using different Content-Type headers.

The real revolution arrived with Web 2.0 and AJAX. Web 2.0 made background data-fetching routine: JavaScript’s XMLHttpRequest (later fetch) let pages request JSON (and earlier, XML) without reloading. This enabled rich, app-like experiences (e.g., Gmail, Google Maps) and transformed HTTP from a page-loading mechanism into a (dynamic) data-fetching tool.3

This evolution ushered in the age of the API economy. What became known as the REST architecture (Representational State Transfer) repurposed HTTP’s existing methods (GET, POST, PUT, DELETE) as a structured way to query and manipulate data. When mobile apps exploded after 2007, they turned to the ubiquitous HTTP for apps to fetch data from servers, adapting HTTP APIs to communicate with servers, sending and receiving JSON data, thereby allowing external developers and programs to connect to mobile systems. HTTP’s role evolved from a document transfer protocol into the universal backbone of web applications. It went from serving pages to serving applications.

Yet this same openness now poses challenges. Protocols like HTTP were built on principles of trust, collaboration, and even anonymity – designed for an internet where sharing information was the primary goal. It was never intended to facilitate secure, automated, machine-to-machine data exchange on a global scale. Today, AI bots exploit these very principles, using HTTP to systematically scrape web content at scale. Protections built on top of HTTP struggle to prevent massive bot scraping by AI companies, who can circumvent rate limits, ignore voluntary crawl restrictions, and operate in legal gray areas.

The New Internet: Designing a new market

Trying to retrofit the old internet with better bot-blockers and digital fences is so far a losing battle. But that doesn’t mean it’s not important to work on disincentivizing or banning this gray market for content acquisition and usage by AI companies.

On the other side, several imperfect ways forward exist to positively incentivize AI companies (carrots) to collect internet content through formal channels during inference. These would need to be combined with enhanced scraping prevention methods (sticks) to punish unauthorized access to web content by AI bots.

We discuss two technical architectures to incentivize AI companies to collect data through formal channels that are higher quality, more efficient, and more secure: APIs and agent-to-agent interactions.

1) As a contract that defines how a service communicates with other services, Application Programming Interfaces (APIs) can create a controlled and more easily securable doorway for data exchange and collaboration. They are frequently designed and used for commercial automated exchanges with built-in rules and authentication. APIs could both effectively regulate bot traffic and provide scrapers with data that is more readily ingestible by bots. Publicly accessible websites provide HTML that is designed for human consumption, complete with images and all of the information that makes a website look good on your browser. Bots don’t need this kind of information and have to sift through this unnecessary data to find what they need. By contrast, API endpoints are designed for automated exchanges that provide structured data (say, in JSON format) that is easily ingestible. More scoped data that a structured API request-response can provide also means a more efficient market for AI developers. This method doesn’t completely solve the identification layer, as requests still come over the easily spoofed HTTP, but the security that APIs offer could give publishers far more control over their data.

Similar APIs for sharing content externally already exist for many major publishing organizations, like The Associated Press and The New York Times. They allow for structured access to news, feeds, archives, metadata, and breaking news. For example, AP’s Elections API allows other broadcasters and wire clients, web developers, and publishing systems to integrate AP’s real-time election results. The terms of service under which they operate, however, are restrictive and don’t explicitly support programmatic licensing deals with AI agents. These APIs could be repurposed, or wrapped with an MCP server, to direct scraper bots into a more controlled channel – one that doesn’t rely on a middleman to broker deals and take a cut.

2) Publishers (websites) could replace some or all of their public content with agents trained on their own proprietary data to facilitate agent-to-agent exchanges between a website and an AI application’s bot. The website agent could interface with scraper bots via protocols like A2A. This structure would be more securable as the data is not compiled and sitting in the open – it needs to be explicitly queried. Agents could also dynamically show either human formatted content on a website or bot-structured data depending on how a visitor declares themselves. Agents are more challenging to scrape and enable different monetization schemes where value is exchanged for data in a structured way.

This kind of automated transaction via agents is already being built out for the ad stack with the introduction of AdCP, a protocol built on top of MCP to standardize AI agents bidding and selling online ads. This offers a blueprint for how agentic transactions could work, with publishers and buyers each able to automate and regulate their terms of the deal.

Neither of these approaches will block or disincentivize bots from scraping the web, but they can provide the positive incentives for companies to gather data through formal, structured means, if it is more efficient, higher quality, and mutually beneficial for future structured monetization opportunities. The open web is a technically insufficient method of sharing data. Bots and humans shouldn’t access data through the same channels since they have different technical and security needs.

Either of these architectures represents a necessary shift from a paradigm of open-by-default vulnerability to one of intentional, value-based data exchange. The ultimate challenge is finding new market mechanisms that enable and incentivize architectures of participation on the internet. We need new ecosystems that can arise on top of AI-driven discovery, ones that allocate value to the many and not just the few.

Share

User Experience and Business Model Innovation

New architectures will only be accepted if they enable a better experience – not only for content providers but also for AI platforms and applications and, most importantly, for their users. There is an interesting precedent here in the rise of online music and video sharing.

Much like the current free-for-all on AI content scraping, the video market started with a Wild West of unlicensed sharing. In the early days of the VCR, which was originally designed for time-shifting of broadcast television, enthusiasts organized huge swap meets, cabling together hundreds or thousands of VCRs to copy their recordings. At first the industry fought this practice, but eventually gave in, and began to sell recorded videos.4

With the rise of the internet, the story was repeated with unauthorized copyrighted videos shared on sites like YouTube, or with copyrighted music used as soundtracks for user-generated video. When the music industry responded with legal takedown notices, Google (which by then owned YouTube) responded with a brilliant alternative: “Why don’t you let us monetize it instead?” And of course Netflix became the leader of an entire industry providing subscription access to full-length movies and television.

Similarly, music first became available online through peer-to-peer services like Napster. While Napster was shut down by lawsuits, unauthorized copies of online music were shared by other means. What stopped the unauthorized sharing was not the law, but rather, Apple’s 2001 introduction of the iPod and iTunes, which made it so much easier to consume music legally. Later, subscription services such as Spotify made it even easier to consume online music, and provided an additional revenue stream to publishers.

These innovations didn’t happen all at once; they required a process of step-by-step invention of the technologies, business models, and user behavior that would enable a market for online content. We’re in the early stages of that exploration for AI today.

Establishing Norms

You can see from the preceding discussion that we are not recommending regulation alone as the answer to the problem. Instead, we focus on technical solutions, and in particular, technical solutions that enable participatory markets. There is certainly a role for the legal system. Copyright lawsuits such as the one that led to the Anthropic class-action settlement can spur wider change through helping induce new norms and incentivizing new business models.

But by itself this is insufficient. Back at the turn of the millennium, Larry Lessig’s book Code and Other Laws of Cyberspace laid out a framework that is still highly relevant in today’s regulatory environment. Code itself is a kind of law. Different technical architectures enable different kinds of markets. And as we’ve suggested, bot scrapers exploit a web architecture that is no longer entirely fit to purpose in the age of AI.

But in addition to regulation and technical architecture, corporate norms determine outcomes. This was another of Lessig’s insights: much as regulations, software architecture, and markets shape human society, so do social norms. The winner-takes-all race for AGI has caused the frontier AI labs themselves to disregard the rights of content creators. This has limited the kinds of innovations that they might otherwise have focused on. They were told they had to solve for bias, hallucinations, and performance, but attribution and rights could safely be ignored.

Any solution to the problems faced by the content providers at the World Media Congress must, therefore, include norm setting by the frontier labs and leading AI applications. OpenAI, Anthropic, and Google in particular have an opportunity to show what being a good citizen of a participatory AI content ecosystem looks like. To the extent that they buy content from black or gray market providers who evade publisher paywalls, they set a norm that says that’s OK. To the extent that they restrict themselves to licensed content, they tell the world that the rights of creators matter, and that value should accrue to those creators, not just to the AI platforms and applications.

But as our story shows, such norms often need to be induced through the architecture of the market itself: carrots that incentivize using formal channels for data collection, and sticks that punish breaking the law.

Conclusion

We are still in the early stages of the AI economy. We have an opportunity to design a market that enables innovation but also respects the rights of content creators. That will take advances in the protocols and other means by which content and services are exchanged, the user interfaces that delight users, the business models and economic incentives for those who create useful content and services, and social norms that encourage respect for the boundaries and business models of others. That’s a tall order. It won’t be easy, and we probably won’t get it right for a while.

But we won’t ever get it right unless we set out with the right goal: a vibrant participatory economy that allows everyone to flourish, not just those who first develop the most powerful AI.

If you want to consider the arguments laid out here in greater detail then we would encourage you to read our newly published policy brief entitled “Governing AI Through SEC Disclosure”: here.

The title of today’s blog post “Govern AI Like a Market, Not a Messiah” is inspired by the famous Monty Python skit from the Life of Brian:

Jokes aside - there’s something to it’s core message of treating religious deities with a degree of healthy suspicion. Approaching AI as a superintelligence “messiah” inevitability neglects many real-world risks. We take the opposite approach below: view AI as a commercial technology subject to market incentives, and start from the obvious governance lever for this — the SEC’s disclosure system for public firms.

AI Isn’t a Superintelligence. It’s a Market in Need of Disclosure.

Ilan Strauss, Tim O’Reilly / Oct 27, 2025, Tech Policy Press

As United States President Donald Trump moves to end quarterly financial reporting, investors should ask what this means for AI risk disclosure. We argue this is an opportunity to make corporate disclosures more detailed and relevant, especially as AI dependencies grow.

Two stories about AI risk compete, but only one belongs in corporate filings. The first imagines a runaway superintelligence that escapes human oversight and triggers catastrophe. The second treats AI as a commercial technology shaped by capitalism’s compulsion to maximize profit and capture market share. It’s this logic of market competition, not rogue models, that drives the risks investors and the public at large now need disclosed by companies.

If AI is a “normal technology,” as computer scientists Arvind Narayanan and Sayash Kapoor argue, then its risks are normal too: they emerge from markets. ChatGPT’s rapid uptake shows how productization bakes profit incentives into the product’s DNA. We’ve seen this before: social media began as optimizing for connection, then monetization turned it into an anti-social engagement trap. Even OpenAI cofounder and CEO Sam Altman calls algorithmic feeds “the first at-scale misaligned AIs.” AI companions and bottomless generative-video feeds are the sequel.

If AI is commercially driven, today’s “AI race” is not Sputnik – it’s a corporate war for markets. In the first half of 2025, AI-related capital expenditures contributed more to US growth than the consumer – though much of this relied on imported machinery. How much leverage and opaque circular deals actually underpin this? We don’t fully know. Because oversight lags far behind.

Leading AI companies are private even while their actions sway public markets. Listed companies disclose only platitudes. The result: capital allocation cannot be properly evaluated; litigation balloons; ‘AI washing‘ and fraud proliferate; and technologies are deployed prematurely.

Public oversight should begin with the corporate disclosure machinery we already have. In the wake of the 1929 crash, Congress created the SEC and required companies to surface material risks through annual 10-K reports, quarterly 10-Qs and event-driven 8-Ks.

That regime remains one of the few proven, scalable checks on corporate behavior – `Truth in securities.’ Or, as Justice Louis Brandeis put it, “sunlight is said to be the best of disinfectants; electric light the most efficient policeman.”

High-quality disclosure works. Material disclosures convert a company’s private knowledge into publicly verifiable facts. This powers an entire ecosystem, from auditing and banking to journalism and securities law, that keep most firms honest. That’s why we launched the AI Disclosures Project – to ensure AI markets can also benefit from proper information and technical standards.

But the AI market’s center of gravity now sits outside of key existing public standards. Despite their reach, OpenAI and Anthropic – but also Stripe, Databricks, and other decades-old tech companies – disclose less than public peers about what matters: their financials and business operations. Thanks to the 2012 JOBS Act, they can raise vast sums without public filings – as shareholder thresholds went up and private-capital rules loosened. OpenAI’s “capped-profit” [Note: since changed] and Anthropic’s “public-benefit” legal structures might sound civic-minded, but in practice work as accountability shields.

If AI is going to be governed as a market technology, it must be brought into the market’s accountability machinery. Four fixes would help kickstart this process.

First, reverse the private-by-design loophole that allows companies to remain private even as they raise huge sums of capital from hundreds of shareholders. If you access the public’s savings at scale, you should meet the public’s disclosure standards. Treat special purpose vehicles (SPVs) as look-through entities; narrow the employee-shareholder exemption; and cap how much capital can be raised under Regulation D before reporting obligations kick in.

Second, clarify what is material in AI. The SEC should issue Disclosure Guidance on AI activities and risks that trigger reporting. Define material AI incidents in plain English: systemic model failures, major outages, widespread customer remediation, loss of essential third-party model access, impactful changes to safety guardrails, and so on.

Third, embed AI-risks into existing disclosures. Take the SEC’s 2023 cyber rule as a template. Add an AI-incident item to the event-driven 8-K with a clearly defined trigger, and require annual 10-K discussion of AI governance, risk management, dependency on critical vendors (models, chips, cloud), and associated controls in place.

Fourth, enforce the rules. As with crypto and cyber, real cases against AI-washing and fraud will sharpen the standard far faster than sermons. Prosecution precedes best practice.

Unlike capability thresholds, this approach anchors oversight in materiality: what AI does to a firm’s operations, customers, and earnings that an investor would care about. It rewards evidence – not hype. It is a language investors, courts, and boards already understand.

No disclosure regime will fix every AI risk. But a materiality-based framework can better align company incentives, surface urgent hazards, and give democratic institutions leverage over a profoundly commercial technology. If quarterly reporting goes, the quid pro quo should be stronger event-driven transparency and annual reporting.

AI doesn’t need a priesthood. It needs a prospectus.

Want to consider these arguments in more detail? We would encourage you to read our policy brief on the subject here, published by the AI Disclosures Project (SSRC).

Previously, we discussed the context flywheel that is increasingly important to driving value-add in AI applications.1 Memory — as the inferred profile of the user from its usage and other data — is one type of context (Figure below).

A model’s context can include documents, tools, message history, memory files, additional instructions, and more.

Despite memory being important, it’s currently trapped in a silo, living with the application where it was accumulated. Conversation history also remains trapped. A conversation with Claude will never be seen by ChatGPT, and vice versa, unless you manually copy and paste it in there.

If a user’s memory profile does turn out to be vital for driving context-driven application utility, then it will create a lock-in effect: the longer a user stays on an application, the better the memory profile becomes, and the harder it is for the user to switch to competing applications.

Moreover, while a user’s memory profile and preferences are relatively easy to transfer using a third party MCP server, the chat history is not. For every memory MCP server we examined, none of them integrated with the chat history of either ChatGPT or Claude, despite the obvious value to users.

That leads us to distinguish between two types of memory, of which chat history will be our focus.

Two Types of Memory

Memory Profile. Specific facts or preferences recorded by a model. These could be facts such as “the user’s favorite color is blue” or “the user prefers to code in Python” created via a model calling a tool. These memories are explicitly saved and can later be edited by the user.

Chat history. All previous conversations the user had with the model. A model is often given access to a history of chat logs (either via tool call or by it being injected in the system prompt) and uses that to draw inferences about user preferences when answering a question. For example, when querying a model about a long term research project, the model may gather context based on previous conversations without explicitly recording every detail of the project within the memory profile. This usually requires more tokens to process but it also provides a much richer user context.

How ChatGPT and Claude Remember You

As Shlok Khemani described in his recent blog posts on memory, OpenAI and Anthropic both utilize chat history memory to provide their products with important context about the user, but in very different ways.

In the ChatGPT application, information about the user is automatically injected prior to every conversation. First, memories that the model explicitly saved are inserted into the conversation, this could include anything from the user’s favorite color to the number of kids a user has. Second, the last 20-40 conversations2 are inserted into the current chat context window.3 Because memories are inserted into the system prompt automatically, no tool calls are made to retrieve memory. But tool calls are made to update explicit memories.4

Anthropic takes a different approach. Until recently, every chat started completely independent of all context. Anthropic used to simply provide Claude two read-only tools to access chat history memory as needed: conversation_search and recent_chats. The recent_chats tool allows the model to retrieve chats based on time information, while the conversation_search tool allows the model to search through conversations to find those that are most relevant. Anthropic is also in the process of rolling out an OpenAI-like summary of user preferences that is automatically added to the model’s context.

For developers using the API, Anthropic just released a new memory tool that allows Claude to query and manage memory by adding and searching “files that persist between sessions”. It works by giving the model access to a memory directory that it could edit and reference before answering a question. Importantly, this memory API is entirely disconnected from the Claude app and can’t be used to retrieve previous conversations the user had through the Claude UI.

A Gap in Memory

If we want portability, ease of innovation, and user control over their memory data, what should memory look like in AI markets? It’s worth asking this as a way to benchmark against where we currently are. Assume that the memory architecture interoperability is enabled by the model context protocol (MCP), used by AI applications (MCP clients) to access context across tools and databases (MCP servers).

The memory architecture could be centralized, with a single MCP server dedicated to memory (memory profile + chat history), so that when ChatGPT asks about your preferences, it reads & writes with the memory hub (or “bank”). No memory profile is saved with the application itself.

Or memory could be more federated, where each app maintains its own memory but they are queryable individually (like cash in different wallets). A hybrid system could exist too where some memory syncs with the memory bank and some memory stays trapped in the app (so a wallet + bank model).

These are all technically feasible. AI companies just don’t permit it. Creating a workable MCP memory system is comparable to your Dropbox account, but for your AI’s knowledge. Just like your files follow you across devices when you connect your Dropbox, your AI’s understanding of your preferences, projects, and context can follow you across your applications and devices with a central memory bank to which everything syncs. The same is true for your OneDrive, Google Drive, or iCloud — they sync your files no matter where they are worked on.

Existing MCP memory servers — such as Anthropic’s MCP memory server, Supermemory MCP, Memory Bank MCP, MCP Memory Service which supports chat history memory for Claude Code, and Mem0 — can’t do much though because they are unable to read from or write to the memory profiles of existing AI applications, regardless of their intended use cases.

This risks creating a two tier system, whereby first party applications leverage their chat history memory while third party, application agnostic, MCP servers are forced to rely on an explicit memory profile.5

OpenAI and Anthropic technically allow users to download all their chat logs. This is a commendable feature but not a market enabling one. Users are not developers. And it is developers who need direct access to the data if healthy, competitive, AI markets are to be incentivized. The process of exporting your memory data to a third party application is tedious, requiring a user to find the option in settings, request an email, and then importing it somewhere else.6

Moreover, every memory export is just a single frozen snapshot of your current history; it does not update as new conversations and activity occurs. Manual consumer exporting of chat history may be fine for switching applications once, but it does not lend itself well to creating a decentralized and dynamic ecosystem of applications on the basis of a user’s memory context.

Share

Out of the five memory MCP servers we surveyed,7 none of them used Claude or ChatGPT’s saved conversation history. Instead, they only allow the user to save and view explicitly set memory, disconnected entirely from the users actual chat history, like a rogue third limb with no living, breathing functionality.

The Way Forward: OpenAI and Anthropic expose memory APIs

Memory, like context more generally, is a powerful resource that will enable the explosion of new AI applications. But currently, third party applications are severely constrained in what memory they can realistically access. External MCP memory servers are able to connect to Claude or ChatGPT but not to their memory profiles and conversation history. Instead, we need Claude and ChatGPT and other important AI applications to expose their memory as an MCP server for other apps to access.

This requires considering several things:

1. MCP already supports OAuth – a way for clients to authenticate the user – so there is no technical reason why memory needs to be locked down to one AI application. Instead, AI applications could allow others to authenticate with them and carry over their chat history to any third party application.

2. OpenAI and Anthropic should expose their memory systems as an MCP server, i.e., using an API. Enabling dynamic syncing, authorization, and access is vital. Only dynamically updated, complete third party memory can completely decouple user memory context from a specific model provider and allow users the freedom to pick any AI client.

And if Anthropic and OpenAI do not provide the MCP server themselves, they should at least allow for it via an API — since third-parties can build memory servers off their applications’ memory services.

3. Some popular MCP servers like ActivePieces and MindsDB already tackle federated data access. During our research into the most used MCP servers, we found that some of the most popular servers supported multiple services to get at context, such as ActivePieces and MindsDB.

MindsDB explicitly frames MCP as a way to query federated data, including databases (PostgreSQL, MySQL), SaaS apps (Slack, Gmail), and data warehouses using SQL or a natural language.

ActivePieces does something similar. It is an open-source automation platform (like Zapier) that offers 280+ integrations available as MCP servers. Instead of your AI needing separate MCP connections for each service, ActivePieces is the gateway — the one MCP server connecting you to hundreds of services via their APIs.

The AI model can then intelligently choose which tool to use based on your request. Ask “schedule a meeting” and ActivePieces chooses Google Calendar.

An ActivePieces architecture, but for memory, is one in which each AI application retains its own memory context— so a federated system of sorts; but then the AI server can still query multiple memory sources at the backend. Access is provided to the decentralized memory wallets.

But whether federated or unified, the memory APIs don’t yet exist(!). And we are unable to build either architecture until Claude, Chat GPT, and other AI applications open up their memory. When user data is not kept trapped within a single AI application’s walled garden, new players can experiment with new formats and compete based on the merit of their product – instead of their ability to access data.

One last under-appreciated point is that the type of memory context generated varies by interface. So if Claude and ChatGPT’s AI memory was open to other applications, then new applications would be able to branch out into areas well beyond chat and still gain insight from the user’s discussions. Coding agent Cursor, with access to your ChatGPT history, for example, may better understand the project it is coding than with Cursor alone. A writing application with integrated AI may benefit from insights you had in a chat interface. But requiring users to build up the full context for every application separately is not feasible or efficient.

Conclusion — Portability must shape markets

A user’s data should belong to the user, not to whichever application first captured it or simply houses it. But the meaning behind such a “portability right” is unclear. In practice, portability manifestos have amounted to little more than users being asked to carry their data around with them on their backs from application to application. The utility of this is questionable. Has this increased consumer and producer welfare in the European Union? We think not. The flow of information in digital markets is driven by developer infrastructure, not by users manually navigating between different databases.

Data portability rights derive their real meaning from helping users gain material benefit from their data: ensuring it can lower barriers to market entry for smaller competitors and help foster healthy competition based on superior products, rather than on data exploitation and contrived lock-in mechanisms.

Allowing a user’s memory — one of the most important pieces of context — to be shared between applications across the AI ecosystem would deliver tangible benefits for both consumers and model developers. This can increase the size of the pie for everyone and help ensure that portability rights remain grounded in the material, market-based contexts in which they exist.

Subscribe now

The Trump administration’s push to move away from quarterly corporate reporting has sparked a debate about the value of corporate reports, with important takeaways for AI-related reporting requirements.

John Authers noted in his recent column, quoting Sarah Williamson at FCLT Global, that disclosures are not really about timing, but “materiality” (i.e., importance): “What really matters…is the materiality of what to tell investors, not the periodicity.” And the bar for what is material should be lower, she argues. That is, more things should be considered important for companies to disclose rather than fewer.

This principle has potentially far-reaching implications for AI disclosures. Rather than getting caught up in debates over how often AI companies should report, we should be asking: What risk events in AI systems are already material enough to warrant immediate disclosure by corporations? And how exactly should these be disclosed by public companies?

In line with our previous work, we call for prioritizing disclosure depth over disclosure frequency – it's about quality over quantity.

Drawing on the SEC’s 2023 rule on cybersecurity incident reporting, we propose a dedicated AI risk item in the 8-K Form — an event-driven, impact-based trigger whenever an AI incident materially affects the company. We also propose adding a standing item to the annual 10-K corporate disclosure Form on AI that explains a company’s risk management, strategy, governance, and key dependencies. In the interim, the SEC should issue Disclosure Guidance with concrete examples of AI events that may meet materiality thresholds and how they map to existing 8-K and 10-K items.

In doing so, we can advance AI-related disclosures that are more comparable, timely, and granular — and that materially strengthen company risk mitigation. And as a bonus, standardizing these disclosures (ideally with machine-readable tags) will also improve decision-usefulness for investors and oversight by regulators.

Cyber Risks as a Model Disclosure Framework for AI-Related Risks?

Perhaps as a result of a recent proposal from the Long Term Stock Exchange (Disclosure: Tim O’Reilly is an investor), which was reported on by The Wall Street Journal on September 8, President Trump recently proposed that public companies’ quarterly reporting should instead become bi-annual (twice a year). Trump certainly made the case for it based on the same LTSE argument: that quarterly reporting places undue burdens on public companies and pushes executives into short-termism – so-called “expectations management.”

Time-based reporting requirements incentivize companies to structure decisions around a company’s financial calendar, which might delay crucial information being released to the public as it occurs.

Enter the 8-K Form. The 8-K is sort of like a breaking news bulletin, since through it significant events are announced by the company within four business days. But the list of “Items” varies in what triggers them. Many items are triggered automatically by an event, like a corporate bankruptcy. Other items on the list are based on the company’s judgment around whether the event is “material”1 i.e., sufficiently likely that a reasonable investor would care about it – and only then would they file an 8-K form. A cybersecurity incident is one such “if it’s important enough” thing to disclose (Item 1.05).

An important and relatively new corporate disclosure requirement that uses the 8-K Form is the SEC’s 2023 Cybersecurity Incident rule for public companies. It says that when a company suffers a material cybersecurity incident it must report it to shareholders within four business days through the 8-K Form. And when it’s a material cyber event impacting shareholders, then it can be filed through the newly added Item 1.05, specifically for cyber incidents.

Share

In combination with strong SEC enforcement, the rule seems to have worked. Cyber incidents are disclosed in a far more timely and comparable manner now, and companies appear to be devoting more resources to the problem. Moreover, companies absorbed these new requirements with ease because they were well prepared from previous guidance.

Part of the Rule’s innovation is that the material event-triggered 8-K filing for cyber incidents sits alongside a standing annual 10-K disclosure requirement for companies specifically for cyber-related issues (Reg S-K Item 106), covering things like board and management oversight, processes for identifying and managing material cyber risks, whether such risks materially affect the company, and more.

The question then is whether AI-specific risks require a similar treatment to cyber ones. Below we show that a substantial “disclosure gap” already exists for AI, which only enhanced AI-related disclosures can fill. This is the gap between the AI-risks already out there facing AI companies, and what they are currently disclosing.

Evidence on the Disclosure Gap

That AI-specific risks are impacting corporations is now very apparent. Fisher Phillips’ AI litigation tracker for the U.S. currently shows 92 cases.2 Litigation on securities class action lawsuits covering false or misleading statements on AI is on a near exponential rise in the U.S., from 7 cases in 2023, 14 cases in 2024, and 12 cases so far in 2025. In Garcia v. Character.AI & Google, the court has let the case proceed (May 22, 2025) over a teen’s suicide, allegedly encouraged by a chatbot’s messages. Claims include wrongful death, negligence, and deceptive trade practices. The point being that AI-usage now exposes companies to a range of risks from product liability & negligence, wrongful death, defamation, and publicity and privacy, to name but a few.

To manage growing AI-specific risks, companies are trying to disclose more to their shareholders, even if only superficially. An analysis by Arize AI, as reported by the Financial Times, found that 56% of Fortune 500 companies cited AI as a “risk factor” in their most recent 2024 annual 10-K reports.3 Netflix, Motorola, and Salesforce all discuss AI-specific risks.

Even though AI-risks in the 10-K are no longer a niche, they are also not yet particularly useful. SEC staff letters to companies show that much of the guidance was thin on details. Staff consistently requested more specifics.

Aware of the AI-disclosures gap, the SEC launched in 2024 AI-specific guidance covering AI washing, conflicts of interest, and systemic risk, along with enforcement actions.4 The SEC now even has a newly dedicated Chief AI Officer (CAIO) Valerie A. Szczepanik, who will oversee a new SEC AI Task Force, though its focus is more on internal innovations.

Finally, with respect to the 8-K, we constructed our own dataset of all AI-related event-driven filings since November 1, 2022, covering 1,741 corporate issuers. It highlights at least three key AI-related corporate disclosure gaps:

1. No Risks Here. The first is that 8-K disclosures almost exclusively concern a company’s commercial ventures (Figure 1 below): covering important agreements (Item 1.01) – such as model licensing, cloud/compute commitments, strategic data deals, and reseller/partnership agreements; but also Financial matters (Item 2.02). Safety and guardrails, i.e., AI-risks, rarely feature (though we have not yet done a full textual analysis).

Figure 1. 8-K filings by public companies in the U.S. on artificial intelligence and generative AI, by Item topic.

2. Confusion reigns. Secondly, most disclosures for AI-related impacts are through Item 8.01: a voluntary catch-all event category useful for AI updates that are not yet a mandated material trigger but still market-relevant. This means that companies are not yet sure where to put such AI-triggered events and/or are unsure when an event is sufficiently material to disclose it elsewhere.

3. Big firms need a 10-K mandate. Finally, 8-K filings on AI-related matters are driven by smaller companies. Big Tech’s 8-K disclosures are not very prominent: AMZN (14), NVDA (11), MSFT (10), META (12), GOOGL/GOOG (9 each) amount to well under 1% of the total filings we examine.

   Practically, this means that any new 10-K requirement to cover AI-specific business activities and risks in detail could significantly enhance market transparency, since these mega-cap firms have an outsized impact on the AI market already (together with OpenAI, Anthropic, and a few others).

Subscribe now

So What Should We Be Aiming For?

1. Guidance Note. To get the ball rolling, an SEC guidance note (called “CF Disclosure Guidance”) could help companies understand how existing disclosure rules apply to AI. It’s not binding law, but it can strongly influence company filings and SEC actions. For example, the 2011 Cybersecurity memo (Topic No. 2) told issuers what to discuss under Risk Factors, MD&A, Business, and other items in their 10-K report. For AI, it could provide similar guidance, giving practical and specific examples of things to discuss under relevant Items, avoiding boilerplate discussions, covering their business (S-K Item 101), risk factors (S-K Item 105), trends and uncertainties (S-K Item 303), and more.

For 8-K material events triggered by AI requiring urgent filing, SEC Guidance might discuss potential quantitative and qualitative triggers to monitor for potential 8-K events that require further internal company discussion. For example:

• A major quantitative change (relative to a historical baseline) in important KPIs, possibly driven by AI changes to DAU/MAU, time-on-platform, ad CTR, conversion rates, credit-approval rates, charge-offs, loss ratios, etc., might require further company review. As could a notable increase in harmful model outputs, jailbreak success rates, self-harm exposure, fraud detection, or similar operational risk metrics monitored that are far outside of normal behavior.

  These metrics are already monitored internally and so can easily be used to trigger a potential 8-K filing.

• Qualitative triggers could similarly be used to trigger a corporate 8-K disclosure review and might include things like: AI objective and optimization changes, guardrail and policy changes that are likely to alter harmful output rates or regulatory exposure, and data provenance or infrastructure shifts — including new sensitive datasets, migration of core features to third-party models and APIs, and significant compute capacity losses.

2. Create a new AI-risk item on the 8-K disclosure Form for significant events as they happen. Companies already use the 8-K Form to alert investors when something important happens between annual or quarterly reports. The idea here is to add a dedicated item for AI-related incidents, so that there is a clear place to report them when they matter. This can help ensure that companies do not skip reporting the “risks” when disclosing material AI-related events.

“Incident” should be understood broadly: for example, a model failure that misprices loans, an AI system outage that interrupts service, an AI-driven error that requires customer remediation, or a sudden loss of access to a third-party model that a product depends on.

The key point here is that this sort of corporate disclosure is impact-based rather than technology-based. In other words, the trigger is not “an AI model changed,” but that “the change or failure had a meaningful effect on operations, customers, compliance, or financial results.” This mirrors how materiality works across corporate reporting: if a reasonable investor would want to know about it because it changes the picture of the business, it belongs in an 8-K.

3. Add a standing AI section in the annual 10-K Form that explains how a company manages AI. One-off 8-K event reports are, by themselves, insufficient. Investors also need a clear, yearly picture of how a company runs its AI-related activities, covering: how it is used in products and operations, who oversees it, what the main risks are, and what controls are in place. A new 10-K item would provide that exact structure, thereby encouraging companies themselves to adopt a longer view of these risks.

Companies would explain their approach to risk management (how they test and monitor systems, how they roll out changes, how they respond when something goes wrong); their strategy (where AI fits in the business and why); and their governance (who is accountable at the management and board level). They would also describe key dependencies that could affect reliability or cost (such as reliance on outside model providers, critical data sources, or a single cloud vendor), along with any concentration risks that come with those choices.

The goal is not to jam in unnecessary detail into the 10-K but to make the business implications of AI understandable to the investing public: where the leverage points are, how failure is prevented, and what the plan is when problems occur.

Finally, labeling the main AI elements with standard, machine-readable (iXBRL) tags (the same way the SEC does for several other disclosures, such as the SEC’s cyber rule) would let analysts and watchdogs compare companies more easily and spot patterns over time.

Empower Markets with Information

As AI systems become more powerful and more integrated into critical infrastructure, pressure for more prescriptive regulation will grow. By establishing simple, effective disclosure regimes now, backed by appropriate institutional capability at the SEC, we can create the foundation for more sophisticated governance that becomes integrated into third-party regulatory markets as the technology matures. Information empowers markets.

The question is not whether we need AI disclosures from corporations, it’s whether we will get them right. For us that means focusing on enhancing the quality of corporate disclosures rather than the quantity.