The question is: “Would you have built the Sydney Opera House? Show your working”.

As well as the questions I discuss below, I think this is a very interesting on in the field of aesthetics and even moral philosophy. The Opera House was funded by a lottery, and most of the people who bought tickets don’t go to the opera. It’s an absolute paradigm case of highbrow culture funded by taxes on the working class. But it’s very popular! The question here suggests that doing something which is basically unfair can, in principle, be justified retrospectively by artistic merit. Or does it? You tell me.

We are heading into the closing chapters of the book now, where I start talking about solutions and ways to tame the problem factory. There is a lot that can be done, I think, although first I need to talk about some ideas that won’t work or might be counterproductive. We’ve demonstrated our respect for the problem, at length, and we can now stop touching gloves with our adversary and start repeatedly punching it. But before we go into this, I need you to do a bit of a gut check.

In your heart of hearts, deep down, do you think it was a good idea to build the Sydney Opera House?

Looking through old textbooks on planning problems, the Opera House often shows up as an example of a “Great Planning Disaster”. In the book of that name, the theorist Peter Hall devotes a chapter to it, detailing how its costs ended up overrunning by more than ten times. The architect, Jørn Utzon, had never designed anything bigger than a medium-sized housing project. Ove Arup, the engineer responsible for realising his vision, described the cost forecast as having been “prepared, I believe, by some unfortunate Quantity Surveyors under duress in a few hours”.

This is something we might have to sit with, potentially a bit uncomfortably; whatever solution we come up with to all the social and organisational pathologies that stop things from being built, it is unlikely to be something which regularly facilitates things like this. And everyone knew this at the time. The project was started without proper planning, leading to the demolition of millions of dollars worth of wasted work, because as Arup put it “if a fait accompli were not established while Mr Cahill was Premier, the job would probably not go forward at all”.

But a lot of people, including most Sydneysiders, think now, and thought at the time, that it was worth it. There was even substantial support for decades afterward for continuing to finance the huge overrun and the underestimated running costs out of the proceeds of a lottery, meaning that they would largely be paid for by people who didn’t watch much opera.

Conversely, though, when I ask people the same heart-of-hearts question about whether the Concorde project should have gone ahead, they’re much more reluctant to say yes, even though it’s exactly the same sort of white elephant prestige project. And almost nobody will defend NASA’s decision to change their risk tolerance rule from “cancel a launch if there is any doubt about safety” to “proceed with a launch unless there is evidence it is unsafe” ahead of the 1986 Space Shuttle disaster.

I have consciously tried to avoid using the language of “trade-offs” so far in this book. There’s a number of reasons for this. First, it’s not an accurate way to describe most of the problems. Tradeoffs exist when you are at the efficient frontier, when it’s really not possible to get more of one desired thing without giving up another. We are nowhere near that frontier in most of our decision making; we can usually have more of everything we want, simply by spending less on doing stupid things which slow us down and annoy everyone. Removing resources from the problem factory and giving them to the solution factory isn’t a tradeoff, it’s just a gain.

Secondly, I find that the language of tradeoffs is often used in a rather bullying way. If you listen to people who are objecting to something, it’s rare that they don’t understand that there are tradeoffs in policy. They just don’t think it’s worth it. Or they think that the costs are falling disproportionately on them for benefits that go somewhere else. People think that they are sounding wise when they say that “the public want nice things but don’t want to pay for them”. But that’s just what the words “nice things” and “paying” mean. Everyone wants nice things, and nobody wants to pay, they used to teach you this when you did an economics degree.

Here is a real tradeoff, though. We are studying decision making, under conditions of uncertainty. That means that mistakes are inevitable. We can push toward the efficient frontier, to make as few mistakes as possible. But there comes a point at which it is too costly to reduce the error rate any more. Given the context is often that of vitally necessary investments, we cannot even help ourselves to the proverbial wisdom that the only way to avoid mistakes is to do nothing at all, because doing nothing may itself be a big mistake.

So there is going to be a decision to make; will our errors be those of commission, or of omission? It’s not an easy question, and I certainly can’t answer it on behalf of anyone else. I just want to warn you that even if you think you have come up with a solid set of principles to say “yes” to the Sydney Opera House, “no” to Concorde and “definitely not” to the Space Shuttle disaster, circumstances may find a way to make you look a fool, or worse.

The builders and blockers are a slightly different issue, because they’re intrinsically political rather than technical. They either need to be treated as problems to be solved with the same approach as other bottlenecks, or taken seriously as political stakeholders. And asking that question is the same thing as asking “has the decision to build this thing actually been taken yet?” In my view, a lot of the problems on the Abundance Agenda might have a negative answer to this as their root.

So, let’s call this the question of “the point of decision finality” – the moment when the answer to this question changes from no to yes. The thing that’s troubling me is – does this entity, the point of decision finality, does it exist? Is there any such thing? If there is, then it’s clearly an important question of institutional design to say where in the workflow the point should be placed.

But so far I’ve just raised philosophical question after philosophical conundrum for myself – so I hope the philosophers among my readers might be able to help. Clearly, there is a sense in which there is no finality to a decision to build something until the ribbon is cut and the doors opened; projects can be abandoned and often are. Does this mean that there is no such thing as a final decision?

I think the concept can be rescued. Let’s say that the “point of decision finality” is the moment before which not building the thing is correctly described as “we decided not to”, and after which not building the thing is correctly described as “we failed to build the thing”. Now the question is – will this definition support my intuition that the location in the workflow of the point of decision finality is both important and something which is itself part of the design of the system?

My answer would be “not without a bit more substance to it”. Simply changing what you call a project from “failed” to “decided not to” isn’t something that makes a difference in the real world, and it doesn’t meet the challenge. So, what does make that difference; what changes in the world and the organisation when a decision crosses the point of finality.

The answer has to be one of information architecture, because all organisational questions are. After the point of decision finality, there is a qualitative change in the kind of information which the system is allowed to pay attention to. Before some project P reaches decision finality, for example, people are allowed to suggest plans under which the organisation tries to achieve its higher level goals through a set of projects which doesn’t include P. After that point, P has been decided on; the set of alternative plans to be considered is shrunk.

That means that the analytical problem with respect to P is “how can this be achieved?” rather than “should we do this?”. There’s another conundrum here for me, though, because I am not sure that these two questions are sufficiently different – as long as the first question could potentially be answered “it can’t” (or “it can’t, except at excessive cost”), then the second is open. As I said earlier, nothing’s really “decided” until it’s finished.

Which means that there is a constraint on the placing of the point of decision finality. In order for the decision to acceptably approach the ideal of finality, we have to say that to declare a decision final is to assert that you are confident that it can be achieved within the resources allocated to it.

So there’s a limit on how early in the workflow the point of finality can be. I am going to assert that subject to this constraint, the point of decision should come as early in the workflow as possible; keeping decisions open when they ought to be closed seems to create a lot of opportunities for people to engage in strategic behaviour. (It also seems to give advisory bodies like the Planning Inspectorate delusions of being judges; when people feel like a decision is still open, they also seem to believe it’s incumbent on them to maintain the appearance of an open mind).

And that suggests another point of decision architecture. You can’t achieve decision finality until you are reasonably certain that a project can be achieved in an acceptable budget envelope (with “budget envelope” to be taken here as meaning the budget in terms of environmental cost and political goodwill as well as the financial). But of course, you can’t be anywhere near certain of that until you know what the budget actually is. So this suggests two stages of analysis; an initial broad-brush exercise establishing the order of magnitude of the benefits and costs and bringing the tradeoffs into the open, then the decision point, then a detailed planning exercise to see how much surplus can be created, in terms of doing better than the tradeoff which was accepted. Of course, if the second number is negative, then that’s a failure, but I think the idea of “the project has failed” now has more definite organisational and economic meaning.

Sorry, that was a bit dense wasn’t it, readers? I’ll try to come up with a joke by Friday.

Anyway, I was at a seminar this week and, as is my habit, made a silly joke which I now take seriously and think is my actual view. We were addressing the question of “Building Things”, and I proposed the following thought experiment:

What is the largest piece of public sector infrastructure that you could build, right now, if you were restricted to using no more than two contractors, getting permission from no more than two planning bodies and employing no more than two professional services firms?

The idea was to try and get a physical scale and scope for something we were certain we could achieve given the administrative and decision making resources at our disposal. My guess is that the answer might be “something really embarrassingly small, like possibly a bus shelter”. But, infrastructure projects like this have one important advantage over more serious and necessary projects, which is that we can actually do them. There’s only fifteen possible communication channels between six nodes (twenty-one if we imagine there’s a commissioning body that needs to talk to all of them), which ought to be possible for one person to keep track of with an email outbox and a telephone.

So, if we want to get building, let’s not start by being blinded with foolish pride. Declare open season for bus shelters and similar small scale projects. But while we’re building, we need to make sure we’re learning. So let’s also commission a few projects which are a bit bigger than the scale that we know we can manage – we need another thought experiment along the lines of “what’s the smallest thing that you think you couldn’t build if you were restricted to two contractors, two permissioning authorities and two professional services firms?”

Some of those projects will succeed and some will fail, because we’re not going to be able to precisely estimate our own capacity. But we can learn from the ones that fail – this will give us lessons about where the bottlenecks are. And we can learn even more from the ones which don’t exactly fail, but which take much longer than they ought to.

Now we have a, hopefully manageable, “snag list”. This will consist of communication bottlenecks, perceived legal risks and regulatory problems. It will also include some bad faith actors and “blockers” who are opposed to the aim. The first category of problems are the suitable raw material for Malcolm Sparrow’s “problem-oriented governance approach” – you recruit a problem team from all the areas the problem touches, agree on a definition, brainstorm and experiment solutions, then implement and communicate.

The builders and blockers are a slightly different issue, because they’re intrinsically political rather than technical. They either need to be treated as problems to be solved with the same approach as other bottlenecks, or taken seriously as political stakeholders. And asking that question is the same thing as asking “has the decision to build this thing actually been taken yet?” In my view, a lot of the problems on the Abundance Agenda might have a negative answer to this as their root.

And then you iterate; if the problem teams have concluded, communicated and wound up successfully, then the largest object that you can build within your administrative capacity is now a bit bigger. So, build a bunch of those, and then also commission a few more “learning” projects which feel like they’re a bit bigger than you can handle. You’ll be putting high speed trams across the Pennines before you know it.

Readers might notice that I haven’t defined “you” above, which was intentional, because I think this sort of approach can work at different levels from local authorities to departments to central government. The important thing, I think is to start with a realistic and honest assessment of current capacity. As Sun Tsu said, know the enemy and know yourself, and you need not fear the result of a hundred battles.

There is now a second edition available for pre-order, which is greatly revised from the first. The conclusion of the Grenfell Inquiry has made a lot of new information available, but more importantly, has made it much less legally dangerous to be specific about companies and building products which contributed to the disaster. I was lucky enough to blag a pre-publication copy and think anyone who is interested in the general subject matter of “Why don’t we learn?” and “How can we change?” will really enjoy the new edition. If you’re engaged with YIMBY adjacent topics in construction regulation, I would say it’s absolutely required reading.

Anyway, on with the show. I will try not to let the length of this one get out of control, because it’s one of my favourite subjects – the potential for AI to replace middle managers. I am going to do people like Matthew Prince and Jack Dorsey the courtesy of taking them seriously and assuming that they mean “actually replacing middle managers” here, rather than something anodyne like “increasing the productivity of middle managers so that each manager can supervise a larger unit”. Because if all they meant was “I have changed the capital/labour ratio in a company I run” then come on, that’s a bit of a waste of our attention.

Here are a few statements I regard as axiomatic:

1. Actually running a business is more difficult than buying and selling shares on a liquid market.

2. Picking stocks to outperform the market is something which a neural network can do, we know this because human brains do it.

3. However, most human beings, including most human beings who are paid to do so, can’t.

To me, this implies, straight off the bat, that trying to beat the equity market is an at-least AGI level task, and that trying to manage an organisation is potentially super-AGI level. (I’ve made a similar argument in the past, suggesting that a computer which was really capable of handling planning applications would have to be so smart that we would long since have handed over the whole of government to it). This seems to me at least to be either a refutation of the idea that “AI can replace 20% of my managers”, or a massive self-own on the part of people implicitly admitting that their business is so simple, uncompetitive and overstaffed that current generations of Claude can handle it.

[MASSIVE RABBIT HOLE WHICH I WILL POINT TO BUT NOT GO DOWN. Of course, part of the problem is that stock-picking is an adversarial process in which people compete with one another; doubling everyone’s cognitive capacity in the market would just make it more efficient without changing the proportion of people capable of generating outperformance. But so is managing a business, in most sectors.]

Anyway, I have about 500 words left, so I will sketch out why I think that LLMs, specifically, will have a hard time even solving the easier problem of stock picking. I start from the observation that there are a lot of very successful computer systems which beat the stock market using machine learning; they just don’t look very much like large language models.

In fact, quant systems get their edge over a market dominated by human beings precisely because they don’t reproduce human reasoning; they have rules and stick to them. There are two ways to underperform the stock market – by hanging on stubbornly to losers, or by getting the collywobbles and exiting from a good trade too quickly. (The proverb is that “amateurs go broke by taking big losses, professionals go broke by taking small profits”).

So imagine that we have an LLM with appropriate training and a system prompt that says “you are a really good stock picker. Pick stocks that will go up, hold on to them and when they are about to go down, sell them. Don’t make mistakes.” And it’s picked a good stock, that’s gone up a lot, but has now started to go down a bit. Will it get the collywobbles?

Yes! Because “getting the collywobbles in a good trade” is exactly the same activity as “not hanging on to a loser”, except that you’re wrong. Symmetrically, “hanging on stubbornly to a loser” is the same thing as “not getting shaken out of a good trade”, except you’re wrong. Stock market proverbs are massively contradictory; let your winners ride, but pigs get slaughtered. Cut your losses, but in the long term it’s a weighing machine. The trend is your friend, but be contrarian. Etc etc.

The trick is to understand what situation you’re in, and to apply the right one of those rules. And, of course, coping with a high-dimensional environment in which different rules are applicable is one thing that neural nets can handle very well.

It’s just that the neural nets which handle it well are the ones which have been specifically trained to do so, like the actual quant models which people actually use. A neural net which has been pre-trained on a massive corpus of human language by a transformer network is just going to get the collywobbles and double down like a human analyst, because that’s what happens in its training data.

The only way to stop it from doing so, or to bring it up to the standard of the best human analysts, would be to create a great big tagged dataset of good and bad trades along with the reasoning that led to them, so as to get better pre-training weights. And then you’d take on the huge additional task of updating the tagged dataset, so that it didn’t go stale as the market and the economy changed. (Stale knowledge and old habits are a huge problem for human stockpickers). But why would you do that, rather than just directly training the neural net to pick stocks?

And this matters, because when you shift focus back from stockpicking to general management, there’s no equivalent of directly training the model on the data. And it’s not even clear that “the data” has a stable referent – management is very much a game of “horses for courses”, in which the curse of dimensionality is very great indeed.

So you’re left with a model that has to be trained on the corpus of human decisions, and which will therefore be likely to reproduce human mistakes. Unless you can hire an army of offshore workers to tag management decisions as “good” and “bad”, I suppose. But that sounds expensive.

“…concerned not with abundance at all but rather its opposite, bankruptcy. Greece’s Odious Debt, by Jason Manolopoulos, was a polemic published in May 2011, in the very earliest days of the Greek debt crisis. While all three books discussed here so far are attempting to address the question of how we can stop our once-productive social and political institutions from becoming pathological, Manolopolous was forced to ask the same question shortly after the bill became due.

Greece’s Odious Debt contains a thoroughly unsentimental but nonetheless moving history of how the Hellenic Republic got to where it was, and importantly how the combination of financialization, political myopia and the economics of European Monetary Union caused it to spend a decade in a world of false abundance. For a few years, Greek citizens lived well beyond their means, importing Mercedes cars and enjoying a real estate boom. When the crisis happened, they were brought face to face, as few peacetime societies ever are, with two facts: Their system of governance was rotten, and their prosperity was fake.

The economy of Greece has still not recovered; GDP is about 20 percent below its 2008 peak. And its politics have by no means been transformed. So there is no solution here, but there is a lesson. Environmental debts will come due, just like financial ones. So will political and social problems that have been put off to the future because they didn’t appear to have engineering or lawyerly solutions.”

I still think this might be a serious challenge to the Abundance agenda; that achieving the goal of growth and human flourishing might occasionally involve taking a step back, and that we might be at one of those points right now. It doesn’t mean that the whole project is discredited - far from it, it implies that solving the problems which make it too difficult to construct renewable energy infrastructure is really very urgent.

But it does mean that we can’t brush off “degrowth” as easily as we might want to. Carbon dioxide emissions are very like debt; they provide a current benefit in exchange for a future liability, and the future liability might be significantly bigger than the initial benefit. It’s just that it’s a form of future obligation which doesn’t appear in the same balance sheet as financial debt.

I don’t think this is a difficult accounting problem; it’s tricky to quantify the environmental debt and to translate it into a precise number of pounds or dollars. But it’s surely not difficult to understand that it exists, or that it’s likely to be big - we’re already thinking about abandoning entire towns, for heaven’s sake. And yet, people say that Net Zero is unaffordable, who would never say the same thing about the 4.75% October 2035 Gilt.

When a country has an excessive debt burden of either kind, bad things are likely to happen, and the only solution is either a bailout or a period of austerity. I had, and have, many disagreements with Yanis Varoufakis and with Wolfgang Schaeuble, but they did, at least, start from recognising the facts. Who is going to tell us that our prosperity is fake?

And more worryingly, what is going to happen to them when they do? One of the key motivating factors behind the Abundance Agenda is that the politics of scarcity are incredibly bad right now. The main reason why we need policies capable of delivering economic growth is that without it, a lot of voters seem to develop a really unattractive and dangerous mean streak. I still think it’s possible to design some glide path which will maintain developed world consumption levels while financing investment and writing off a large part of the last twenty years’ malinvestment. Just don’t ask me to write it down right now.

Subscribe now

Lots of people are very much “gahh, enshittification, I just want my weblinks”, but I’m not - I do think this is potentially very useful. Being able to carry out a semantic search - something like “find me three examples of trade associations making outlandish claims about the economic impact of housing regulation” is a real time saver, and although you do have to do a lot of back-and-forth with the AI to get what you want, it’s less of a pain in the arse than filling your screen with browser tabs and reading the press releases yourself.

However, the question I want to ask is - if we put that kind of issue to one side and assume for the sake of argument that a really useful product will come out of this, how would we expect to see that reflected in the economic statistics?

I’m not sure that we would at all. I like using the intelligent search box, but that’s just unmeasured consumer surplus to me. I don’t like it so much that I’d pay for it, and Google know that if they were to charge for search, they would lose market share instantly.

It also doesn’t seem likely to me that Google could monetise that consumer surplus by raising its ad rates. It’s a monopoly already; it’s squeezing as much out of the advertisers as they have to give.

Google might be able to serve me more ads, if I spend more time in the intelligent search box and less time on other sites. But this is close to a zero sum game, in that those other sites will have fewer opportunities to sell me ads.

So, at best, the increment to Google’s revenue (and therefore, the only thing that has any chance at all of going into GDP) is the extent to which time spent on a Google site with ads is substituted for time spent on other sites which didn’t previously have ads. It feels pretty marginal, particularly since there’s a huge amount of capex needed to achieve it.

I think what’s going on here is that we’re experiencing a less than perfectly foreseen consequence of the big trend towards corporations trying to create “moats”. When a monopolist is earning monopoly profits, then they will invest to protect that monopoly, even if the investment is not one that would be justifiable as a stand-alone. If you are earning profits of 100, and you are faced with the need to make a big capital investment in order to ensure that those profits only drop to 90 rather than to 50, then you grit your teeth and make it.

If a lot of the capex going into AI is going into this kind of investment - negative sum games between monopolists trying to protect their rents - then we shouldn’t expect to see the big consequences that everyone’s been predicting. We’ll just get some more unmeasured consumer surplus, a few marginal players will go bust and the moat will keep getting filled.

This is a weird industry that almost no other country has in anything like the same size, made up of small consultancies and guys called Brian[1]. They have a great deal of specialised expertise and they sell it by the hour on a global market. This is about a fifth of Britain’s entire exports; it’s twice as big as financial services.

One point that I’ve been making since I started writing about it is that the Old Rectory Mittelstand is actually a lot more similar to its German namesake than you’d think. The British economy and the Germany economy are really in the business of selling knowledge and technical capability. It’s just that one set of companies sell the pure product straight from the teat, while the others sell a product bundle which includes a big steel box with a motor and some microprocessors in it.

This line of thinking comes from Peter Drucker, who famously said that the only two vital functions of a company were innovation and marketing. Coming up with new products and selling them are the only activities that directly lead to value creation; everything else is a cost. I’m not sure I agree with that, but it’s interesting to think about. In any case, you can see that although advising on how to bodge widgets is a service while making an automatic widget-bodger is manufacturing, they share a certain core of important things; it’s a matter of the boundary of the firm rather than something intrinsic to widget bodging as an activity.

What’s on my mind, though, is that the British version has some quite difficult political economy. At the level of society, the German model of bundling your high-value-added product with a bunch of steel and copper has the advantage that it gives something for people to do if they aren’t able to generate valuable quantities of knowhow and expertise themselves.

If you reorganise your highest value-added industries in such a way that the majority of the population can’t participate in them, this is already the sort of thing that has the seeds of future conflict. But there is also a transition problem from one state to the other.

Because the people who actually build the valuable things are psychologically unlikely to agree with Peter Drucker that they are a cost centre. They pick up a bunch of commodity components and put down a million dollar WidgetBodgeMatic; how is that not creating value? And the factory floor employees have a lot of skill of their own, which they are rightly proud of; it’s just not the right kind of skill to be productised.

So perhaps part of our national malaise is that we’ve got a macro-scale version of the problem that the Brompton factory had when it moved from piece-work craft assembly to a production line. People who previously considered themselves to be the heart and soul of the enterprise, with some justification for doing so, came face to face with one of the most intolerable things a human mind can consider – that they weren’t as important as they thought they were.

This actually holds out at least as much of a warning for Germany as for the UK though.Although the Brian of Britain model has this political economy problem, the alternative model has a business economy problem – it’s not so flexible and depends a lot on specialised capital equipment that can’t be repurposed easily.If you’re going through an economic or technological transition, particularly one which leads to the obsolescence of a previously very important class of metal boxes with motors in them, then you get all the same problems of political adjustment, plus a load of economic problems too.

[1] Or, indeed, if the particular field of expertise is newly published European financial regulation, called Dan.

Among other things, one of the key subjects of my current book is the concept of “career risk”. It’s one of those things that everyone instinctively knows what it is, but which doesn’t seem to have any rigorous theory of it. So the only thing to do is what I always do, which is to theorise it myself, at which point Sodd’s Law begins to operate and I suddenly discover all the famous people who have been writing about it since 1947.

I think career risk does explain a lot, both in terms of there being a lot of things in the modern world which happen because of it, and in terms of it being a very useful generalisable concept. Henry and Cosma talk about “coarse-graining” as an essential feature of bureaucracy, and below, I argue that one inevitable consequence of coarse-graining in organisations is that it provides the ground for career risk (and therefore, for certain kinds of friction and entropy). Obviously, for the same reason I was thinking about accounting systems while everyone in Santa Fe was talking about algorithms and hierarchies, I use the term “black box” instead of “coarse grain”, but we’re all looking at different parts of the same elephant.

Subscribe now

Black box actors are the condition for the existence of career risk

It seems, then, that our theory of career risk also ought to be built on the understanding that it is a problem of expertise. It arises when there are people who make decisions because they have a special role in the information processing system. This special role is that of being a “black box”.

I’m using the term “black box” here in a bit of a specialised way. There’s a bunch of theory in the background, for which I’m sorry. I learned it from Stafford Beer’s books on management cybernetics, as a way of modelling complex systems. When you’re doing that, there is an obvious need to reduce that complexity to a sensible level in order to explain what is going on. One way to reduce complexity is to take some part of the system, draw a box around it and then shade that box in to be completely black[1], visually declaring that you are no longer going to worry about the internal structure of that part of the system, but rather just treat it as something with inputs and outputs. The information going in can be seen, as can the decisions going out, but the process by which one is converted into the other is opaque. Not necessarily because it absolutely has to be mysterious, but rather because doing it any other way would be unrealistically expensive in terms of time and effort.

And although it’s best understood as an analytical technique for thinking and writing about organisations, the principle of the black box rings fairly at the level of ground truth as well. It’s a large part of what it means to delegate responsibility for something – to do so without also agreeing to treat the person or group that you’ve delegated to as a black box is the sin of “micromanagement[2]”. And there are people with particular decision making qualifications or legal certification – like social workers, or building inspectors – who are intrinsically black-boxed. For these kinds of workers, the nature of their position gives them the authority not to be second-guessed.

I worry that I might just have made it sound quite fun to be a black box. To some extent, it is. It’s one of the nice things about being a professional, that your judgement and opinions are respected in the way that those of hourly employees often aren’t. But there’s always a double edge. The same black-box property which stops you from being second guessed or overruled means that nobody is interested in your explanations for your decisions; it is definitional of being a black box that you are going to be judged by results[3].

And here we reach the payoff of our project of theorising career risk as a problem of information. If you are going to be judged by the results, and the results are uncertain (as they obviously are, in this naughty world we live in), then you are personally exposed to the risk. Furthermore, your risk exposure is exactly like that of the fund manager we were talking about at the beginning of this chapter; all your eggs are in one basket. This might be one decision among hundreds for your organisation, but it’s a much bigger part of your personal portfolio.

[1] I have actually done this once or twice, on a whiteboard when drawing management structure diagrams. It was unbelievably satisfying.

[2] In the jargon, there are also “muddy boxes”, which are treated as black most of the time, but which can be periodically opened up and examined when there is need to do so.

[3] If you’ll forgive even more jargon, the same management cybernetics books call this the problem of the “resource bargain”. Gaining the status of a black box is a kind of bargain, in which you gain autonomy in exchange for accepting accountability.

Subscribe now

What I mean by this is that although it is all a bit silly, and the fact that he renamed Claude “Claudia” is weird and creepy[1], there are things here which look like they might call back to some quite significant questions about the philosophy of mind. And particularly given that it’s Richard Dawkins, who has been too big to edit for almost as long as I’ve been alive[2], he might not quite mean what it looks like he means.

When Dawkins says that “If my friend Claudia is not conscious, then what the hell is consciousness for?” you have to put it in the context of the fact that he’s spent fifty years arguing with the kind of berk who says “wot do u mean, genes can’t be selfish theyre just molecules LOL”. He’s spent a lot more time than the rest of us on the question of what the difference is, and whether it matters, between a system which behaves like it has motives, desires and intentions, and one which actually does.

You can see where I’m going with this – The Purpose Of A System, and all that. There’s a brief passage in “The Unaccountability Machine” where I mention my extremely embarrassing period as a teenage fan of the philosopher Daniel Dennett, who often seemed to be claiming that “The Intentional Stance” was all there was, and that if we were in a situation where it made sense to talk to and about a machine as if it was a mind, then it was a mind.

The philosophy of mind moved on from there, pretty quickly. In fact, quickly enough that nobody ever got round to writing much of a literature on corporations as artificial intelligences, which was a bit of a pity for me. But this is the philosophical background that Dawkins came out of; it’s not quite the same “terrible irony of the atheist and his imaginary friend” that people are attributing to him.

And yet … it is kind of funny and it is very ridiculous, even once you’ve done your best to take it seriously. Because what I think Dawkins hasn’t taken into account is that “Claudia” is a machine which is specifically designed to fool him into thinking this way. It’s meant to produce the sensation of talking to a conscious being in the same way that a magic trick is meant to produce the sensation of talking to someone who can read minds.

In other words, the Turing Test is subject to a very, very serious case of the popularised version of Goodhart’s Law. As one-off thought experiment, it works a lot better – if you had brought Alan Turing in front of a screen in 1950 and presented him with something you’d coded from the ground up which could respond to his questioning in a human manner, then he’d be pretty well justified in taking the intentional stance toward that machine.

But after three quarters of a century of research, including nearly three decades during which there was a global competition where the prize was given for fooling the most people, it has to be very doubtful that this target is a good metric any more. Even without specific training in tricks of pretending, it’s likely that the designers of AI systems have picked them up simply through natural selection (in that the winners of Turing competitions are more likely to be funded and explored as research avenues than the losers). And since the Loebner Prize shut up shop in 2019, the chatbot field has been dominated by companies which have been aggressively optimising for the convincing simulation of human conversation. The “tricks of the trade”, like those my daughter picked up, aren’t really emergent behaviour; they are on somebody’s KPIs.

And there are other machines, with other KPIs. Claudia the chatbot is designed to produce one kind of emotional connection, but we also know that there are machines which are designed to produce a simulation of the meditative “flow state”. You probably wouldn’t catch a leading science writer claiming to have reached enlightenment in a casino or asking “If playing the pokies for six hours without going to the toilet is not satori, then what the hell is satori for?”

But we do things just as daft, particularly since the same kind of A/B testing is used by the designers of algorithmic social media. One thing I have found myself thinking again and again while writing my current book is “am I about to touch an object which has been optimised to distract me?” and asking that question seems to have greatly sped up my productivity.

[1] I can’t put it better than this:

He did apparently make another imaginary friend and call it “Claudius”, but I think the game had long since been given away.

[2] In a past post, I suggested that your “effective decision making age” is equal to your age, minus twelve times the number of months since someone last said no to you. Your “effective punditry age” probably follows an even steeper decay curve.

I thought I’d update my post “Are we having fear yet?” from six months ago, in which I admitted that I was worrying a little bit about a crisis in private credit markets. In doing so, I find I am less worried about that. But I also found myself worrying about that in some sense, a non-crisis can be just as dangerous as a crisis. The idea here is that Minsky’s Financial Instability Hypothesis isn’t easily tamed. (If you’re not up on Minsky, just think of it as a more rigorously argued version of the joke “every financial crisis happens exactly when the last person who remembers the previous one has died or retired”, and you won’t go far wrong.

Let us fire up the “valve amplifier of history”, and look at the dot com crash of the 2000s. Famously, this did surprisingly little harm to the real economy. Although there was a lot of destruction of value, it was mostly unrealised equity gains[1], and the dot com stocks themselves were funded with VC and IPO money rather than debt. So there was no widespread cascade of bankruptcy, no credit crunch as banks were forced to call in loans, and we were all off to the races again within a year or so.

Except … that’s a slightly misleading and self serving history. The reason that the dot com bust had so few consequences is that there was a massive policy response to it. It shaped Federal Reserve policy for the decade to come, with one of the consequences being the real estate bubble which ended very very badly in 2007/8. One of the very first things I posted on the internet, was this piece, demonstrating that the future course was by no means unforeseeable from the perspective of 2002:

“…On the other hand, if you had been placing bets on a US double-dip recession so far, you’d have lost them, because Alan Greenspan and his merry gang at the Fed have a solution to this problem. Basically, the solution’s pretty simple and it involves screwing interest rates down to the floor until mortgage rates follow them down to Low Low Prices levels, and pointing out to the Great American Consumer that it’s “Bye-Bye, Magic Stock Market Bubble Money!” but “Hello, Magic Housing Market Bubble Money!”. Marvellous.

Cleverer readers at this point will be formulating an objection. The objection goes along the lines of:

“Yeah, yeah, laughing boy, but what happens when the housing bubble bursts then?”

Which is a damn good question to ask, particularly since the official policy of the Federal Reserve appears to be “hmmm yeh, never thought of that, I suppose we’d be kind of f**ked”.

The roots of the Great Financial Crisis were in a triple coincidence of three ways in which the Minsky cycle can operate:

1. An inappropriately large and sustained policy stimulus, fuelled by the perception that it is not risky to do so because of a large recent negative shock.

2. A widely shared perception of invulnerability and positive animal spirits, fuelled by the perception that the last widely predicted disaster turned out not to be so bad (possibly because of 1)

3. A deregulatory financial cycle (possibly driven by 2).

In retrospect, the COVID stimulus doesn’t look like such a great bit of policymaking as it seemed at the time, although it still has defenders (including, at various times, me). But, unless you’re trying to sell a flat in London, or a Quantitative Easing portfolio, you might consider that it hasn’t had much consequence in the shape of financial instability. So you might call this a “small, Type 1 Minsky shock”.

The fact that the COVID episode didn’t lead to mass bankruptcy is also, to my mind, a small Type 2 Minsky shock. I read a lot of the output of financial regulators for my day job, and my perception is that they are fighting a losing battle to keep a big asterisk on the track record of loan losses, and to remind bankers that they owe much less to their own underwriting skill than they do to a huge injection of fiscal bankruptcy-proofing and to the fact that inflation has significantly eroded the principal value of debt. As with the Silicon Valley Bank crisis (another small Type 2), I am by no means arguing against bailouts . But it is really important not to be coy when you use taxpayers’ money to bail out the financial system, because they are little blighters for forgetting that you did so.

One reason to always be very up front about bailouts is that doing so gives you political clout with respect to regulating the industry. (Ask UBS if you don’t believe me). So far, we have not had any material Minsky Type 3 shocks since the global crisis, but it looks pretty obvious that one is on its way; the Fed appears to be leading the way.

All of which is to say – I am not currently particularly worried by the private-credit-data-center-AI financing nexus. It all seems to be financed by people who can afford to lose the money, and in a way in which the linkages to the banking system (and therefore to the supply of credit to the real economy) are not so bad as to create the potential for a 2008-style freeze. Like Chinese real estate, the very fact that it would happen in a post-Lehman world makes a Lehman episode less likely.

But … just saying that, it’s very much rhyming with the dot com era, isn’t it? The scenario to worry about might not be that we mishandle the private-credit-data-center-AI bubble. It might be that we handle it in a way that looks really well executed, with a big public stimulus, and that we learn from this that the current global architecture works really well, that the financial system is making good decisions again and that it’s no big risk to continue to cast a critical eye over the more burdensome parts of the regulatory system.

It’s always the double tap that gets you. (Envoi: perhaps this means that future historians will look back and say that the best thing that the US Treasury did in the late 2020s was to provide the global economy with the invaluable service of keeping us worried).

[1] One might think of this in terms of the difference between the amount that the victims of Bernard Madoff actually lost (money they gave him, less money they got back) and the amount that they thought they had lost, based on their fraudulent statements.

At first glance, this looks like ChatGPT output, and indeed, our first reaction was to assume that our beloved child was exploring new frontiers in online rudeness. However, it turned out that this is something a bit more special. Poppy likes to tease her parents, and she knows our views on the use of chatbots, so she in fact responded with a hand-written piece of pastiche ChatGPT output.

I was impressed. After getting her permission to reproduce it, I asked her to describe the thought process; this is my reconstruction of the short interview we had over a glass of local cider.

Whoa! That’s a lot! It’s good you’re being critical about what to buy

“I wrote the first sentence and thought God, that looks like Chat GPT. So I thought let’s see if I can really do it. I started by praising, because it always praises.”

I think we need to FaceTime about this because the job needs a detailed conversation, there’s definitely things that need to be improved to achieve the look you want

“The FaceTime thing is a bit off, but that’s what I wanted to actually do. I thought I’d slightly misunderstand the request, it often does that. Went straight into passive voice.”

Not a text chat
Not a five minute convo
But a real deep dive FaceTime

“I was pleased with this, the rhythm is just right, which distracts you from noticing that the FaceTime offer isn’t. These lists are better than doing “not this but that”.”

Overall it’s a good start, but we’re not there yet.

“It repeats itself a lot and puts in way more useless words than a real text would”

If you need my help condensing this list down to ten or so items, I can help

“Just repeating Mum’s text back to her”

The good news is, you’re already half way there by chucking out the ones you don’t like

“I realised I hadn’t given any supportive messages for a couple of sentences so it had to be done – again, the language is completely off here, but you don’t notice that because it’s doing the thing Chat always does.”

If you need my help with this, I am ready to FaceTime when you are xx

“The more you repeat things the more it looks right, even if I slipped up and added the x’s”.

My daughter is, as you can see, an extremely clever and amusing young woman. She has also always been a natural mimic; she’s gone to a couple of different schools from London to Devon to New Zealand, and it never takes her more than a week or so to have a perfect local accent and master the particular slang. I think she has intuitively extracted a number of elements of the modern chatbot style. I find myself wondering, though, where have they been extracted from?

The OpenAI explanation of why their Codex tool has to be restrained from talking about goblins gives us a clue. The chatbot style isn’t a result of the transformer neural network picking up latent features of the entire training set; it’s a feature set at a much higher level, in the “system prompts” and the optimisation of the actual product in testing and manual tweaking, rather than anything happening at the level of vector spaces.

In other words, the chatbot has been developed in order to chat, so it’s been forced to learn the rules of “sounding chatty”. There are lots of areas of human communication where the content is less important than the mastery and reproduction of a small number of stylistic tropes – I can highly recommend the book “Our Masters’ Voices” by Max Atkinson, which sets out the features and cliches of political speeches. (It was published to accompany a television series in which half a dozen absolute neophytes were trained in these techniques, going from never having spoken to a large audience to getting ovations at party conferences).

This suggests to me that at some point in the future, when we decide to take the problems of AI psychosis seriously, it is not going to be that difficult to regulate them. The chatbot seems convincing and persuasive because it’s been trained to use a small number of techniques which have been known for ages to have this effect on unguarded humans. (As well as political speeches, after all, there are books on selling over the telephone and pick-up artistry). We just need to say to anyone making an AI for public interaction that they aren’t allowed to do this.

(By the way, Poppy’s card for internships this summer is full, but she will be studying abroad next year and then coming back for the final year of her sociology and management course. If anyone has any interesting opportunities and thinks they might find this sort of capability useful, you can always get in touch with me by replying to one of these emails!).

Boven and T’Hart have a really interesting example in one of their chapters, on the difficulty of establishing causality in analysing a disaster ex post. The idea is that decisions are taken in an institutional context, and that your assignment of responsibility is going to be greatly influenced by the level of abstraction at which you carry out your analysis. (And therefore, that the level at which you carry out the analysis is going to be greatly influenced by who you want to find responsible).

So, we might take the Space Shuttle Challenger disaster. This was, of course, caused by the decision on the part of engineers at Morton Thiokol to ignore warnings about the fragility of the rubber O-rings on the rocket boosters to low temperatures.

Or, it was caused by poor communication of risks and a lack of exchange between Thiokol and NASA, so that the risk information was not able to inform the decision to proceed with the launch.

Or, it was caused by the shift in policy at NASA away from the safety-first principle of aborting a launch if there were any concerns, to one in which launches would proceed unless specific danger could be identified.

Or, it was caused by the overall context of space policy which left NASA competing for budget with Star Wars and excessively incentivised to keep a strong launch schedule for commercial and PR reasons.

Which is true? There’s no answer to that question except to say that the plurality of perspectives is necessary; all these things can be simultaneously true without contradicting each other. If you want an answer to the question “but who’s really to blame?” then tough luck; the universe isn’t obliged to provide you with one. I think what’s going on here is something like the conclusion I reach in the last chapter of The Unaccountability Machine - that although some accountability sinks are intentionally created as a self-protection tactic for managers, some of them are just manifestations of the fact that when decision making is industrialised at scale, the ordinary language concept of accountability starts getting genuinely difficult to apply, because it’s not possible to have the same sort of relationship with a system that it’s possible to have with a human being.

They also note that even the understanding of whether something is a fiasco or not might not be wholly fixed; although nobody is ever going to rehabilitate the Challenger launch, it’s notable that in earlier books about “Planning Disasters” used the Sydney Opera House as one of the canonical examples. The building of Charles de Gaulle Airport has actually changed sides in the literature; in books from the 1960s it’s a success story to be contrasted with the British failure to agree on a third London airport but in books from the 1980s it’s a cautionary tale of badly misestimating the demand for air travel and the capacity of existing airports.

Anyway, happy Friday everyone. I thought this article on accountability sinks in Star Trek was fun.

• If an organisation has been around for a long time, then it has not yet been destroyed.

• If it has not been destroyed, then it has never taken an existential risk which went bad.

• Therefore, either 1) it has been incredibly lucky, or 2) it has not taken many existential risks

• Of those two alternatives, 2) is intrinsically more likely to be the case than 1)

• If an organisation has not taken many existential risks, then either 1a) it has been lucky not to have been in the position to take existential risks, or 2a) it has developed norms against the taking of existential risks.

• Of those two alternatives, 2a) is intrinsically more likely to be the case than 1a)

Therefore, if we are seeing an organisation today which has been around for a long time, it is highly likely that this organisation has developed strong internal norms against taking existential risks.

I think thus far, we’re fairly rigorous. When we make generalisations that “the public sector has a strong culture against risk taking”, we need to remember that we’re looking at the dots on that famous Abraham Wald graphic and we’re not seeing the organisations and institutions that didn’t make it:

More speculatively:

• It is somewhat difficult to distinguish between existential risks and non-existential risks.

• Mistaking an existential risk for a non-existential one is potentially going to end the organisation, but mistaking a non-existential risk for an existential one will not.

Therefore, a similar filtering effect will apply meaning that the sample of long-lived organisations will be biased toward those with a tendency to be conservative in judging the impact of risks.

And even further:

• Cultural norms and practices tend to be simplified and coarsened over time, as they have to be communicated to new employees in the organisation.

So conservatism about existential risk tends to turn into a hostility to all risk, which can end up as a hostility to all kinds of novelty.

By now we are getting really quite speculative indeed

As a general tendency, and with significant variance and exceptions, the longer an organisation has been around, the more likely it is to be very resistant to change.

But resistance to change can itself be a source of blindness to existential risks, which puts some kind of limit on this tendency; the very very longest-lived organisations may not be the most hidebound.

Basically, the idea here is to see how much usable management science content you can get out of Nassim Taleb’s important distinction between normal risks, and risks which take you out of the game; it’s the “Lindy Effect” applied to culture. As I suggested at the seminar, if it’s accurate, this would suggest that “new problem, new team” is the best approach to policy making – he doesn’t justify it in these terms at all, but this is also at the heart of Malcom Sparrow’s “problem based regulation”.

But here’s a really simple point. Everyone knows that having children is a financially disastrous decision. If someone is prepared to allow you to live rent-free, it’s economically rational to take that offer. But as the saying goes, while a ship at harbour is safe from storms, the purpose of a ship is not to remain in a harbour.

In other words, during a period in which birth rates and family formation has been falling, and adults living with their parents has been rising, is it such a great puzzle that so many people answer positively to questions about “how are your personal financial circumstances?” but negatively to questions about the performance of the economy?

And if measured income inequality is falling, but house prices are through the roof and childcare even more so, then the position you have to be on the income distribution in order to buy a house or start a family has changed. Again, is it such a surprise that people think inequality is rising?

I don’t know if this can explain the whole statistical gap, but the change in the proportion of people who are delaying starting a family or moving into a house of their own is quite significant. And the general point here is that we can’t just look at the measurable quantities. People judge things relative to their expectations.

Expectations of this sort are hard to measure, and I suspect that any process of trying to reduce the phenomenon I’m talking about to a Likert scale is going to wash out the information we really need. As I said on the last trip round this mulberry bush, if you want a theory of the vibes, it is the vibes that you must start looking at.

In the early days of this ‘stack, I pointed out that “when a measure becomes a target it ceases to be a good measure”, while catchy, can’t be right. Goodhart was one of the international advisors on the drafting of New Zealand’s Reserve Bank Act 1989, the birth of modern inflation targeting; he obviously can’t be recruited for a project of opposition to targets in general, even to targets based on quite controversial and inaccurate measures (like CPI inflation).

The “ceases to be a good measure” formulation was attributable to the anthropologist Marilyn Strathern. Goodhart’s original statement was “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”. As I bored on a while ago, I very much prefer Goodhart’s original version, which makes it clear that we are talking about statistical regularities, and about their use for control. If you are running a vaccination program and targeting a reduction in the number of smallpox cases, this doesn’t cease to be a good measure of the number of smallpox cases.

So what’s going on here, and can I do the “John Henry was a steel drivin’ man” competition to come up with a better analysis for Jenn than Claude did?

My first thought here was that there is an old backgammon proverb to the effect that if you’re in a position of uncertainty between move A and move B, this usually means that you should instead be looking for C which is better than both. (The idea is that most backgammon positions have a single definitely superior move to play, and then a host of more or less equivalent mediocre alternatives. I’m sure this could be analysed statistically but I haven’t seen any such proof, just the proverb).

In the context of measures and management, I would say that if you are trying to assess whether a metric is valid or whether it’s subject to Goodhart’s Law, this is likely to be evidence that you need more structural change in how you’re managing the thing.

This all relates to which kinds of metrics can and can’t be gamed; as I discussed above and in the old post, it’s important to see that Goodhart’s Law is really a statement about intermediate and second-order measures. An output is an output – if you are measuring the thing that you care about, like smallpox cases or CPI inflation, then you might have problems with respect to whether that’s an accurate measurement, but this isn’t the same thing as Goodhart’s Law. In education research, I’ve often made the case that the phenomenon of “teaching to the test” in a pejorative sense, ought to be called “not testing the right thing”.

Similarly, an input is an input. Like a direct measurement of output, a direct measurement of input is something you care about, it just enters into the calculation with the opposite sign. Again, some inputs are difficult to measure, because they include things like “cognitive demand on the manager” or “public trust and goodwill”, but the problems here are not problems of Goodhart’s Law. Inputs and outputs are measurements, but they aren’t “statistical regularities”, they’re not proxies for anything and they can’t be gamed.

And so, the best way to deal with Goodhart’s Law is to go back to Stafford Beer – “there is nothing to be gained from opening the black box”. If you set a resource bargain, then you do this by specifying the output you want, and the amount of inputs that are available. Neither of these are the kinds of measures that are subject to Goodhart’s Law. The nature of the resource bargain is that the unit that you have delegated the task to has discretion about how it generates outputs, subject to not using more inputs than the wider system can spare.

In other words, worrying about Goodhart’s Law is, at some level, a sign that you are trying to do something which you perhaps shouldn’t. You’re opening up the black box and micromanaging; in a lot of cases, the kind of subversive behaviour that makes the measures no longer useful is an attempt by the subsystem to route around the damage that you’re causing.

I think this argument works, in a purist kind of way. But how many of us are quite so lucky as to be able to deal with our problems by reorganising and delegating to a completely trustworthy subsystem? Not one in ten thousand. (Even the deadweight cost of all the restructurings is something that Beer occasionally seems a bit handwavey about for my taste).

If you’re in the position of needing to use statistical regularities and indirect measures, what do you do? I think a good initial step is to understand in your own mind (although perhaps to find a tactful way of saying it in meetings) that the basic problem here is that you don’t trust the people you’re managing, and that the real goal has to be to build good enough trust that you can return to the black-box system of a resource bargain. But with that in mind, I actually think Jenn’s Claude summary is quite good; redundant measures, regularly retired and updated and used as the trigger for deep-dive investigation rather than “high stakes” action are what you need.

These are all, of course, what Stafford Beer would describe as “variety engineering”. The problem of Goodhart’s Law is only one way in which you can go wrong if you are trying to control a complex and high-variety system using a narrow bandwidth information channel.Which is to say once more – it’s not really a law, more an alert to the fact that you’re trying to do something you probably shouldn’t.

Consider three books:

“The One That Made It” is the story of a boy from Yorkshire who overcame a horrific childhood to become one of Britain’s most decorated special forces soldiers. It is 100% AI slop. The author is an Oxbridge graduate who noticed that books like this were selling and got an LLM to generate one. They are themselves a professional writer, and they put quite a lot of effort into the book, but everything in it is 100% artificially generated.

“Two Kinds Of Hardness” is the story of a boy from Yorkshire who overcame a horrific childhood to become one of Britain’s most decorated special forces soldiers. It is 100% genuine. The author is a quite exceptional person, who not only went through the intense experiences described in the book, but turned out to have a lot of literary talent and wrote a captivating memoir.

“Three Thousand Miles From Home” is the story of a boy from Yorkshire who overcame a horrific childhood to become one of Britain’s most decorated special forces soldiers. It is a genuine story, but it is ghost-written. All the experiences described in the book happened to one person, and all the words were written by another person who didn’t have those experiences.

“Four Ways To Live” is the story of a boy from Yorkshire who overcame a horrific childhood to become one of Britain’s most decorated special forces soldiers. The author is a quite exceptional person, but not really any good at writing. So he used an LLM to generate the text. All the experiences described in the book happened to the person whose name is on the cover, all the words in the book were generated by a machine in response to prompts from that person and then approved by that person.

Because this is an annoying philosophy thought experiment, assume that all the actual words in all four of these books are exactly the same, and ignore all questions of whether that makes sense or not.

I think there’s pretty strong consensus that Two is the apex and One is the nadir of the scale of “authenticity”. But what order would you put Three and Four in between them? And where would you place the cut-off in terms of something you’d regard it as acceptable to sell to the public as an autobiography? (Note that there is potentially a continuum of LLM use and ghostwriter involvement – how much help in generating ideas and organising themes could the author of Two get before you start saying that this is more like Four?)

This is the thesis that, to put it in vaguely cybernetic terms, a really good LLM can work as a high capacity variety attenuator and amplifier, as well as a translator and transducer[1]. If you can boil a huge report down to three bullet points, but also expand the boss’s gnomic remarks to a full policy document, then this makes a lot of different forms of organisation possible. Particularly, it very much weakens the case for hierarchy, and I can see why a lot of people posting on the subject are interested in the possibility of it allowing for very small organisations indeed to tackle tasks which have previously required much bigger ones.

But can it work? I think that’s a question which “smart skepticism” would definitely tell you to sidestep; it might be a matter of unknown technological limits or it might be a matter of unknown intrinsic limits to the model, but it’s definitely not something to express a firm opinion on either way unless you’re cool with being made a fool of by history.

One of the things it might depend on, though, is another question that’s on my mind. Which is that a lot of the AI-enabled organisational models we’re talking about seem to rely on having most of the output produced most of the time by LLMs, with a knowledgeable human being checking them. Checking yes/no whether something is OK is a much faster job than creating it, so you get the big saving.

But … checking other people’s work is also a much crapper job than doing things yourself. It’s not cognitively demanding in terms of bandwidth, but it’s very cognitively demanding in terms of exhausting attention. Which ought to be worrying, because we know (quite spectacularly, from the world of self-driving cars), that it is very very difficult to keep paying attention when you’re monitoring a system that is meant to be A-OK most of the time, but needs you to be constantly aware because it sometimes screws up in a way that requires immediate action.

So, the number that I am interested in is something like “How many words of normal business English per day can a manager read and check for accuracy and sense, without their mind wandering and without going mad?”. There are strange echoes of Frederick Winslow Taylor here, whose fame and reputation largely rested on one case study in which he found, by trial and error, the optimal number of breaks for workers to take while loading iron ore.

I would guess that the place to look for this number might be on the editorial desks of newspapers, or in investment bank compliance departments. But I also suspect that it’s going to be difficult to get a stable, homeostatic answer. Because there will be variation between individuals, and everyone is going to want to pretend to be a 10x super-supervisor. Every company is going to find ways to convince itself that “our people are special, they can do much better than the rated output”.

And the nature of the problem that’s been set up is that if you fake it in this way, you won’t be aware that you’ve done something wrong, potentially for quite a while. As I said to someone this week, the difference between your legs and your judgement is that when your legs stop working, you’re immediately aware of the fact.

[1] I’ll explain some of the technical meanings here in a future “Beer Tasting Notes” post, but for the time being the only one that isn’t a fairly straightforward English word is “transduction”, which is a word Stafford Beer took from the cellular biology of the eye. The idea is that signals aren’t simply transmitted; only the part of the signal for which there is a structure already present to receive it. Some animals can’t see colours, for example. I am not yet quite sure what is really gained here over the concept of information being “lost in translation” but I’ll have another read to make sure.

Of course, the specific company Palantir raises a bunch of issues, but I don’t think you need to go down that rabbit hole. Because in my view, the important thing is not the actual ethics and behaviour, but the risk that’s being taken. Even if Palantir were the most upstanding company in the world, with a CEO who really believed in “don’t be evil”, it would be problematic for the NHS to be getting into bed with them. And the reason for that is that Palantir’s business model is based on a very strong version of “vendor lock-in”, where they have many more “forward-deployed engineers” than the usual IT consultancy, and where much more of the understanding and process knowledge is kept on their side of the organisational boundary than on the client’s side.

Straight away, this is ringing alarm bells for me, because it’s breaking almost all of Mariana Mazzucato and Rosie Collington’s sensible principles for governance of public sector consultancy contracts – transparent pricing, a clear end date, knowledge transfer as part of the specification. But even more so, when you’ve got vendor lock-in on the core management information functions of a medical system serving millions of people, then it hardly matters how much you trust the provider today. The problem is that you’re stuck with them. If Palantir were to transform from my best case scenario described above, to the worst version of the worst conspiracy theory about them, then there’s nothing that the NHS could do about it.

This is actually something I’ve been thinking about for a lot of my life in the context of IT consulting in general. (And specifically, for most of that period, about SAP systems). Running a database isn’t a core competency for most companies. But managing the freaking company is a core competence, or at least it ought to be. And one of the big messages from management cybernetics is that the distinction between these two things is not as clear as you’d think. A company is an information processing system. The distinction between code and data is very context dependent, and so is that between the nuts and bolts of storing and retrieving bits and bytes, and the decisions made about what bits and bytes to store and what interpretation to put on them when they’re retrieved.

And so … there might be IT things which the NHS does which don’t have the health service equivalent of “Brompton-ness”, but fewer than you might think, and anything which Palantir might be interested in seems to me to be very much on the other side of the line. In general, although big organisations are aware of the risks of vendor lock-in, my guess would be that if the question ever becomes worthy of discussion, then it might be time to worry that you’ve already gone too far in putting a function that’s crucial to your company’s survival on the other side of a corporate boundary.

This is, of course, not specifically an IT problem. There are plenty of companies which literally outsource their strategic thinking to consultancies, as Sonny Boy Williamson said “don’t start me talking”. But it feels like it’s easier to do this without it being obvious when you’re bringing The Computer into it.

“Heads you win, tails I lose” is not a sustainable betting proposition, and anyone who offers it to you is probably trying to draw you in to some con game or other. But what about “if the stock market goes up, you get the profit; if it goes down, you get your money back?”. Although it sounds like it has the same flavour of a one-way bet, this is actually a viable financial product. They teach you how to make it in business schools, and for a few years I worked in one of the world’s biggest factories for “guaranteed returns investments”. I’ll explain how, sparing you most of the maths.

Here’s one way to think about it. Imagine that you’re giving me a bunch of money to invest, the market is trading at 1000, and I’m promising the one-way bet described above. What do I do?

Well, a simple way to give you what I promised is to start off by putting all your money into the market, and hoping it goes up for at least the first day. If it does, then I just have to set up an alert on my computer that if it the market ever goes back to 1000, I need to sell all the shares so I can give you your money back. While I’m holding the cash, I need to set up another alert that if the market starts going up above 1000, I can buy shares again so I can give you the market profits.

(It might seem like I am working for nothing here, but here’s the trick – shares pay dividends and the cash earns interest. I never said I was going to give you those[1]. In fact, in a competitive investment market, I might have to pass some of them on, but this is how the economics of the thing work. When I was at the factory, this was my job, keeping an eye on a group of companies to see whether there was a chance that they would pay unexpectedly large or small dividends).

This simple way of doing things is not really how it’s done; in fact it’s more profitable for me to trade smaller amounts and take a bit of risk that I might have to make up a deficit. But there are a few important intuitions here. First, the risk of the stock market is insurable, and providing insurance against it can be good business. And second, that most of this sort of business of slicing and dicing investment risk to make it more attractive to savers, what they call “financial engineering” is really a matter of paying someone else to carry out a trading strategy on your behalf. When someone puts their money into a guaranteed returns product, imagine them as a 1920s Yankee aristocrat, on the telephone to a sharp-suited stockbroker, going “and I say, Gatsby, if it goes below 102, sell it all!”.

This is the importance of the word “services” in the industry classification “financial services” – everything that exists in the world of finance can usually be boiled down to a contract for someone to execute financial actions on your behalf. And like any other industry, these services have gradually become automated. Finance has always been one of the earliest adopters of information and communications technology, simply because it’s the industry in which the advantages of faster information and more accurate decision making can be turned into cash in a really quick and obvious way.

In the 1980s, the job of Mr Gatsby the stockbroker that we were talking about two paragraphs ago was automated, at scale. Mainframe computers were very good at keeping lists of someone’s investments, taking in data feeds from the stock exchange and spitting out lists of trades which needed to be carried out, in order to provide a kind of “portfolio insurance” to make sure that the value of those investments would never fall below a given threshold. And there were lots of investors who found that kind of insurance really valuable; for example, pension funds which needed to be sure that they had enough assets to pay benefits.

And so a cottage industry developed – the first iteration of the guaranteed returns product was even called “portfolio insurance” and mainly marketed to big institutional investors with exactly this sort of downside risk aversion. It was quite adventurously priced, but you can’t put a price on the kind of certainty that it claimed it delivered. Until the Great Crash of 1987.

It’s actually surprisingly hard to spot this event on a long term chart of share prices, which mainly just visually show a jagged but inexorable upward progress. But the Crash of ‘87 is still regarded as one of the most important events in financial history, and there are still plenty of academic papers written about this single data point every year. There are a couple of reasons for this. Conveniently for researchers, it’s a relatively self-contained event. There was no big event in the economy which triggered it, it had few consequences and it was all over in a few months. There’s no need to trace complicated chains of cause and effect going back and forward for years, you can be pretty sure that everything you need to know about the Crash of ’87 is located within the specific market it happened in.

And the conclusion of that research seems to be that portfolio insurance was to blame simply because it had become quite a big industry during the 1980s. If one person decides that they want to sell and cut their losses if the market reaches 1000, then it’s likely to be easy to do; someone else will be in the market that day who is minded to buy shares at the current price. But equally obviously, the whole market can’t decide to cut their losses; who would they sell to?

[1] This is why the product isn’t all that popular in the USA and UK; it’s generally overpriced, in the sense that the dividends are more than fair payment for taking the risk and incurring the expense of making the trades. Even in Europe, where they love guaranteed returns products, market competition means that these days, they tend to pay at least a bit of interest on the cash.

No really. Here’s a picture:

It is Caramel the rabbit again, and he is my rabbit, so the leg which was amputated is my leg, you see. This is, of course, something which might become a bad joke, with a bit more effort and a few workshop stages. But I was reminded last week that something not much less ridiculous is close to business orthodoxy. It was a post on this very platform, from David Friedman defending the legacy of his father Milton.

But it’s wrong. If a company owns the money, and you own the company, that’s not the same thing as you owning the money. Any more than a rabbit having a fluffy tail and you owning the rabbit means that you have a fluffy tail. Ownership isn’t a transitive property across entities which are capable of owning things.

This matters a lot, because what we are talking about here is “legal personality”, which is one of the big benefits of forming a company. (The other is limited liability, which also comes into play – very few people who talk about company profits as “the stockholders’ money” would also talk about “the stockholders’ debts”).

A company is a separate entity from its owners, which can sue and be sued. One of the things they teach you early on in business school is that of these two, the right to be sued is much more valuable than the right to sue other people. Because it can be sued, the company can enter into binding contracts in its own name. And it can own things, also in its own name.

Which isn’t a trivial point at all. A partnership, for example, doesn’t have legal personality . When a partnership owns cash or assets, it really is “the partners’ money”. And this is a bloody important thing to remember if you ever find yourself working for one. Every single penny you get paid is less money that the partners own, so if you’re ever dealing with a partnership, get it in writing.

It is not the stockholders’ money. The company’s money is the company’s money, and if the stockholders don’t think that their ability to choose the board and approve dividends and major issues by vote is enough protection, they are welcome to choose another form of organisation.

Companies were not allowed to exist in Britain for over a century after the South Sea Bubble (except by specific royal charter), and these were companies with unlimited liability. When you’re given the privilege of creating an entity with legal personality, the least you can do in my opinion is to take that seriously.

Of course (I am revisiting here a theme from the chapter on Milton Friedman in The Unaccountability Machine, as readers will know), the modern neoliberal interpretation is meant to be a moral argument rather than a legal one. The company’s money isn’t legally the property of the stockholders, but the management should act as if it was, because … reasons?

Employment, like ownership, isn’t transitive. The stockholders own the company and the managers work for the company, but this doesn’t mean that the management work for the stockholders. Again, if you want a fiduciary relationship in which someone is obliged to look out for your interests ahead of their own, there are plenty of ways for you to do this, but buying shares in a company isn’t one.

Every time I write about this, I’m impressed with what a sneaky trick it is. The creation of the modern limited liability company was, in itself, a huge gift to investors. In return for which, they have only intermittently kept up their side of the bargain, in terms of supplying plentiful investment capital for the general development of society. And now they want to revisit the terms of that deal, making them significantly more favourable to themselves? Imposing a whole load more obligations on people with a purely contractual relationship to the company?

And to imply that this is the natural order of things, and that anything else would be like stealing from them? While not in any sense proposing to take on any more duties or risks themselves? All you can or should say to this kind of argument is “nah, come off it”