Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:39 - What functional programming is

09:18 - Downsides of functional programming

10:53 - Specialized hardware for functional programming

21:47 - Haskell is useless

25:59 - Rust vs C

28:26 - Haskell vs OCaml

35:26 - Side effects in Haskell

44:26 - Type systems

57:30 - How the Haskell compiler works

01:04:35 - Why Haskell is talked about more than used

01:09:07 - Avoiding success at all costs

01:11:12 - LLMs and programming languages

01:13:57 - New programming language design

01:15:59 - Should students continue to learn programming

01:22:33 - Why Excel is is 2nd favorite programming language

01:25:04 - Advice for his younger self

Transcript

00:39 — What functional programming is

Ryan:

[00:39] I wanted to start by asking you, what is functional programming in your words? And why does it matter to people who might use the other languages, the imperative languages?

Simon:

[00:57] Functional programming is about programming with values instead of mutation. So in a conventional imperative programming, computation proceeds by, if I want to add up the numbers between 1 and 100, I have a mutable cell containing a running total. And then as computation proceeds, I keep adding to my running total. So there’s a cell that changes its value over time. Computation proceeds step by step.

[01:23] There’s a program counter. Now, that’s very unlike mathematics. If I say three plus four times seven, you don’t think of a mutable cell that changes its value over time. You think, well, three plus four is seven, seven times seven is 49. Just the answer is 49. So, in a sense, it’s more declarative than imperative. Another way to think of it is it’s more like what a spreadsheet does. In a spreadsheet formula, if you put in a cell, you’d say, in A1, you put the formula =A2 * A3 + 7.

[01:55] Well, then that’s just the formula that is evaluated. And you expect to compute the values of A2 and A3 automatically before computing A1. It wouldn’t make any sense to compute A1 from the old values of A2 and A3, right? So, and there’s no notion of a program counter. There’s no notion of do this and then do that. It wouldn’t be sensible to say equals print X plus 3, because who knows when or even whether that cell will ever that formula will ever be evaluated.

[02:23] Now, the surprise is that it is possible to do useful things in such a limited language. When I learned programming, I learned it by writing machine code in which the fundamental mode of computation is registers and program counters and mutation of registers as you go along. And memory, like, computation and mutation seem to be inextricably interwoven, right? That’s how computation gets done. So it was a surprise to me when I learned that, well, it is possible to do useful things with functional programs.

[02:56] I still remember that amazing revelation. Now, in fact, one way to look at it is this: right back in the late 1920s, early 1930s, Alan Turing was doing a PhD in Princeton under the supervision of Alonzo Church. Now, Alan Turing, as we all know, invented the Turing machine, which is fundamentally about mutation, right? It has a little tape, and you can read things from the tape, and then you write back new things onto the tape, and it has an immutable program.

[03:30] Now, Alonzo Church at the very same time, was working on something he called the lambda calculus, in which there’s no mutation. The lambda calculus is like the essence of functional programming. Programs execute just by reduction. Now, so these were two different models of computation. You might ask, what can you compute with a Turing machine? And what can you compute with a lambda term? And the astonishing thing that turned out, which is completely nonobvious, is that anything you compute with a lambda term, you can compute with a Turing machine.

[04:02] And anything you compute with a Turing machine, you can compute with lambda calculus. So that means that the two are identical from the point of view of their computational power. Big surprise to me. Okay, so they’re equally powerful. And then the question is, which might you want to choose? As I say, I started life from my baseline of imperative programming. Of course, that’s the way you write programs.

[04:26] And functional programming is a kind of weird, anomalous, rather nerdy, academic thing. But I became hooked on the idea that by excluding side effects by default, i.e., imperative programming has side effects by default. By excluding side effects by default, we could make programs that are easier to write, easier to maintain, easier to reason about. And then the question is, what are the practical obstacles?

[04:53] Is it slower? Is it less convenient, and so forth? So, in effect, I’ve devoted my research life to exploring that hypothesis. The wonderful thing about being a researcher is you can take something which, at the time back in 1980, functional programming was very nerdy, geeky, academic. But I was allowed to spend my professional life just saying, what would happen if you took that one idea, programming with values, and just ran with it?

[05:23] Where would you get? And I think we got somebody somewhere pretty interesting. I don’t want to say that it’s better fundamentally, because after all, Haskell on that journey started with functional programming. Then we learned how to mix functional imperative programming together in a way that we might talk about later, right? So it’s not really an either-or; it’s more of a both-and in the end. But my position now is, as I often say, when the limestone of imperative programming has worn away, the granite of functional programming will be revealed underneath.

Ryan:

[05:57] And why do you say that?

Simon:

[05:59] Well, functional programming is the language of mathematics. That’s how we fundamentally think about things when we want to be formal. Indeed, if we want to reason about and say, what does this imperative program do? We have to wheel out a lot of mathematics and Hoare triples and so forth. But that’s all in the world of maths, which is just functional programming. Functional programming is just mathematics brought to life and made executable.

[06:22] So you might wonder, I suppose you may be saying, well, isn’t imperative programming just enough? And I think that there you have to just look at the broad sweep of things. I think what’s happened is that from the world of functional programming, lots of ideas have got adopted into the imperative arena, right? So they grew up over here in functional programming, and then they have been adopted by the imperative programming thing.

[06:54] So you might think garbage collection, you might think lambdas, you might think language integrated query, you might think about lots about type systems, polymorphism and static typing, tons of that. So at least it has been an intellectually interesting experiment that has been a very fertile laboratory in which to grow up ideas that can infect the mainstream. Now my instinct now is that we should start from functional programming and do imperative programming where necessary.

[07:24] I think probably the mainstream is we should start with imperative programming and sort of add bits of functional programming where possible. And I’m not going to say that they’re wrong to do that, just to say that I think it’s a pretty interesting dialogue. Now you do have to think very differently about the act of programming, right? You do have to rewire your brain quite a bit. But just let me.

[07:46] This is something we might want to come back to. I said easier to maintain. As programs get old. Think of a 10-year-old program, right? If you have a piece of 10-year-old code written by somebody who’s long since left your company that mutates some more or less global variables, then you might be a little bit afraid about, oh, and there’s other bits of code scattered about that also read or write those global variables, then you might be a little cautious about making modifications, right?

[08:16] Lest, oh, if I put these two things, if I do B and then A instead of A and then B, maybe the global variable that I read somewhere deep inside the code no longer has the value that it was expected to. So reading and writing shared variables that are visible outside the scope, sort of somewhat global variables, is a form of coupling between bits of code that is very invisible, very invisible.

[08:44] So functional programming forces that to become visible. It forces it into the open, and by the act of forcing it into the open, that strongly encourages a style of programming in which you don’t have such interactions because they are painful to manage. So it encourages a style in which programs execute in modular boxes without interacting. Now, every imperative programmer would want to do that, but I want them to do that in a provably secure way so you can know that these programs don’t interact, rather than just say I’ve done my best.

09:18 — Downsides of functional programming

Ryan:

[09:18] Then I guess on the flip side, what are the downsides of a functional programming language?

Simon:

[09:25] Oh, well, it can be very tiresome and inconvenient, right? So you’re somewhere deep in the, you know, a subroutine of a subroutine of a subroutine of a software stack, and you want to reach out and, I don’t know, grab the time of day. Oh dear, that’s a side effect. But we’re in a pure functional language, the time of day. And indeed there’s a good reason, because everybody else higher in the stack is assuming that if I call this function, apply to, you know, seven, it will give me the same answer if I call it again, apply to seven.

[09:56] But if it could interrogate the time of day, certainly it might give a different answer, right? So a simple thing as I was asking for, the time of day, invalidates a whole set of assumptions and so is not by default allowed at all. And that can be inconvenient because all you say, look guys, all I want to do is get hold of the time of day. Don’t give me grief. And so every functional programming language, including Haskell, provides you with a little trapdoor.

[10:21] Haskell‘s is called unsafePerformIO, and it says, trust me, I’m going to do something that has side effects, but you can pretend there weren’t any. All right, so it’s called, it is literally spelled unsafePerformIO, with a long name that has unsafe in the title, so that you, the programmer, are taking on the obligation to make sure that what you’re doing is okay. Whereas in imperative language, everything you do could be unsafePerformIO-ish.

10:53 — Specialized hardware for functional programming

Ryan:

[10:53] I was watching one of your lectures, and you did go over a little bit of the history of functional programming languages. And you mentioned somewhere that there was this idea of people building hardware for functional programming languages. This just made me curious because I didn’t. I only know the von Neumann architecture. Do you have any idea what that might look like? And some high-level thoughts?

Simon:

[11:20] Well, I did. I now think it’s probably a bad idea. But back in 19, when was John Backus’s Turing Award lecture? I think it was 1978 or thereabout. What was it called?

Ryan:

[11:34] Can Programming Be Liberated from the von Neumann Style?

Simon:

[11:38] That’s it. Now he was saying two things. Firstly, the designer of Fortran, a quintessentially imperative language, was saying, guys, we should do functional programming. That was what the talk was about. And he introduced a language called FP for functional programming. It was a very particular, somewhat idiosyncratic functional programming language. But there it was. So that was rather amazing, right?

[12:06] This was 1977. He was saying, guys, give up on imperative programming. Just do functional programming. Here’s how you do it. But he also said in the same lecture, we might imagine designing new hardware to execute this new kind of language. So if you like, we’ve got Turing machines, they give rise to register machines and microprocessors and so forth. Maybe lambda calculus. Perhaps if we designed a machine from the ground up to do lambda calculus, it would look different.

[12:37] So that was, of course, very exciting. And similar kind of thinking led to an actual commercial machine called a Lisp machine. That was a piece of hardware that was designed by Symbolics to execute Lisp programs. Lisp is a mostly functional programming language. And so, at the time, I was in my early 20s. We were thinking, oh crumbs. What would a piece of hardware look like that was primarily designed to execute functional programs?

[13:06] Would it have a different kind of instruction set and architecture? So the nearest we got to it actually was, although there was a big strand of work then that was called the MIT Dataflow project. So dataflow languages, functional languages, they’re definitely the same kind of category, right? And so MIT had this great group led by Arvind called the MIT Dataflow Group. And they designed, in the end, a machine called Monsoon that was specifically designed to execute dataflow programs, that is, functional programs.

[13:39] It was based on a token store and matching, and tokens flowing around and getting matched up and executing. You might imagine data flowing, going through a graph, when 3 arrives at a plus node and 4 arrives at the other input of the plus node. Bam. You could fire that plus node, right? And the hardware did all of that. At a similar kind of time in England, my colleagues Thomas Clark, Joe Stoy, and others built something called the SKI machine, SKIM.

[14:07] There’s a paper about this in the FPCA conference. And SKIM was designed to execute SK combinators. So at the time, David Turner’s written a lovely paper in which he described how to take lambda calculus programs and translate them into SK combinators. So you can read about that in my book. It’s a very simple translation. You can give the whole translation in half a dozen lines, but then you have this mess of S’s and K’s, and S and K reduction is very, very easy.

[14:40] So it’s like the machine code of functional programs.

Ryan:

[14:45] What is a SK combinator? Just for context, there are three combinators,

Simon:

[14:50] S, K, and I, and they have simple reduction rules. Here they are: I of X equals X, KXY equals X, S X, Y, Z is X of Z applied to Y of Z. End of story. That’s all so very simple, right? So now all I’ve got to do is take my lambda, translate it into a giant tree of SK combinators, right? S applied to K applied to I of 3 and S of Z of just a huge tree of S’s and K’s, right? Now simply apply the rewrite rules I gave you, and that is program execution.

[15:25] It’s rather astonishing that that can execute arbitrarily complicated programs, but it can. Don’t you think that’s rather amazing? Now, I can take an arbitrary Haskell program and could translate it into lambda terms, and I could translate those lambda terms into S and K. Very simple transformation. And I can just write those, run those three rules, and it’ll produce the output of the Haskell program.

[15:47] That’s pretty amazing. And indeed, there is an implementation of Haskell that uses exactly this. It’s called MicroHs, and my colleague Lennart Augustsson has built it. And its execution mechanism is SK combinator reduction. And if you sit in MicroHs, you can ask it to show you the S’s and K’s that it produces. If we had a shared screen, I could probably show you. So, long story short, then the SKI machine was designed to take SK trees, big trees of S’s and K combinators, and just execute them directly.

[16:21] So it was built in hardware and it ran perfectly well, reasonably fast. Now, why didn’t this catch on? Why don’t we have functional programming machines? I mean, at the time it was a big idea. We even had a conference called Functional Programming and Computer Architecture. FPCA was right there in the title of the conference. But slowly we became aware that what was really happening is we were doing at runtime what we could do instead at compile time.

[16:53] And that is always a bad idea. Imagine an interpreter for, I don’t know, Pascal or something, or for Java. We can compile Java to bytecode and we could interpret the bytecode, right? So then we’re doing something at runtime. At runtime, the interpreter is looking at the bytecode, saying, dispatching to say which instruction, blah, blah, blah, right? Now what does JIT do? It takes a sequence of bytecode and says, oh no, no, I’m going to take that sequence of bytecode and compile it to a sequence of machine instructions that, when executed, will do the same thing much faster because it can take advantage of bytecode sequences, for example, to say if you swap and then swap, it’s a no-op or something like that.

[17:44] So what, in fact, it turned out is this whole SKI business, and any other, and indeed dataflow machines also were simply doing at runtime what you could do better at compile time. We were building an interpreter in hardware. So don’t build the interpreter in hardware. Instead, build a compiler that translates your lambda term into a sequence of machine instructions that, when executed, will behave as if you had done this reduction business.

[18:15] And then you might think, well, machine instructions for what machine? Well, from a practical point of view, it was very hard to compete with Intel, right? They were just spending bazillions of man years on making x86 go faster, and Arm likewise. So we can leverage all of that work by compiling into their instruction set. But even if you say you are God, you are the chief executive in Intel and can tell them to add new instructions or computation mechanisms to support functional programming.

[18:48] What would you add? Not very much. Little bits to do with garbage collection, barriers, I think, and such like, but nothing major. So, in other words, I think the whole build hardware to execute functional programming directly thing turned out to be a mistake. Inspiring mistake, fun mistake. I had a great time. But a mistake. It’s better to build a compiler.

Ryan:

[19:17] I would have thought there might be some unique opportunities for parallelism given the immutability in these functional programming languages. So yeah, you described that graph or that SK combinator graph. I imagine you could execute that all in parallel, assuming there’s no connections across the graph.

Simon:

[19:40] And indeed, that was the dataflow machine. The MIT Dataflow machine was all based on that idea. So you throw the whole graph into the token store and just run all the nodes that are runnable.

[19:51] But that’s incredibly fine grained. You’ve got to imagine this big tree of a million nodes, and your processor is sort of wandering around over it, doing little things in parallel. There’s a lot of memory traffic going on there. And if these little threads are operating on the same bit of tree, then there’s synchronization costs. So very, very fine grained parallel computation like this turns out to be impractical, or not impractical.

[20:18] You could do it, but it’s very slow. So the dataflow people, their history, if you look at the history of the MIT Dataflow project, they started with micro parallelism. Every individual instruction was a separate parallel computation that might take place, and the token store would match it up. And then they built more and more compiler technology that grouped these things together into larger units.

[20:42] So they expanded from micro threads of single instruction threads into 10 instruction threads or 100 instruction threads. Right. But even that never really caught on. So you’re right though, that if I say in a Haskell program E1 plus E2, then I can do E1 and E2 in parallel. And GHC does support you in doing that. But nowadays, rather than expecting the compiler to do that automatically for every subexpression, you can spark a computation. You say E1 par E2, and that says do E1 and E2 in parallel.

[21:17] There’s a project at Carnegie Mellon for a strict power, a strict language called ML that has a very good variant of Parallel ML going in a similar way. So essentially, we’ve moved away from the dream of automatically getting parallel, very, very fine grain to the programmer. We give the programmer clues about where it’s a good idea to do parallelism, and still the compiler will try not to fork two tiny grains.

21:47 — Haskell is useless

Ryan:

[21:47] I was searching on YouTube and I searched your name, and this video came up. It said, “Haskell is useless.” Simon Peyton Jones, and it’s a video from a long time ago, and someone’s got a low-quality handheld camera. There’s a group of researchers that are all talking. Butler Lampson is across the table. And it’s a fun little video. And in the video you lay out this two-dimensional graph where on one dimension there is the useful versus useless axis, and then on the other dimension there is the safe versus dangerous.

[22:26] And then you’re placing programming languages on this two-dimensional graph. And I think you started by putting C as very useful, but it’s incredibly dangerous. And then you put Haskell on the polar opposite side. You said it’s useless, but it’s very safe. What are the least safe and most useful languages? And why do you place them there? C, for instance.

Simon:

[22:52] Well, yeah, so let’s see. In C, you program by mutation. So it’s unsafe in the sense that any function can mutate any variable at any time. It has a lot. You pass pointers around a lot, and functions mutate the memory pointed to by those pointers. And moreover, typically they can mutate it anywhere. There’s no array bounds checks or anything. So it’s kind of like super unsafe. And in fact, and this is demonstrated by the fact, can you imagine all of these exploits that we get every day, right, that Mythos is discovering in huge numbers?

[23:24] But why is the Internet so insecure? Primarily because all of our software infrastructure is written in unsafe languages. If we’d written all of our Internet software and operating systems in Haskell or maybe in OCaml or ML, 99% of all these exploits would be removed by construction. It’s like we’ve built a boat out of paperclips and we’re surprised that it’s leaky.

[23:55] I mean, you shouldn’t build boats out of paperclips, right, because they have holes in them. You should build it out of a secure substance, right? But then it’s too late. So we spend incredible amounts of human ingenuity and effort patching the holes in our boat built of paperclips. It’s tragic. It’s tragic how much effort and ingenuity and money has been lost and waste of resources just because we wrote our computational infrastructure for the world in an insecure language.

[24:25] That’s what I mean by unsafe.

Ryan:

[24:27] I mean, are there not vulnerabilities? If I wrote something in Haskell, there’s gotta be some new type of.

Simon:

[24:34] Of course there are vulnerabilities. I just said 99%, I didn’t say 100. If I write a Haskell program that says receive message, if the message says tell me everything, then spit out my entire database in reply, no language can stop you doing that, right? But if you look at the program and it doesn’t have any such things, right? So, I mean, nothing can prevent you against high-level attacks to insecure programs.

[25:03] Or, I mean, another example might be deadlock, right? Services, no matter how securely written. If A waits for B and B waits for A, deadlock. Sorry, no. And no language is going to stop you doing that. You might hope for some high-level verification tools, but it’s like surely if you’re trying to do something hard, like prove that that doesn’t happen, you want to have a foundation that is, you know, in which you’ve got some bedrock to stand on.

[25:34] If you’re standing on sand and trying to prove some advanced property, it’s very, very difficult. But good point. I’m not talking about 100% security. Absolutely not. But how many exploits are based on buffer overruns or pointer manipulation that’s gone wrong? If you couldn’t have a buffer overrun, you couldn’t do pointer manipulation gone wrong. Those bars just wouldn’t exist.

25:59 — Rust vs C

Ryan:

[25:59] So, I mean, C’s maybe half a century old now. What about more modern versions of those lower-level languages like Rust?

Simon:

[26:09] Much, much better. Much, much, much better. Right. If we rewrote all of our software infrastructure in Rust, things would be way, way better. I’m not actually even sure whether Rust has array bounds checks built in, but suppose. But it must have the ability to promise that you’re not indexing out of bounds. I don’t quite know quite now, but if you compile all your code with that switched on, you are in a way better situation, way better.

[26:37] So yes, this is not just functional programming. But you did ask about why I thought C was an insecure, unsafe language.

Ryan:

[26:44] If we imagine that graph, there’s the upper right quadrant, which is useful and safe, and you described that as nirvana in the video. And I guess maybe over time, I mean, C is somewhat really unsafe, really useful, yes.

Simon:

[27:01] So C, Rust has moved along the axis from useful but very unsafe to stay useful and become safer. Now, Haskell started life as being very safe but useless. So it’s worth just rehearsing that because in the first version of Haskell there was no I/O. The only Haskell program was a function of type string to string. It’s functional programming, after all; it could do I/O. All it could do was take a string and produce a string.

[27:31] So obviously that’s not very useful. It’s a little bit more useful. A language that is very, very safe and completely useless is no-op, that does nothing ever, very safe, but very useful. Haskell was a bit better, right? At least it lets you apply a function to a string and get back another string. So, of course, we then worked on how could we do I/O in Haskell in a safe way that will lead to monads and stuff, but just as the, you know, we’re moving from C horizontally to go safer and safer, but staying useful.

[28:09] So from Haskell, we’re moving sort of vertically to get more and more useful without stopping being safe. And nirvana is when we sort of meet up, right? But the point is not to say one is better than the other, but just to say that we both seek things that are useful and safe.

28:26 — Haskell vs OCaml

Ryan:

[28:26] When I think of functional languages, I guess in the mainstream, or what people are kind of thinking about, I hear Haskell. I also hear OCaml. How do these two programming languages compare if someone was trying to pick a functional language to work with?

Simon:

[28:41] So OCaml is a strict language that is called by value. So when you say F applied to 3, 4, it’ll add 3 plus 4 and then call F to get 7, right? In Haskell, if you say F of 3, 4, it’ll build a little suspension that says, well, if you ever need this three, four, you can evaluate it, but maybe you’ll never need it. And then it calls F. That’s a lazy language. So because OCaml is strict, it has a defined order of evaluation.

[29:11] So if you say F of 3, 4 and a second parameter, 8, 9, it’ll evaluate 3 plus 4 and then 8 plus 9 in that order. So it makes sense to say you can say F of print hello, print goodbye, right? Because you get hello and then goodbye printed. So once you have a defined order of evaluation, then it’s pretty easy to incorporate I/O, right? Even though it’s, quote, a functional language, OCaml allows you to do I/O without saying unsafePerformIO or anything.

[29:47] In fact, it has a defined order of evaluation, and it’s perfectly kosher to do that as a sort of unintended consequence of being called by value. It was also, by default, impure. Now, good OCaml programmers won’t use many side effects, but OCaml doesn’t prevent you. Haskell prevents you. Why does it prevent you? Because if we’d allowed you to say F of print hello, comma, print goodbye, those would be thunks, right?

[30:18] Not yet evaluated. Right. So if F happened to evaluate its first argument and then its second, we’d print hello and then goodbye. If it evaluated its second and then its first, we’d print goodbye and then hello. If it didn’t evaluate either, we wouldn’t print either of them. That doesn’t sound very good if you want to control what I/O is happening. Laziness forced Haskell to stay pure. Strictness allowed OCaml, which grew out of the ML tradition, by the way.

[30:45] ML is another functional language that predated Haskell. It was always strict, and it always had I/O by default. It wasn’t a goal. But that’s just the way it worked out at the time. Nobody thought about it. It was just obvious.

So that’s a big difference between OCaml and Haskell, right? Is that now, then, since then, they’ve both grown up. Haskell‘s gained sort of monads. OCaml’s gained lots of fancy type systems.

[31:13] Some of them look at how Haskell and OCaml have learned from each other. OCaml has got an interesting effect system recently and a whole lot of new extensions. It’s an absolute hotbed of innovation at the moment. OCaml. So I view OCaml and Haskell as kind of siblings. Brothers and sisters, right? We love each other, we learn from each other, we compete with each other, all of the things that siblings do, but they’re not the same, right?

[31:37] So siblings don’t say, I’m just better than you. You shouldn’t exist. We say, let’s enjoy our life together. And that’s what it is. That’s the way it is.

Ryan:

[31:44] You describe the laziness and the strict. And as a programmer, my initial thought is I would like to control the order of execution, and when I say print X and then print Y, I want it to happen that order. Otherwise it would be a little unintuitive. What is the main benefit of laziness?

Simon:

[32:05] Why is laziness good? Well, John Hughes did this rather well way back, in 1980-something. He wrote a paper called Why Functional Programming Matters. And one of its main theses was that lazy evaluation lets you compose programs in a particularly modular way. So imagine a program that is, I mean, his classic example is a program that plays chess, right? So one thing you could do is you could imagine building a tree of all possible moves, starting from the position we’re at.

[32:38] It would be a very, very big tree. It would be too big, because first of all I compute the tree and then I’d start deciding what move to do. It couldn’t possibly do that because the tree of all possible moves has more nodes in it than the number of protons in the universe. So let’s not do that. Now, if instead you could build the tree, having got this tree, supposing you had it, you could then walk over the tree saying, ah, let me do some minimaxing, and seeing, oh, this looks like a good position.

[33:06] Let me go this way. I could explore the tree right now, then. So in OCaml, then you would have to put the tree generation and the tree exploration in one function, right? In Haskell, with lazy evaluation, you can generate an infinite tree, and then the explorer that prunes the tree and explores just the bits of it that are necessary is completely modularly separated. I can build a different generator and a different pruner.

[33:36] They’re just completely separate programs. So lazy evaluation is very powerful glue that lets you glue together two programs that you’d like to be distinct. Strict evaluation forces you to merge them together.

Ryan:

[33:53] Yeah, this reminds me, so in Python, for instance, they have the idea of a generator where you could lazily retrieve things.

Simon:

[34:01] Every strict language, every serious strict language has lazy evaluation in it. They’re called iterators or generators, and so does OCaml. All right, but in Haskell, that’s the default. Now instead, in Haskell, so this is a bit like, again, the two are converging on the middle, right? Haskell, lazy by default. But you can make things strict. You can put in exclamation marks to say, please evaluate this before the call, right?

[34:29] Or the IO monad to say value. But that’s a more brutal strict annotation. So in Haskell, you can make things stricter. In OCaml, you can make things lazier. So it really boils down to what’s the default? We both want to have a mixture of the two. And then it becomes a bit cultural. As to which you prefer, I don’t know. Some people sometimes ask me. They say, well, well, if you were designing Haskell again, would you make it strict by default with really good support for laziness?

[35:02] And I often say, well, I might. Yeah, that does seem attractive because frequently I found myself cursing laziness as an implementer. But I strongly suspect that 10 years after that I’d be thinking, man, if only it was lazy by default. Oh, when I said strict by default, I would definitely mean strict, but pure, right, no side effects.

35:26 — Side effects in Haskell

Ryan:

[35:26] A few times in our conversation, you’ve mentioned the word monad, and it seems like it allows us to do the side effects. And what is a monad? And how does it help us do side effects and preserve ordering?

Simon:

[35:41] One way to think about it that’s very easy to understand and use is just to imagine that the do notation is somehow built into Haskell. So you can say do print x, semicolon, print y, and that has type IO unit. So a value of type IO unit means I do some input output and return a value of type unit. An expression of type IO int is an expression that when you run it will do some I/O and return an int. Okay, so do lets you combine I/O-performing computations together.

[36:23] So print 3 has type IO unit. It does some I/O, namely printing 3. GetChar has type IO Char. It does some I/O, namely reading a character from standard input and returns a character. All right? Notice that’s different from just Char. So quotes x quote has type Char. It’s just the character pure, right? GetChar, the I/O-performing operation, has type IO Char. It does some input output and returns a character.

[37:03] Okay, the do notation lets you combine together I/O performing computations. So if you could say do and then you say x left arrow getchar; putchar x, then that x left arrow getchar says run the getchar computation and get me the character. Call it x. The putchar x says run the putchar computation to put x, and we combine them together with the do notation that combines two computations to make one I/O performing computation.

[37:45] But these things are completely first class. That’s what’s new about monads compared to just make it into C. x left arrow, getchar semicolon, putchar x. That’s a computation whose type is I/O unit. Let me give it a name, so I can say let foo. We type I/O unit equals that do. Now I can pass foo as an argument to something. I could put foo in a data structure. I could return foo as a result; it’s a value just as much as three or plus in particular.

[38:21] For example, I could say do foo, foo that takes foo, uses it twice each time. Do foo; semicolon, foo says do foo and then do foo again, right? So I’ll read a character and print it, and then read a character and print it again. So these values of type I/O for some type T are first class values, right? In C, I can’t take X colon equals 3 semicolon, you know, Y plus 4 and pass that as an argument to something and expect it to happen wherever it’s used, right?

[39:01] It’s not a first-class value.

Ryan:

[39:03] When I first was learning this monad idea, it seems like you’ve said in a few places it’s a way to do the side effects, but keeps the language pure. But when I saw it, my first thought was it’s almost like this little place where we segregate the dirty things we want to do. But then how does that keep the language pure?

Simon:

[39:26] Oh, because. Because we are lying. But look, if your function has type int to int, can it do any I/O? No, no. If it has type int to I/O int, it could do arbitrary I/O. So yes, so it’s nice. Pure int to int function or dirty int to int function. So you may say, perhaps you’re about to say, that’s a bit of a blunt instrument. Either completely pure or completely dirty. Right. It gets you an awfully long way.

[40:02] But nevertheless, it would be cool if you could say, oh, I am effectful, doing reading of files only. Right. So, in the type, you’d like to say, what kind of effects can it have? Can it throw exceptions? Can it spawn new threads? So you’d like to enumerate the effects this computation could have. Right. That would be cool. That’s called an effect system. And there’s a bazillion programming language papers about effect systems.

[40:44] And it turns out that you can indeed, in the type system of Haskell, and indeed OCaml is rapidly becoming the same. You can express not just all or nothing, does it do I/O, game over, but rather which particular effects does it do, including no effects at all? That’s pure. Right. A good place to start is a library called Bluefin, which my colleague Tom Ellis has designed. It’s a very nice take on how to do effect systems in Haskell.

Ryan:

[41:17] I guess the purity comes from that. This dirty stuff is signaled through the type system.

Simon:

[41:25] Correct?

Ryan:

[41:26] Okay, so we can do it. But here it is. And be aware.

Simon:

[41:31] Yeah, and because you have to thread, you know, because the type system gets in your face, you were trying to map, say, no, map says I apply a function to every element of the list. Well, so it has type A to B to list of A to list of B. If you want to apply an I/O performing function, so the function you’re applying has type like int to I/O of character, you could map that over a list, but you just get a list of I/O of chars that hasn’t done any I/O yet.

[42:03] Right. You want something that says, take a list of IO chars and perform those actions one at a time. So I want a function that goes from type list of IO char to IO of list, and you might want to perform all those actions top to bottom, or maybe bottom to top, who knows? That’s what this function of type. Yes. So you could do it in various ways. So sometimes it gets in the way. Right. You say, oh, shit, can’t I just use Map?

[42:34] Well, in Haskell, no, sorry, you’re going to have to do a little bit more work to tell me in what sequence you want your effects to happen. Because by default, Haskell does not specify sequence. So, adding monads to control effects does get in your face a bit. And that’s the tax we pay. In effect, that’s part of the big experiment that Haskell is doing: to say, suppose we upfront say we’re willing to pay that tax.

[43:04] How many followers can we get? And the more. But I will note that monads have infected quite a lot of other languages. Like F# is definitely a call-by-value impure language. And yet F# had these workflows that were definitely monad. And monads have appeared in Scala and in many other languages. So something that’s very monad-like has appeared lots elsewhere. It’s been a very unifying concept.

44:26 — Type systems

Ryan:

[44:26] And so I kind of want to ask you, maybe just even on the highest level, what is a type system, in your words?

Simon:

[44:35] Okay, so what does a type system do? It lets you reject silly programs upfront. So if I have a function that adds one to things and I apply it to a character or to an I/O computation, that’s going to fail at runtime, right? Because I can’t add one to a character. Or maybe you overload plus. Maybe I reverse a list or something. If I’ve got a list reverse function, and I give it something that just isn’t a list, then it’s a bit silly to allow that and only fail at runtime.

[45:13] So fundamentally, type systems are about rejecting at compile time programs that you do not want to run because they will fail at runtime. Okay, now, we’ve had static type systems for a long time, going back to Pascal and earlier. But simple type systems are annoying because they get in the way. Imagine a function that reverses a list. In Pascal, you could write a function that reverses a list of integers.

[45:46] But if you wanted to reverse a list of characters, sorry, you can’t. That function that reverses a list of integers, it has type list event to list event, so you can’t apply it to a list of characters. Game over. You have to write another copy of the code with a different type. That’s a bit annoying. So what are we going to do? We need polymorphism. We need a more powerful type system. We want to give reverse the type for all a list of A to list of A.

[46:18] So that upfront says, I work for any type A. You give me a list of integers. Fine, I’ll produce a list of integers. You give me a list of characters. Fine, I’ll produce a list of characters. Notice that’s much better than just list to list. I want to keep the fact there’s a list of integers so I know if I apply, when I look inside the list, I know I’ve got an integer. But also if I just list the list, I might get back a list of some completely different type.

[46:43] But I know that reverse returns a list with the same type of things, right? So parametric polymorphism, super valuable, right? If you want a size of type system, it must have parametric polymorphism. The message is, if your type system is too simple, it gets in the way. Very important lesson, because it means that we cannot really say a type system is useful or not unless we say which one, because we know that weak type systems are very inconvenient.

[47:16] It must at least have parametric polymorphism. And that is another idea that was born in the world of functional programming. It was born in ML, incidentally, Robin Milner’s famous dictum: well-typed programs don’t go wrong. ML was the first, I think, the first parametrically polymorphic functional programming language. But generics in Java and object-oriented programming more generally is exactly the same idea, right?

Ryan:

[47:42] into the mainstream and you mentioned polymorphism, and so there’s this parametric polymorphism in the type system. But in the example you mentioned where maybe you want to do an operation on a list and then the type of the input changes, I was just thinking, what about when people do polymorphism in the classes?

Simon:

[48:03] Yeah, okay, but so now you’re into a whole more complicated world, right? So as soon as you say class, you’re talking static type system again, right? So object-oriented programming is another approach to polymorphism. So in an object-oriented language, we say if we have a Ford that is a car and a car is a vehicle, then anything that works on vehicles should also work on cars and should also work on Fords.

[48:32] So the code that we write for cars works unchanged for vehicles and for Fords. Okay? Now that’s a form of polymorphism, not parametric polymorphism. That’s what you might call object-oriented polymorphism or subtype polymorphism. Okay, so now we have polymorphism in the sense that the same code, the same executable code, actually the same machine instructions work on values of different types. Okay. In both cases, both reversal lists, same machine instructions.

[49:08] Code that works on vehicles works on things on Ford. Same machine instructions, right. Okay. That’s what polymorphism in general means. Parametric polymorphism means this for all. A list of a to list of a stuff. Subtype polymorphism means if it works on vehicles, it works on any subtype of vehicles. Okay? Now the interaction of the two, which you get by adding generics to an object-oriented language, is pretty complicated.

[49:36] Oh, and by the way, adding side effects as well. And so your question involving classes and superclasses and so forth is smack in that complicated world. I’m not sure it’ll be very fruitful for us to get a lot more details of the type system sorted out to know what to say. But at this very high-level overview, the big point I’m trying to make is weak type systems get in the way.

[50:04] Type systems are meant to reject programs that will go wrong. But my function that reverses a list of integers, it will also reverse the list of characters. So it’s tiresome to be told, no, that is a bad program. I want to be able to write that it has type, for all a, list of a to list of a. So the idea of making the type system more complicated is to say programs that you want to run, you can still write in your static type system.

[50:29] Now you might say, well, blimey, if that’s all, why don’t we just get rid of the static type system altogether? Now we could run all of those programs, but the trouble is, you can run too many programs now. You can run programs that will crash at runtime, and that’s very, very, very bad. And we all know the costs of programs that crash in deployment, that you could have crashed before you even started to run them, before you even ran your first test, right before you even linked it into an executable.

[51:00] That’s really good. But the biggest benefit of a static type system, in my humble opinion, is maintainability. If you have a program written in, I don’t know, Perl or Ruby or Lisp in its basic form, and it was written 15 years ago and the original author has left, and all of the people who were involved at the time it was written have left, then that program is very difficult to maintain, and it becomes almost immutable.

[51:31] Nobody dares change it anymore. What they do is, it’s an immutable piece of software, and you do stuff around the edges to impedance match what you really want to do to this now immutable blob. Now of course you have lots of tests, so test-driven development, I’m totally with it, I’ll develop all of that. But still, GHC, for example, is itself written in Haskell. It’s 35 years old, and yet I do large-scale systematic refactorings of it fearlessly because the type system keeps me safe.

[52:02] In fact, often what I’ll do is I’ll change a few types and then start compiling, and then a sort of wave of changes propagate through, forced by—I just get type errors. So I know what to do. The thought was that I’ve changed the representation of this data structure a little bit. I’ve added a field to this data structure. Where in the entire compiler might that field be read, written, or freshly allocated?

[52:27] I can’t imagine how anybody maintains 30-year-old software and makes large-scale changes like that. Without a type system, it’s unimaginable. So for me, the benefit of type systems is maintainability. Oh, and designability. Right. So I often write the types of my programs up front. The data types are super perspicuous. Novices say it’s a bit like writing the classes of an object-oriented language, right?

[52:55] But if you have no types, no classes, nothing, just I don’t know, S expressions, types are my design language. They’re the way that I start writing my designs.

Ryan:

[53:07] I’m trying to understand the other mindset. Let’s say C, for instance. I remember a lot of stuff when I was learning it in college, for instance, many times, right. I’d add a character to a pointer or something, and it just works because it interprets the character as a number. So is there any value to having that kind of type system, or is that just strictly unredeemable?

Simon:

[53:34] Just use stronger, I think there’s no benefit to weaker.

Ryan:

[53:39] Just off the top of my head, one thing I think of is there is a set of programs that will work but don’t satisfy the type system.

Simon:

[53:49] Oh, yeah, that’s why.

Ryan:

[53:51] But those are. I mean, I don’t know if they’re good. That’s subjective.

Simon:

[53:54] Oh, no, they may be good. So we started, if you have Pascal and you write a function to reverse a list of integers, then if you apply it to a list of characters, the same machine instructions would work, but it is rejected, right? So we have rejected a perfectly decent program. Bad. Right. Our goal is to expand the collection of programs that satisfy the type system to include as many as possible of the programs we want to run without including any of the bad programs that we don’t want to run.

[54:32] Okay? Parametric polymorphism is a big step in that direction. Other type system innovations are a big step in that direction. But there will always be some programs that would run perfectly well that the type system rejects. Imagine a tree that contains integers at every node. But if you are 17 deep in the tree, or 34 or 51, if you’re in a multiple of 17 deep, the integers can be characters instead, or the integers all turn out to be characters, right?

[55:17] Now, you could write a program that generated such trees, and you could write a program that consumed such trees, knowing that every 17th layer we switched characters. But most static type systems would make it pretty hard for you to accept that program. And yet it will run. Now you might say, “Oh, but I really want to write that program, guys. Don’t get in my way.” Well, then we should provide a way for you to bail out into dynamic typing, right?

[55:49] What I’d like to do is say, okay, so if all else fails, then at least you can, as it were, pair up a value with its type representation. There’s one reason we don’t want to interpret an integer as a double precision float, for example, is that they don’t even have the same representation, which is just nonsense. One possible way untyped languages let you do this is to tag every integer and every double precision float.

[56:14] With the fact I’m an integer, I’m a double precision float. But that has a lot of overhead. So one merit, I think it’s not the biggest single merit, but it’s a major merit of static type systems is you have no tags. You know that if it says it’s an integer, it’s going to be an integer. You know that if it’s a double precision float, it’s going to be a double precision float, right? But if you’re not sure, maybe we could make a way to make a pair of a type representation.

[56:42] And this value, the type representation, is now like a runtime tag. It’s like a little runtime data structure that describes the type. And then in your program you could say, now I want to say, oh, I’ve got this type dynamic. We’ll call this pair a value of type dynamic. Now when I want to take a value of type dynamic and treat it as a character, I want to say, ah, look, look at the type dynamic.

[57:04] See if it says it’s a character. If it is, return the character. If not, crash.

Ryan:

[57:08] Right?

Simon:

[57:08] Runtime failure. That’s fine, you can do that. So Haskell has good support for dynamic typing where necessary. So my story would be static typing should expand to carry as much as possible, and where you absolutely cannot do it, sorry, then use dynamic typing. And we provide facilities to support that.

57:30 — How the Haskell compiler works

Ryan:

[57:30] One thing I wanted to talk with you about is the compiler. How does GHC work on a high level?

Simon:

[57:37] So GHC takes a string like any other compiler. The source code of the program parses it, and then it type-checks, it figures out, is this a type-correct program? Then it converts it to lambda calculus. Now Haskell, the Haskell AST, the original source tree, has 50 different data types, 50 different kinds of nodes, some of which have 30 or 40 different variants. So it’s a really big, diverse, complicated data structure.

[58:15] The AST lambda calculus has this variant of the lambda calculus. It has eight WANDA data, maybe two or three data with eight constructors. So it’s like taking a gigantic language and squeezing it down into a tiny one. And that tiny one we can then optimize, right? That’s what the optimizer works on. So the front end does parse, rename, type check, desugar into lambda calculus. That’s the front end. The lambda calculus particular language is called GHC Core language.

[58:52] I’m quite proud of it because it’s been very, very stable. It’s 35 years old, and it has barely changed since birth. That’s amazing, right? Because Haskell has changed a lot. A lot. So almost all of the innovation in Haskell has been in the front end. Very little in core. Now the back end, the core optimizer, has changed a lot too. But all of the changes that were useful there would have been useful 30 years ago.

[59:25] Right. So they’re two completely separable things. So I’m quite proud about that. Core then. We do a lot of core-to-core passes that simply take core programs. We’ll use core programs, lots and lots, long pipeline. Then we convert it to C, which is a prototypical imperative language. Think of it as a portable assembly code. So that bit is meant to be platform independent. So it’s simply. That’s the compiler that takes.

[59:54] Remember I said if you take lambda calculus and compile it, that’s that step. Right. I want to compile the lambda calculus into machine instructions, really. But I don’t really mean machine, I mean portable machine instructions. That’s called C. Then I want to convert C into actual machine instructions for various platforms. And there we could either do that directly with the native code backend or go via LLVM.

Ryan:

[01:00:18] Interesting. I’ve never heard of C. Why not just go directly to it? I would have thought maybe assembly or something like that.

Simon:

[01:00:25] Well, because. What happens in assembly? In which processor? Please?

Ryan:

[01:00:30] Oh, I see. I guess it’s the. I would have thought the lambda calculus part was already portable.

Simon:

[01:00:36] So you just. It is, yeah. You could go straight, but, but, but. So there’s work to go from lambda calculus. You could go all the way to x86, then throw all that away and now go from lambda calculus to PowerPC. Oh dear. I’ve just duplicated a lot of work by going from lambda calculus to C-- and then from C-- to x86, x C--. We’ve avoided duplicating the work that went from lambda calculus to C--.

[01:01:05] When you see that, it’s pretty obvious, isn’t it? You got to. You want to make the platform-specific bit as small as possible. You would like to have, as it were, a generic architecture, one that can do addition and has a program counter and a stack and so forth. That’s all C--. It’s just a portable assembly language. And then you say, oh, the nitty-gritty of whether you have double-precision add and set, the floating-point bit here and there, that.

[01:01:31] Well, that’s platform specific. The core is itself statically typed, but in some ways it’s surprising because no other compiler has this property. No other production compiler by core is statically typed. I don’t just mean that the initial program was type correct. I mean that a core program, you could run a type checker on that. I might say, why do you need to? Because after all, if GHC is correct, it started with a type correct core program because it came from a type correct Haskell program.

[01:01:57] Assuming the desugaring was right, and all the optimizations, if they’re right, will generate a type-correct Core program. So why do you need to type check Core to discover bugs in GHC? Now, these are serious bugs, right? If you ever take a type-correct Core program and an optimization pass produces a type-incorrect Core program, what will happen if we don’t have the type checker for Core? We’ll generate machine code, and we’ll run it, and we’ll get a segfault.

[01:02:29] Now we have to backtrack from a particular test program that now crashes all the way back up the pipeline. Up the pipeline, up the pipeline, up the pipeline. Oh, it was this pass of GHC that was faulty. That’s super hard to do because you’re getting out GDB on some runtime failure. That is an indirect and perhaps distant consequence of the fact you just generated a type. You just made a.

[01:02:56] You had a bug in the optimizer. So it is amazing to have a type checker for Core. Now, why does nobody else do this? Well, it’s because their intermediate language, typically, and this I really am talking typically, because I know of no other compiler that has this property, none. Production compiler, typically. Then they’re complex syntax trees decorated with all sorts of pragma information and things hanging onto it here and there.

[01:03:27] There’s no type checker for it at all, and there’s no hope of one. So I’m very proud of the fact that Core is statically typed. And I also think it’s a. The most delightful thing is that the way that it is statically typed, it is an implementation of something called System F. So System F, when I said lambda calculus, lambda calculus, as Alonzo Church had, it was untyped, had no type system at all.

[01:03:55] But Girard defines something called System F, which is a statically typed lambda calculus, a rather powerful one. And GHC Core is essentially System F. So he literally adopted something from the nerdy theoretical computer science, logic community, logic of mathematics community, and adopted it directly in a production implementation. So I’m very proud of that. I think GHC Core is. And the fact that we can do we have 35 years worth of development that has not just not been impeded, but actively aided by statically typed intermediate language is really a big marker in the ground.

01:04:35 — Why Haskell is talked about more than used

Ryan:

[01:04:35] Watching all your talks, reading everything, one of the interesting data points that you brought up was that Haskell is talked about more than used. When you compared Stack Overflow volume and actually GitHub volume, like who’s actually using the programming language, why do you think that is?

Simon:

[01:04:55] So Haskell embodies one key idea: immutability changes everything. There’s a quote from Pat Helland’s talk that I think you also looked at or read. So it says programming with values changes everything about the way you think about programming. It’s just mind changing. It’s not necessarily better, but it is different. And so Haskell takes that idea and runs with it. Everything is driven by that one idea.

[01:05:23] Everything else is incidental. In the early days, that meant we just said, well, guys, suck it up. We’ll keep changing the language, and if it breaks your program, too bad. So it is a bit peculiar and therefore also it felt a bit academic because initially it was really not very powerful. It was taking the key idea, but you couldn’t do very much with it. We talked about that. Right.

[01:05:44] So over time, GHC and Haskell in general has become more and more powerful. The type system has become less and less in your way and more and more useful. All the obstacles that make functional programming harder become better. The compiler generates faster code, it can browse faster and so forth. So it has become less peculiar. So we’ve become more and more taking into account the needs of our users.

[01:06:12] But in a way, the sort of cultural heritage is we never give up on the one core principle. We’re just not going to give you unrestricted side effects. Sorry, you want to say unsafePerformIO and put up with the consequences. So we’re going to stick to one core principle and then we’ll do lots of work around the edge to make that better. Right. So that does limit our community somewhat. Right?

[01:06:34] It does mean you really have to think in a different way. Immutability changes everything. That means you’d have to think a different way about programming. Maybe you don’t want to think in a different way. That’s fine. Then don’t use Haskell.

[01:06:44] So in a way, we started from a very small user community, a very sort of pure and nerdy one, and growing larger and larger. But all slowly, slowly, while all the time maintaining faithfulness to this core principle.

Ryan:

[01:06:58] I thought that was very unique about Haskell because I feel a lot of the other programming languages are user-centric. I mean, if people want something, they work on it, they add it, whereas Haskell feels more principled. Or it’s all starting from these ideas. And if you don’t satisfy these ideas as a user, yeah, well, that’s fine. For instance, in one of your talks, you mentioned somewhere that there was a release of the compiler where if a file wasn’t type-correct, then the compiler would delete the file.

Simon:

[01:07:37] Oh, wait, it reported the error message first.

Ryan:

[01:07:39] Yeah, it reports there. But I thought that was absurd. I mean, very hostile, I guess, to the. I mean, well, if you’re type safe, no problems. But.

Simon:

[01:07:49] Oh, it was a mistake, right? It was a bug; it wasn’t deliberate, and it only happened, you know, how did the bug get out? Because, of course, if it always did that, we’d have noticed. How did it get into a release? Well, it was because it was only on Windows and only when you compile a module that was not in the current directory. But at that stage, our users were very forgiving. And, you know, somebody wrote to us and said, well, by the way, Simon, you might like to know that, you know, GHC does this, but, hey, don’t worry about it, you know, I just copy all my files somewhere else before I compile and then I copy them back.

[01:08:21] So, of course, those days are long gone. We pay a lot more attention to our users and have much more rigorous CI testing than ever we did. Right. So that’s a story from a long time ago, but it’s a good cultural story because it suggests that we care about our users very much, but we care about users who want, in their hearts, want to be principled. We’re trying to appeal to. One of the things I like best about Haskell is people often say, I just enjoy writing Haskell.

[01:08:48] Right. It’s fun, right? My boss doesn’t allow me to because it somehow doesn’t fit with my production shop. But for me, I would go for it. I love writing this stuff. Every time. Every time. Yeah, that’s very rewarding to me.

01:09:07 — Avoiding success at all costs

Ryan:

[01:09:07] There’s also this interesting—I don’t know if it’s a cultural value—but it’s a statement that you say often in the context of these older talks, that you say that you avoid success at all costs. Could you explain what you mean by that phrase?

Simon:

[01:09:21] Oh, yeah, this was just a little play on words. It was in a retrospective on Haskell. I gave an invited talk at POPL a long time ago, probably 20 years ago. So it’s a little play on words because you can read it as either avoid success at all costs. And that’s what we’ve been discussing. Success at all costs means compromise your principles in order to satisfy your users or think that you’re satisfying users, give them what they say they want, if we build it, they will come kind of deal.

[01:09:55] So avoid success at all costs. Or if you parenthesize it the other way, it says, avoid success at all costs, at all costs. Avoid success. And that’s saying, that’s a little joke. But it says if you’re too successful and have too many users, it becomes more difficult to make changes. And we experience that right now. So I devote many, many more of my personal cycles to backward compatibility issues than ever I did.

[01:10:28] I’ve devoted hundreds of hours and hours, well, days and days, weeks and weeks in the last year or two to the following: what seems to be a very simple property. If you can compile a program, a package, a whole program with GHC 10.0 and we release GHC 10.2, you should be able to compile that same package unchanged with GHC 10.2. Seems reasonable, right? After all, 10.2 should just be better. But no, GHC has never had that property, and making it have that property has turned out to be very, very time consuming.

[01:11:05] Previously we just never cared. Then we started to care, but thought it was a lot of work, and now we’re investing the work.

01:11:12 — LLMs and programming languages

Ryan:

[01:11:12] I guess that was from a long time ago. I think nowadays software engineering, there’s been a major shift in the last year where a lot of code is being generated by these models or these LLMs. How do you see programming language design shifting to accommodate a world where a lot of the code is no longer written by humans?

Simon:

[01:11:34] I think it may be the best thing that’s happened to statically typed languages for a long time, because as we’ve been discussing with a static type system, you cut down the space of programs that the LLM can generate because it is perfectly capable of running the compiler and saying, “Oh darn, that was a bad program, better fix it,” right? So a zillion iterations get done behind the scenes, whereas in an untyped language, the first one it coughed up, you’d have had to run it against a test suite or who knows what, but it dramatically tightens up that cycle.

[01:12:11] Right. So I think that statically typed languages are a huge boon for LLMs because it’s too easy to. And programs are just strings, right? It could just generate the next plausible word.

Ryan:

[01:12:31] Away if there’s a slider on, I guess, the strength of a type system. And the other side of the week, what do you see as the absolute strongest type systems among programming languages?

Simon:

[01:12:43] Oh, Haskell. I think Haskell is exploring the bleeding edge. There is an exception, which is that module systems are a... And in particular, sort of functor-style module systems are explored much more deeply in the OCaml world and in the Haskell world. We’ve essentially never gone there. But otherwise, I think Haskell‘s right up there. Now, of course, a language like Scala is also as... So Scala has almost everything Haskell has, I think, not quite, but it also has subtyping and object orientation.

[01:13:24] So that’s a lot more complicated. A lot more complicated, I think. And they pay a price for it. I think Martin Odersky would agree that they pay a price for it. So in complexity, it’s probably more complicated than Haskell‘s, and maybe in terms of power. So maybe I should have said Haskell and Scala are the two leading Haskell, Scala, OCaml, perhaps I’ll just put them in an equivalence class for now.

[01:13:52] They’re not strictly comparable. They all have things which they’re more powerful than the others, probably.

01:13:57 — New programming language design

Ryan:

[01:13:57] What do you think are the important problems to solve in the future of programming languages today? What are the unsolved problems in the domain that are top of mind for you?

Simon:

[01:14:08] I think it’s actually hard to identify, to say, here’s a problem we want to solve, let’s try to solve it. Well, I think another way to tackle it is to say, what are interesting new languages out there that are exploring very different parts of the design space? And there I think I do have a candidate. So the language that is my day job. I work for Epic, and we’re designing a programming language called Verse.

[01:14:31] Now, Verse is a very exotic language. It’s really a functional logic language, so it’s yet more expressive than Haskell. It has a path-? type system, but a very different one to Haskell‘s. So if you like, the way I think of it is like this. If you look at C and Fortran, they look pretty different if you’re an imperative programmer. But if you look at them, if you zoom out still, you can see functional languages.

[01:14:56] Then C and Fortran are pretty close together, along with object-oriented languages. They’re all in a clump, right? And then there’s some functional languages, Haskell and ML. I know OCaml and Scala out here. If you zoom out still further, then the imperative languages and functional languages altogether, and Verse is way out here, right. So Verse is exploring a very new point in the design space.

[01:15:19] But just like functional programming back in 1980, it seems sufficiently interesting and cool and unusual and weird that it’s worth exploring. Right? So back in 1980, nobody would have said, we’re definitely going to do functional programming and it’s going to be useful for practical applications. They said, that’s pretty weird. By all means, give it a try, guys. And that’s kind of where I am with Verse.

[01:15:40] One difference is that back in 1980 we were purely academics, and now, Verse is being developed by Epic Games, so we got much more muscle behind it than was behind functional programming to begin with. So we’ll see. It’s a very interesting intellectual endeavor, adventure, I should say.

01:15:59 — Should students continue to learn programming

Ryan:

[01:15:59] Yeah. I think a lot of people, like students, are maybe worried about AI or studying computer science. Would you recommend people learn how to program today, given that AI is starting to write reasonable code now?

Simon:

[01:16:13] Oh, yeah, yeah. So I think people are right to be worried in the sense that I think there’s going to be considerable dislocation, right? Because if you were in the Industrial Revolution, then lots of people lost their jobs as spinners and weavers, and it wasn’t easy for them to get a new job in the new economy. Now, the new economy had, in the end, had more jobs, but there was.

[01:16:36] If you were one of the people who just lost their job, that was not a happy place to be. From our perspective, you know, from an Olympian perspective of a few hundred years later, we think, well, it’s just a blip, right? If you’re part of the blip, not a problem. So I think people are right to be worried. We don’t know how things will shake out. I’m actually optimistic that in the medium term, if we don’t destroy ourselves with some truly existential thing.

[01:17:00] But from an employment market point of view, I’m optimistic that in the end we’ll just be in a higher place, that AIs will just be a bigger power tool. I mean, everyone likes using compilers, right? We don’t like machine code anymore. Compilers make us more productive. Maybe LLMs can make us more productive. That’s what I hope. I sort of believe, modulo dislocation effects. Now, should we teach children or even undergraduates how to program?

[01:17:31] So I think still yes. And the reason is because one way to say it is copilots who need pilots. I think Copilot was quite a good title that Microsoft gave their tools because it encourages you to believe it’s your partner, not your boss. If LLMs spit out a pile of goop and we literally do not understand what it does, we just try it and it kind of works. That might be okay if we’re just throwing up a quick visualization.

[01:18:00] It might be less okay if the quick visualization is going to drive our policy choices about us a nation, whether to go into lockdown because of COVID or if this program is going to run my aeroplane or train signaling system. So now those are extreme ends of the spectrum from quick and dirty things. It really doesn’t matter if it doesn’t work, but it kind of does a lot of the time, absolutely fine too.

[01:18:27] This is a 30-year code base. It’s going to last a long time. I really want to make sure that it’s like putting new stuff into GHC. If somebody sends me a pile of AI-generated code to put into GHC, I’m not going to put it in unless I’ve reviewed it or somebody’s reviewed it. Because in 10 years’ time, I’m going to want to change that code. How do I even know what it does? If it’s simply a magic incantation that somebody’s done that kind of worked on the test they did, but maybe won’t work in deployment, that’s no good.

[01:18:51] So I really want long-lived, maintainable code to be well reviewed. Sorry. So. And to do that, I need reviewers who can write code, who know. Let me mention one other perspective. If you think about what every child should know. When I think about what every child should know about computing, I would include binary and bits, not, just as for physics, I would include atoms and molecules. Now it’s not that in real life anybody manipulates atoms or molecules or takes decisions which are based directly on their knowledge about knowledge, but somehow knowledge that all matter is made up of atoms, you know, constituted of a finite number of elements that atoms.

[01:19:38] That knowledge underpins everything we understand about the natural world. If you literally have never been told that, you are sort of emasculated as a citizen, let alone as a scientist. So if you literally do not know that everything is composed of bits, that words and music and text and LLMs and everything is all just bits, I think you’re crippled. So I want every child to learn, you know, it’s like I want you to learn the bottom.

[01:20:07] It’s all bits, nothing else. It’s all just bits. I even have a talk. The talk is called Bits with Soul. It’s so easy to grab for. It’s a talk I gave to an audience that was not a computer science audience at all. It was a completely lay audience ranging from 14-year-olds to professors of quantum mechanics. Pretty difficult audience to address. And it was meant to be about codes and coding and bits.

[01:20:33] So it tries to get at the essence of why I think it’s important that every child, every person, every human being should understand something about the computational universe that surrounds them and that is founded in bits. Now, just to develop the analogy a bit further, I would then say, and it also, I think, I want them to also know about programming programs. I want them to know that computers fundamentally execute by following machine instructions blindly.

[01:21:00] Right. That is not magic, it’s not hocus pocus, it’s just remorseless and very dumb. Right. It’s incredibly empowering then. And also to learn the basics about how neural networks work. In the same talk I explain how a one neuron neural network works, and that’s enough. Then it’s actually true to say, not distorting, but that’s to say ChatGPT is just a trillion of those wired together, and astonishingly, that very simple.

[01:21:32] So all ChatGPT is, is a trillion floating-point numbers and a lot of floating-point arithmetic. That helps you make sense of a question like, can ChatGPT have feelings? Well, it gives you, I mean, of course that’s a philosophical question, but it informs your discussion about if you know that all it is is trillion floats and a lot of floating-point arithmetic. That’s all, nothing more.

[01:22:02] Anyway, sorry, long answer to your question, but so I... So yes, basic programming, absolutely, yeah. Becoming fairly skilled in how to use, you know, Django framework, maybe not so much maybe. I think one thing that LLMs are very good at is knowing the arcane and complicated APIs that many of these frameworks present when there’s just 10,000 functions you’ve got to know.

01:22:33 — Why Excel is is 2nd favorite programming language

Ryan:

[01:22:33] You mentioned somewhere someone asked you, you know, what’s your favorite programming language? And obviously Haskell‘s got to be number one. Yeah, but for your second favorite programming language, you said it was Excel at that time. Why was Excel your favorite programming language?

Simon:

[01:22:49] Oh, because it’s the world’s most widely used functional programming language. The formula language is a functional language, isn’t it? Doesn’t have any side effects. You program entirely with values. So the formula language of Excel is a functional programming language, a very weak one. It doesn’t even let you define new functions. And it has a very limited collection of data types, namely just flat arrays and numbers and strings.

[01:23:17] So when I was working for Microsoft, I took it as my war cry to say, let’s take that idea. Excel is the world’s most widely used functional language by three orders of magnitude. And oh, it’s the most widely used programming language by three orders of magnitude, not just functional language. Excel is used by many, many more programmers, users, domain experts than any imperative language.

[01:23:51] Imperative language is a few million users. Excel, hundreds of millions of users. Even if you just restricted users who are using formulae. So how can we delight those users? Answer: take ideas from functional programming and use them to make Excel’s formula language more powerful. It took me 20 years, but Excel did finally add Lambda to Excel. Look it up, there are blog posts about it and many YouTube videos about it.

[01:24:25] So you can now program in Excel using full lambda as Alonzo Church originally defined it. You can write anything, because lambda is computationally complete. You can write any computation in Excel now. It would be a bit slow, but you can. And much more practically, you can take formulae that previously you just copy-pasted here and there and wanted to make reusable, wrap them up in a lambda, and now you can just call the lambda.

[01:24:53] And now it is a proper grown-up functional language that is Turing complete. Just search for Excel LAMBDA, those two keywords will get you lots of raw material.

01:25:04 — Advice for his younger self

Ryan:

[01:25:04] Last question for you is, knowing everything that you know now from your career, if you could go back to when you just graduated from college and give yourself some advice, what would you say?

Simon:

[01:25:15] All of these people who you see very successful wandering around, looking as if they’ve made it, I guess you might class me among them. Now, they are all just making it up as they go along. They feel insecure, uncertain, not sure what to do next, not sure what their next steps are, not sure what next big problem they’re going to tackle, unsure about whether what they’re doing is going to be successful or not.

[01:25:40] And so all of their confidence is, I mean, they project confidence. Maybe that’s partly a life skill, but often they’re not. And so the fact that, in those days, of course, I felt very not confident. I would say, since all of these successful people are making it up as they go along, it’s fine for you to be as well. And they’ve been lucky. Moreover, they’ve been lucky. They’ve had.

[01:26:08] But if you want to be lucky, you do need to put yourself in a position where accidents can happen to you, and that means taking risks. So if you want to be lucky, you need to put yourself in positions where lucky things could happen.

[01:26:28] And that means taking some kind of risk. So if you’re very conservative and never take any risk, then it’s very unlikely that the accident that is life transforming will happen. That’s a balance, of course. But it means that accidents are not so bad. And of course, dying is bad. But lots of accidents may change your life in a way that might be surprising to you, but turns out to be not so bad in retrospect.

Ryan:

[01:26:50] Awesome. Well, yeah. Thank you so much for your time. I really appreciate it, Professor Jones. We will.

Simon:

[01:26:54] Nice talking to you. Thanks, Ryan.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

01:08 - P vs NP

14:51 - What if you relaxed correctness

25:38 - Why NP complete problems are equivalent

30:33 - Space vs time complexity

43:06 - Why people use SAT solvers

45:53 - Randomness is a resource

55:48 - Randomness depends on computational power

01:21:20 - Zero knowledge proofs and their significance

01:38:30 - Quantum computation and why it matters

01:56:24 - Math vs computer science

02:08:16 - Major breakthroughs and his experience

02:12:31 - Advice for his younger self

Transcript

01:08 — P vs NP

Avi:

[01:08] Well, in the simplest way, and I will talk at a high level, we are interested in various problems in life, in all sorts of problems: mathematical problems, scientific problems, medical problems, personal problems, intellectual problems. We are in the business, in computer science, of solving problems by computers. Now we would like to somehow classify those problems that we can solve, to know what we can know.

[01:43] The problems we can solve are the things we will know, we will understand. There are all these problems that we want to understand. There are many of them, and there are problems that we can understand but can’t solve. In a very basic way, the first class is NP. All the problems we want to solve are really NP problems. It’s a class of problems, and a subset of it is the problems we can solve.

[02:13] They are the problems we can currently solve, but we are interested in all the problems we can solve in principle. The question of P versus NP is whether we can solve all the problems we want to solve, whether we can know everything we want to know. This is basically about the limits of our knowledge. Of course, I have to justify why these classes, which are mathematically defined, are actually captured by the intuitive meanings I’m giving them.

[02:52] P is very easy to justify. What we can solve is what we can solve in our lifetime using whatever we have, let’s say all computers that we have using efficient algorithms. Problems we can solve are problems you see in our apps, in our phones. If there’s a navigation app, it means somebody invented an algorithm that’s efficient enough to solve any shortest path problem between any two places in any kind of map. All the problems we want to solve are NP problems.

[03:32] What is NP? NP is a class of problems in mathematical definition, problems such that we don’t know whether they are easy or hard to solve. But if somebody hands us a solution, then we can easily check that indeed it is a good solution. Now, why are all problems humanity is interested in of this nature? You can think there’s no limit to what we may want to know. But I claim that essentially, in any human endeavor, if you are embarking on any problem, if you really seriously want to solve a problem, the least you want to know is that when you hit upon a solution, you will recognize it as a solution.

[04:19] Why would you ever start looking for something if you will never recognize it as what you looked for? So you think mathematicians are looking for proofs of theorems? Certainly, if somebody gives a proof and writes a paper about it, we can verify it. If a scientist wants to explain data about the planets or about bacteria or whatever, they want to develop a theory. So that’s what you are searching for in this case.

[04:54] And yeah, again, scientists write papers about their theories. They have to be consistent with the data. We have a way of recognizing that this is a good solution. Engineers are usually tasked with creating something, like a bridge or a phone, under some constraints. It can be monetary constraints, physical constraints, all sorts of limitations, such as what you can use but not this, and so on.

[05:24] But anyway, if an engineer designs something, whoever gave them the task can look at it and say, “Well, you violated some constraints,” or “No, it’s good, you really delivered what we were looking for.” So I’m giving you examples. I mean, I can give many more. Detectives are supposed to solve crimes, and we know from detective books they eventually provide solutions. So these are examples of very fundamental human endeavors that encompass almost everything I can think of in all of them.

[05:58] What people are searching for has the property that when a solution is found, it is easily recognized. Repeating in the beginning, NP are all problems that we can honestly say we really want to solve. NP are those that we can solve. If they are equal, then everything we would want to know, a cure for cancer, anything you can imagine, can be just the very fact that a solution can be easily recognized, which would imply that it can be efficiently found.

[06:38] That means that we can know everything we ever want to know. It’s a fundamental question about human knowledge.

Ryan:

[06:44] So I know this is one of those Millennium Prize problems. There’s a million-dollar prize if you can provide a proof for this. I can imagine you could prove that they’re equivalent. You could also prove that they’re not. If you had to guess when and if a proof comes, which direction would your intuition say it goes? And why?

Avi:

[07:07] I think that my intuition, and that probably holds for almost all members of the theoretical computer science community, is that they are different. I think there are several reasons for that, and I will already tell you now that they are not that convincing. One is the intuition that most of us have that finding something is really hard and just checking that it’s there.

[07:37] If you lost your keys or your phone and have no idea where to look, but if somebody points out, “Oh, you left it on the windowsill,” you might say, “Ah, yeah, I did.” We have this experience from life that finding something is typically a harder task than just checking that it is what we look for. Another reason is that real NP problems occur all over the place. They occur in optimization, in logic, in other aspects, in mathematics, and in verifying that algorithms, protocols, and security systems actually work.

[08:22] All sorts of. In this generality, these optimization problems that are NP problems, maybe NP-hard problems, people try to solve for completely egocentric reasons to make their company richer. There are thousands of these, and they manifest themselves in many different forms. Over the last 50, 60, or 70 years, people have seriously looked for algorithms for these very different ones and couldn’t find any.

[09:02] This may seem like there’s no such solution. Getting a bit more technical, NP problems, NP-complete problems in particular, seem to require search over an exponentially large space, like all the possible solutions to some system of equations. Somehow, a solution is about having a general method for cutting down this exponentially large search space into searching over something much smaller in a clever way.

[09:48] And this should apply to all these problems. So that would be equal. And people don’t think that there is such a way. But as I said, these are intuitions that are maybe pretty strong. And I think that’s the only, tomorrow somebody finds an efficient algorithm for NP-complete problems using some new idea that nobody ever thought of.

Ryan:

[10:22] That would change the world. In software engineering, we use algorithms all the time, and they often have these very satisfying properties where the solution we use takes something where the brute force is much larger down to something that’s polynomial time or something very reasonable. But in all these NP problems, it’s pretty unsatisfying that the best we could do is brute force.

[10:55] Am I understanding that correctly?

Avi:

[10:57] Yeah, you’re understanding it perfectly correctly. I think that maybe what you are getting at is that NP-complete problems are, a particular NP-complete problem is really a family of instances. I mean, you want to find a Hamiltonian cycle, some Traveling Salesman Problem through some map. There’s this map and that map and this network and other network. And there are many, many instances when we talk about an algorithm for them.

[11:32] Typically, we say an algorithm is efficient if it’s efficient and correct on all of them. In software engineering and many aspects of optimization in real-world problems, the instances are a much more restricted set. They come from trying to verify a particular protocol or, I know, protein folding. If you want to think about the biological example, the way it is phrased as an NP problem is that you want to minimize the energy of a system under various constraints that have to relate to the chemistry of the various molecules and atoms that appear in the protein.

[12:25] And this kind of optimization problem one can easily prove is NP-hard. Nevertheless, our body folds proteins all the time, extremely efficiently. Now why is that? Of course, I don’t really know why that is. But it would seem that evolution designed proteins in such a way that this, you know, folding them, only them, we don’t have that many proteins. There are not exponentially many proteins in the body.

[12:57] There are only a few that may be more prone to efficient energy minimization. More generally, I think in software engineering, at least in verification and testing, you often want to check that something meets the specifications. You usually translate it into a satisfiability question of Boolean formulas. The formulas that you get, the instances that you get out of these have some structure, and maybe for them, various greedy approaches or clever methods can be applied.

[13:38] Not just greedy, but clever and efficient methods work. In other words, making it short. Instances that come up are not the worst-case instances. We have many examples of this that we can actually prove. For example, early on it was recognized that the Simplex method, which is an algorithm to solve linear programming problems, on the one hand, requires exponential time for the worst-case system of inequalities.

[14:16] But in practice, it seems to run in linear time. Evidently, the set of linear systems that we tackle in real life has some extra structure that allows this particular algorithm to solve them efficiently. Of course, today we know an algorithm that’s efficient for all linear programs, but it’s much more complicated, and I don’t know how much it’s used in practice. But the Simplex method is very simple to run, and usually, it works.

14:51 — What if you relaxed correctness

Ryan:

[14:51] You mentioned protein folding, and that’s an interesting one because it’s an NP problem, but it was solved to an acceptable extent. When I was reading about it, I found that it wasn’t solved for 100% of cases. It provides a solution and a confidence score, but it doesn’t give you the answer that is 100% accurate and fully computed. So when you think about these NP problems, what if you relaxed the criteria of being correct?

[15:27] Could the asymptotic complexity be less than the exponential that we’re talking about?

Avi:

[15:33] Yeah, it’s a very natural and good question. So in the protein folding, you are talking about AlphaFold, which is an algorithm or heuristic that was learned from existing proteins and works very well on instances it didn’t see but are still proteins coming from the body. Even if it did perfectly on these, that would still be following the discussion I mentioned before because the instances it is trying to solve come from real life and probably have extra structure that allows efficient solutions.

[16:13] But you are right that one of the things that people naturally do when they cannot solve an optimization problem exactly because it’s NP-complete or NP-hard. Maybe I should say NP-complete and NP-hard problems are those problems in NP that are as hard as all problems in NP. If you solved one efficiently, you solved all efficiently. If you prove one hard, then you prove all of them hard. So they somehow capture the complexity of the class.

[16:46] The Traveling Salesman Problem is an example. Portraying folding in this framework of energy minimization is one, and Boolean satisfiability and so on. Okay, so you want to optimize something, and it’s NP-complete or NP-hard. That’s a sign that you shouldn’t try to find an algorithm that works always. But one thing you can do is an approximation. That’s a very natural relaxation. You don’t want the optimum.

[17:16] You are willing to live with a factor of 2 from optimum or 10% from optimum or some factor away from optimum. NP-completeness was discovered in the early 70s. In the early 90s, there was a breakthrough called the PCP theorem. The PCP theorem allows you to argue hardness of even approximation problems. For example, you can show that satisfiability of Boolean formulas—it’s not just, so you want to, let’s say, find an assignment to a bunch of constraints, let’s say Boolean constraints, let’s say on three variables, for simplicity.

[18:06] So you want to satisfy all of them. The relaxation would be okay, I want to satisfy 90% of them, or as good a fraction as I can if I cannot get 100% of them. The PCP theorem, one way to phrase it, and suddenly one of the major consequences is that we know that you cannot even achieve a very good approximation. To illustrate how tight this understanding is for some problems, when there are only three variables per constraint, if you just guess at random the values to the variables in the assignment, you will satisfy 7 or 8 of the constraints.

[18:55] This is because each particular constraint, usually the disjunction of three variables, will be true with probability seven-eighths. So if you randomly assign values to the variables without thinking or looking at the formula, you can satisfy seven-eighths of them. The PCP theorem, along with a significant strengthening by Johan Håstad, tells you that if you want to satisfy seven-eighths plus epsilon, that’s already NP-hard.

[19:30] So we know today to argue the difficulty not only of finding the perfect solution, the optimal solution, but even how close to optimal you can get for lots and lots of problems, large classes of problems, in particular constraint optimization problems, you know, of the type of satisfiability.

Ryan:

[19:56] So even if you solve just the slightest epsilon for any trivially small epsilon, it’s just as hard as doing.

Avi:

[20:06] It’s as hard as finding the optimum source.

Ryan:

[20:09] You mentioned NP-hard, NP-complete, and P. I know there are other complexity classes. Maybe you could give some more context on the whole space of complexity classes.

Avi:

[20:22] Yeah, so I’ll just remark that we have a zoo of complexity classes. In fact, there’s a website called the Complexity Zoo, which has in it hundreds and hundreds of complexity classes of all types. The website was started by Scott Aaronson. Once we realize that there’s a whole zoo out there, we want to know the relationship between these classes. Okay, so what is the point about these classes like P and NP?

[21:02] We want to classify problems by the amount of resources they take. So P (polynomial time) is a class of problems that can be solved in an amount of time that relates polynomially to the size of the data. Of course, you expect that as the data grows, the instance grows, and you’ll spend more time. But if it’s linear time, quadratic time, or cubic time, you say, let me call it at least theoretically efficient.

[21:35] And NP, you classify not the time to solve, but the time to verify a solution if it was given. You want this to be polynomial time. So it’s very different. But first of all, there are many more time functions than just polynomial, exponential, and doubly exponential. And you can go to finite, and one fundamental result, starting with Turing’s paper, is that there are problems that are simply unsolvable by computers.

[22:16] So really, the first complexity class is a class of solvable problems, decidable problems. Turing shows that these are not all problems. There are natural problems that are not decidable at all. But time is only one resource, and we are interested, both from practical and theoretical reasons, in lots of other resources. Memory is a costly resource, and we want to minimize the space invested in a computation.

[22:47] In problems with several parties, you are interested in the amount of communication exchange, not the internal computational effort that they exert themselves, but how much. If you talk to a satellite, you want to minimize the communication. You can think about energy; there are plenty of other resources, like parallel time. How much can you speed up the computation of a problem if you work on it not with one computer, but with n computers?

[23:21] If your data is of size, then does it allow it to solve it immediately? Or maybe it doesn’t help at all? Complexity classes classify problems according to how much resources they need. There are many of these, and one important part of that is a major theme in the methodology of complexity theory. Coming with it are attempts to understand complexity classes as problems that relate to each other, even if you don’t know their complexity.

[24:02] So, for example, NP is a class, and we don’t know the complexity of all problems in this or NP-complete problems. We don’t know whether they are all easy or hard, but we do know we have efficient algorithms to translate one into the other. It’s not obvious that if you want to solve a satisfiability problem, you don’t know how. But you have an algorithm to solve Sudoku problems, not just three by three, but also four by four. If you had an efficient method to solve Sudoku problems, P equals NP, because you can efficiently take an instance of satisfiability, the Traveling Salesman Problem, or factoring integers, and from the solution of the Sudoku problem, you can translate back and get the best solution for your original Traveling Salesman Problem.

[25:04] So we are interested in efficient reductions between problems whose complexity we don’t know, but we want to relate them to each other. We build some kind of partial order of the hardness of different problems. There are really, you don’t want to know how many problems we have summarized for constraints that real-world technology may impose or cause for. Some are naturally arising mathematically because they are interesting.

25:38 — Why NP complete problems are equivalent

Ryan:

[25:38] You mentioned that these NP-complete problems are all equivalent in some way. What’s the proof to equate a SAT solving problem to a graph coloring problem or something like that? Are they all similar in nature? Are they kind of bespoke to the problem you’re going to and from?

Avi:

[25:59] That’s a very good question. You will find that in most of these, they call them reduction algorithms to translate one problem to another. Most of them are relatively simple, but not all. I talked about the PCP theorem before, showing that approximation is hardest is very highly non-trivial. But most NP-complete problems prove equivalence, like between graph coloring and satisfiability. This goes both ways; they are both complete.

[26:32] So it goes both ways. It is simple. The first one was the satisfiability problem. The original papers of Steve Cook and Leonid Levin, defining NP-completeness and proving that NP-complete problems exist, both focused on this. The first problem they proved complete was satisfiability. Why is this so? Why can all these other problems be reduced to satisfiability? The reason is really that computation is local. Computation is local.

[27:10] I mean that if you think about your computer, your laptop, or any computational device, usually it manipulates bits by some local operations. You look at these two registers and you add them, or you take these two bits and exclusive OR them or AND them or something, and you do a sequence of simple operations and you arrive at a conclusion. This locality can be translated to a bunch of constraints or what would be a consistent sequence of states of your machine.

[27:48] If you look at the time evolution of any computation, memories in some states and in another state change locally. If you could guess, and that’s the NP part, if somebody provided you the table of computation, you could check that it’s consistent by writing a bunch of local constraints or some conjunction of very few variables all over this table of computation and write down a satisfiability problem.

[28:25] I want this to be true, this to be true, and this to be true for some assignment. What the assignment has to satisfy should be a table like this. All these constraints should be satisfied, should be true, and it has to be consistent with the input given, right? Some of the inputs have to match the first time zero content of the machine. This locality allows for many of these translations.

[28:56] If you realize this, you will quickly find a reduction from coloring to satisfiability. It’s also not clear why coloring is an evolution of computation, but it starts already with local constraints. You want a few colors and that every constraint is satisfied. In some cases, it’s even easier. But the reduction from any NP computation—that’s what NP-complete means. Anything in NP can be translated to satisfiability.

[29:29] NP is just a computation of a Turing machine, say only that you don’t know what the computation is. If somebody guessed it or showed it to you, then all you need is to verify this verification is local. And that’s the source of many simple NP-completeness reductions. Let me give you a really simple example. Why is factoring integers reducible to satisfiability? Because if I gave you factors of an integer, you could just multiply them and check that they equal the input.

[30:15] This multiplication is a simple algorithm, right? I mean, we know how to do it from second grade or something.

30:33 — Space vs time complexity

Ryan:

[30:33] You really, the factors you mentioned, time complexity and space complexity. Intuitively, in software engineering, we often trade those off for each other. Is there general theory on that, on how these two trade off?

Avi:

[30:50] Yeah, we have all these questions. They are natural questions for complexity theorists. How do two resources relate to each other, and can they trade off? Or maybe you can minimize them both at the same time. It depends on the problem. There are problems for which we know results, like multiplying time and space. It has to be at least the square of the length of the input. So you can either do it with very small space, like logarithmic space, but you need to pay maybe quadratic time.

[31:32] And if you want linear time, you have to pay linear space or close to linear space. There are results of this type in various models of computation. That’s another thing you play with. Not all of them are equivalent in principle, but if you care about exact time or exact space, they are not equivalent. So there are problems for which there is a trade-off. There are problems where you can solve them in linear time and logarithmic space.

[32:05] One very important general result that is the breakthrough from last year is a result of Ryan Williams. I’ll tell you what it is, but first, I need to describe what we knew about time and space. There’s a clear inequality, right? If you run in time T, you will never use more than space T, because you will never visit so many cells or registers in your machine. So space is at most time. Fifty years ago, Valiant, Paul, and Pipin improved this a little bit.

[32:41] They said that if you run in time T, there’s an equivalent computation that will do this in slightly less space than T over log T. That was a big result. The running time, of course, will blow up. It will be very, very expensive time-wise, but at least you can save on space a little bit. This was believed for almost 50 years to be the best you can do. In fact, we had arguments that in certain models, in stylized models, models of computation, you cannot improve that.

[33:19] I will not describe this restricted model, but in some way, natural restricted models, you cannot beat this. What Ryan Williams found out last year is that, in fact, much better can be done. Any computation that runs in time T can be simulated by another algorithm that uses only the square root of T space, far less than before. The algorithm is quite sophisticated and uses an earlier result of James Cook, the son of Steve Cook, known for defining NP-completeness, and Ian Meltz, which was an essential technical ingredient.

[34:07] But anyway, there’s a really interesting way to save space. A very non-trivial way to save space. Again, this algorithm will run in a lot of time, but at least you have a sense that the two parameters are related in a highly non-trivial way.

Ryan:

[34:29] Is this for a general problem?

Avi:

[34:31] General problem for Turing machines? Okay, it may not work on random access machines or.

Ryan:

[34:37] Yeah, what’s the trick? If you could explain intuitively, or is it too deep to explain?

Avi:

[34:45] It is really technically explaining it. It’s too deep. But I’ll give you a much earlier result which indicates that really mysterious things can be done in small space. Again, it’s probably over 40 years ago, people discovered the following. Dave Barrington discovered the following really interesting phenomenon. So suppose all you want is to count. I give you a sequence of bits, and you want to count.

[35:20] Say you want to know whether there are more zeros than ones, like the majority problem. Who wants the vote, zeros or ones? Okay, well, you can count. You can just add them up, and this takes logarithmic space; it seems the best you can do. You want to count up to N. To represent N takes log N bits, right? Any number between 1 and N takes log N bits. It seems essential. It turns out that there’s an algorithm I have to define exactly how it works.

[35:58] I will not define exactly how it works, but it solves this problem. In constant space, it seems like you can count arbitrarily high. All you need is access to whatever bit you like, whenever you like. I want the 17th one, I want the 81st one. If you can have random access to the bits, even if you have constant space, you can tell whether there are more zeros than ones or ones and zeros, regardless of how long the input is.

[36:33] The trick is that somehow you use noncommutative algebra. Noncommutative algebra is not something everybody’s familiar with. I mean, we usually put the cars behind the horse and not in front of the horse. It matters. In software engineering, we do this before that; it’s very important that the order of things matters. So you can think about applying permutations one after the other.

[37:10] If I rotate a circle and then take a mirror image, it will give me the... So these are two permutations, right? I mean, say I have endpoints in a circle. I can shift it by one, or I can, let’s say, flip it around some diameter. There are two permutations, and they don’t commute. If I first flip and then rotate, I’ll get something else than if I first rotate. Think of all the points as colored, so they don’t commute.

[37:45] Barrington is using permutations to encode the bits in the input, specifically permutations of size five. When he sees a one, he may rotate, and when he sees a zero, he flips. The non-commutativity of these operations allows him to carry out a general formula, a formula of ANDs, ORs, and NOTs. Any formula of some size S enables him to simulate the ANDs, ORs, and NOTs in this formula through rotations and flips of these five-sided pentagons.

[38:45] And so just to remember the configuration of this pentagon, you need five bits, right? So you need constant space. How this captures the computation, I can tell you one more sentence. Maybe it will not be clear to everybody, but these non-commuting things allow you to do the following type of operation: do a rotation, do a flip, then rotate back and flip back. This is called the commutator, and it can simulate an AND gate.

[39:33] I can. It’s too much to describe in words without how it does it. But there’s a very famous analogy, which is a riddle that captures this NP-complete problem. You want to hang a painting in the following way: you have a painting, it has a string connecting the two sides, but you want to hang it not on one nail, but on two nails.

[40:08] Okay. There are two nails on the wall, and you can do with the string whatever you want. The property you want is that if the two nails are there, it’s hanging, and everybody’s happy. If you pull out one nail, it doesn’t matter which, the picture falls to the floor. How would you loop the string around these nails so that this happens? Clearly, this is an AND gate or an OR gate. Right? If one... Yeah. And this has to do.

[40:43] I mean, if anybody finds a solution, they realize what non-commutativity I’m talking about. It’s a nice fiddle. Anyway, this type of trick, where you can really do this type of result in small space, is something you wouldn’t imagine possible. It’s a striking example. When I heard this result for the first time, I was a postdoc, and somebody told me, and I just didn’t believe it was possible.

[41:18] You cannot count arbitrarily high with. The trick that Cook and Mertz have in their algorithm is a solution to a natural problem in much less space than you would think. It uses some sense tricks of this nature.

Ryan:

[41:44] If you’re counting arbitrarily large, that is information. You’re saying maybe the intuition is that information is encoded in the sequence of operations.

Avi:

[41:55] It is encoded. Of course, you are not delivering the final count. You just say whether there are more zeros than ones. If you have to write it down, and if the answer takes some number of bits, then you need this number to write it down. If you have a decision problem, say like yes or no, like are there more zeros than ones? Then you can do it for any size input in constant space.

Ryan:

[42:26] That’s incredible.

Avi:

[42:28] Pretty incredible, yeah. And by the way, this highly applicable result used in cryptography in fundamental ways is used in various places, and it’s an extremely useful result here. Unlike the Ryan Williams result, if you have a formula of size S, you can do it in constant space. The time of this algorithm does not blow up really a lot; it’s just quadratic. So it’s quadratic time.

[43:00] So this is something actually doable, useful, efficient, and magical.

43:06 — Why people use SAT solvers

Ryan:

[43:06] We talked about the equivalence between these NP-complete problems. In practice, a lot of people solve other problems using SAT solvers. I would have thought that would be less efficient, though, because you have to translate it and then you’re doing it in almost like a different problem space.

Avi:

[43:25] Yeah, you usually also enlarge the instance. In translation, you enlarge the instance. So what your question is, why do people use SAT solvers? I would say that there’s no problem other than satisfiability that people thought so hard about really optimizing the heuristics that efficiently work on many instances. There are very clever ways in which you can try to start. I mean, you are not going to guess all the N bits, but you want to start by guessing some bits that seem more pivotal. If you like in dominoes, maybe when you set them to one value, they force many other values to be set.

[44:22] And if that happens, you cut down your subspace. There are many heuristics of this type and also more clever than this that allow solving such satisfiability problems if they have such structure. Again, in the worst case, it will not help you. There is, in fact, a conjecture that is much stronger somehow than NP-completeness, you know, NP-complete problems taking exponential time. It says that we don’t expect any savings.

[44:56] We expect satisfiability problems to require really true to some constant times N. It’s not going to be two to the square or N to the log N or something like this. The worst case may be hard, but people optimize attacks on many such formulas that somehow have all sorts of structural properties arising maybe in practice. I think that’s the main reason they are used. It’s also convenient.

[45:33] I mean, it’s very common. People do it for testing specifications that programs or protocols meet. The specifications are usually easily translated into simple constraints, and that’s almost a satisfiability problem.

45:53 — Randomness is a resource

Ryan:

[45:53] We talked about all the different types of resources in an algorithm, and in one of your talks, you said something where you consider randomness another resource for an algorithm. What do you mean when you say randomness is a resource for an algorithm?

Avi:

[46:10] In the early 70s, of course, randomized algorithms existed since antiquity, and everybody was tossing coins for a lot of reasons. Statisticians do sampling and use randomness. But when algorithmics, designing efficient algorithms, became big—once we had computers—people realized that it can really enhance all sorts of computation. For example, we had no idea how to test the primality of a number.

[46:46] And in the 70s, both Michael Rabin and Sulwin Slauson found probabilistic algorithms, which are fast to test primality. Okay, so what does it mean, a probabilistic algorithm? It’s an algorithm that’s allowed to make random choices. You can think that there is an internal little person or device inside that tosses coins. Of course, that’s not what happens. There’s no little person sitting in your laptop.

[47:15] So the question is, where do you get these random bits? It’s very important to stress that in all these probabilistic algorithms, the underlying assumption is that the bits you get are perfect. They are half, half each one and independent of each other. It’s like a uniform distribution on all possibilities. Now, where do you get this? I mean, where seriously do you get this? If you run a public algorithm on your laptop, since it doesn’t have this person inside tossing coins, it does something well.

[47:52] High-quality randomness of this type costs money, time, and memory. What do I mean by “cost money”? You can have a very cheap solution. You can just measure the thermal noise in your computer or use one of the Intel chips. You can measure Internet traffic and sample it, believing it’s random, and all of these things are used. You can do something mathematical, have some simple procedure that is actually deterministic, like a linear congruence generator or something like this, and believe it’s random.

[48:32] Or you can take the digits of π. It also looks random in some sense. There are many things you can use, but you don’t know they are random. If you want them to be random, one source— even this is not a perfect source— but we have quantum mechanics. People believe it works in life. It seems like a true theory of nature. There are all sorts of arguments about it, but let’s put them aside.

[49:01] They predict. If you measure photons coming out from some source and you measure their spin, whether it’s up or down, the prediction is that each one is half, half, and it’s independent of each other. That’s very nice. But if you are going to build this device, it’s going to cost you a lot of money, and let alone that it will not be perfect. But let’s leave this aside. Anyway, since you want high-quality randomness, you have some way of guaranteeing, I mean, you are running it really on your laptop.

[49:34] It’s not okay. The paper was written; we can test primality with a probabilistic algorithm. Now we want to run it. What do you use? It makes sense to ask lots of questions about randomness. What guarantees the quality of the randomness? Maybe you can minimize the use of randomness. By the way, another issue with randomness is that the outcome is a random variable, and it’s not always correct. Right?

[50:02] The whole point, in most of these algorithms, is that you just have some small probability of error. If you have a deterministic algorithm, there’s no error, so you also don’t like the error. Treating it as a resource is simply a convenient, complex theoretic way of saying how do we understand the amount of this resource we have to invest, or the number of bits we have to invest, their quality, and so on.

[50:30] So it’s almost automatic. If you think like a complexity theorist, how do we minimize for a particular algorithm, like primarily testing? Do you really need this? Do you need them to be independent? Maybe you can generate them in a, you know, maybe you need n bits, but you can start from squared n bits or from log N bits that are truly random and make from them in some deterministic way, a sort of pseudo-random sequence you can call it, which is a good name actually, that will, for the purpose of this algorithm, look as if it was random.

[51:13] If you could do that, you don’t need all these bits; you can use much fewer. This whole theory of reducing randomness, derandomization, and removing randomness is a huge field, and there are many problems and many ways of doing it. The story with primality is actually fascinating. I mentioned these two algorithms that were invented, and people were wondering about the deterministic algorithm for primality.

[51:45] It’s a problem that has been articulated already by Gauss in the most complex theoretic way you can. Back in his days, hundreds of years ago, and maybe 150, I don’t know, he was asking for an indefatigable calculator. Calculator, of course, is a person. But he wanted this to be efficient. That’s basically what he was looking for, a primality test for large numbers. Because they really wanted to know about numbers, maybe only a few tens of digits.

[52:23] They wanted to know whether the numbers are prime or not. There was no efficient algorithm. You can wonder about the Miller-Rabin or Solovay-Strassen algorithms; maybe you can use less randomness or structures, randomness that you can generate from fewer bits, but nobody has any good idea. Then, in the early 2000s, Agarwal, Kayal, and Saxena devised a different probabilistic primality test, and it’s different.

[52:55] So the analysis of randomness in it is different. Once you understand how randomness is used, you can say, “Ah, we don’t need to have totally independent bits. It’s okay if it has some structure.” They found, using number-theoretic methods, a tool to generate pseudo-randomness from very few bits. That’s how it works. Sometimes, deterministic versions of probabilistic algorithms are discovered simply by understanding how the algorithm uses randomness, analyzing the algorithm, and realizing it doesn’t use all that much.

Ryan:

[53:49] You mentioned a few times the quality of the randomness. How do you quantify the quality of random bits?

Avi:

[53:56] This is basically a question about what pseudorandomness is. The general answer is that it depends. You want to fool a particular algorithm, let’s say, for primality. You want the randomness to fool this algorithm; you want it not to notice that you switched from perfect randomness to something that’s really far less random, has far less entropy, but the analysis works nonetheless. For another algorithm, maybe you need a completely different type of pseudorandomness for it.

[54:35] There are a set of examples of algorithmic problems for which, when you look at the analysis, they are much simpler than this primality testing. It seems to use not the independence of all the n bits. You really need only every pair of them to be independent, or every triple. Spaces of random variables that have this property are much smaller. You can generate them from log N bits, and so you can fool them with this.

[55:08] So the quality of randomness is a phrase. I like that. The quality of randomness is in the eye of the beholder or in the computational power of the beholder. You really need it to be just as good as the tests that are applied to it. You just want to be completely pragmatic. You don’t care whether it’s random or not. You just care that an observer of a particular structure or particular computational power will not distinguish the pseudo-random distribution, which may have much less randomness in it than the perfect one.

55:48 — Randomness depends on computational power

Ryan:

[55:48] In one of your lectures, actually, you mentioned that, and you gave a good example. I was wondering if you could give the intuitive example of why randomness is a function of the observer’s computational power.

Avi:

[56:00] Yeah. So if you watch this talk, the example I give is taken from this fundamental paper of Manuel Blum and Silvio Micali. They tell you to consider three experiments, and the experiments go as follows. They are always between you and me. You are the observer, I am the cause. I have a coin on my finger, and I toss it. Just as it leaves my finger, you are supposed to predict what the value will be when it falls on the floor, like in two seconds.

[56:38] You have to immediately say, heads or tails, okay? Before you have to predict, before it falls. Well, this is the first experiment. And you know what I ask and what they ask? What do you think is a success, your success? What are your chances of predicting it? And the obvious answer is, you know, one half. You know what, how can it help? You know, what can you do? The second is when you sit there, but you have a laptop like you have now.

[57:10] And, yeah, what can you do? I don’t know how fast you type, but the coin will be on the floor in a second. The third experiment is where your laptop is connected to a Cray supercomputer, and the Cray supercomputer is connected to a bunch of sensors and cameras and whatever devices you want. They are all trained on my finger, okay? And so as it leaves my finger, the coin. This apparatus certainly is more than enough to calculate all the angular momentum of the coin and the distance to the floor and the humidity in the air and whatever parameter that completely determines its motion.

[57:57] In particular, how it will land in a split second, far less than the time needed. So the real point in this example is that the experiment, the random coin toss, did not change in all of them. It is the same what masses of people in math, physics, and philosophy have defined randomness in various ways. There are many definitions of randomness, and they all focus on this event, on the chronos or sequence of coin tosses.

[58:37] We in computational complexity theory, starting from this paper, don’t care about this event. The event stays the same, and all it focuses on is the observer. The only thing that changed in these three experiments is the computational power. We want to know how much entropy is in the coin. If you don’t have enough computational power, it seems to be full entropy. It’s half. Half. You don’t know what it will be; if you have enough computational power, you can predict it completely, and then it has zero entropy.

[59:12] So what changed is the observer. The observer is this algorithm we talked about before. Any test that’s applied to a distribution to test the quality of randomness, depending on the test, you want to make sure to use as little true randomness in a way that the observer will not notice. This is the source of many of the important theorems that show you can remove randomness or reduce randomness in probabilistic algorithms with or without assumptions.

[59:55] And really, they are behind this understanding that we have today. Randomness in algorithms is not as powerful as we thought. It’s like if you know that P is different from NP or you have a Traveling Salesman Problem that is exponentially hard or sustainable. If you know that, then in fact, I can give you a pseudorandom generator that will derandomize any. We know that under this assumption, we have what we call P equals BPP.

[01:00:28] Anything that has an efficient probabilistic algorithm also has an efficient deterministic algorithm. You just need to know that there is a hard function somewhere. Under what assumptions do we know P equals BPP? The very natural assumption is that one of these NP-complete problems, or in fact even problems in higher classes, requires a lot of hardware and requires exponential size circuits to solve. We believe that it’s like the P versus NP problem in strength, and you cannot count down the exponential subspace.

[01:01:14] And so maybe people are happier with this assumption, but this assumption about time complexity has no randomness in it. At first, it’s shocking that it’s actually related to the problem of removing randomness from algorithms. So that’s one fundamental connection in this hardness versus randomness paradigm. But we know even more. We know that it goes both ways. We know that if you are trying to remove randomness from some algorithms, if you can do that, then you’ve found the hard function.

[01:02:02] So there’s really an almost. It’s not an exact if and only if, but again, there are many variants of this. In some variants, if and only if it’s stronger, but we really know that it’s one or the other. Since it’s hard to believe that all these hard problems are easy, or the seemingly hard problem like P versus NP, we tend much more to believe the other alternative, which is that randomness in algorithms is weak and you can remove it.

Ryan:

[01:03:20] Is there an intuitive explanation for the relationship between problem hardness and randomness?

Avi:

[01:03:25] Yeah, there’s almost always an intuitive explanation for this. So what’s the hard problem? I give you the input. You are a limited computer. You are a polynomial time algorithm. I give you the input. Let’s say it’s a Traveling Salesman Problem, or you don’t know the answer. So there’s some entropy in the answer, right? In the same sense we discussed before, there’s some entropy, a tiny entropy, maybe exponentially small entropy, because if the input is random, maybe only a few instances are hard, and all the others you can solve. But at least you see a hint of a relationship between hardness and entropy.

[01:04:12] This, of course, is not satisfactory because we want half. We want the answer to be like a coin toss, not like a very biased coin toss. So we need to find ways to amplify this uncertainty for you. Amplify the hardness. We want not just that some instances will be hard, but since they are random instances, whether the answer is yes or no, for polynomial time, the observer will be half and half.

[01:04:46] It will not be able to gain an epsilon advantage over a random guess. For this, we developed all sorts of methods that amplify randomness. Now, this doesn’t solve the problem at all because I managed to take n bits of a random instance of a problem and manufacture one bit. That’s hard for you to guess. But I could have taken the first random bit I picked.

[01:05:20] What you want to do is the opposite. You want a method that will take a few and generate many. That’s a pseudorandom generator. The pseudorandom generator starts with a few truly random bits and generates many. For this, we need other tools. There’s something that I developed with Noam Nisan called the NW generator. I like to say that people say making the first million dollars is easy, and afterwards, you can make many more millions.

[01:05:57] You can think of it this way. The hardness in the way I describe it allows you to make $1 million. But once you can make one, there are methods to make many more, in fact, more than the number of bits you started with. But still, their quality depends on the hardness of the function you use. We still use this and your limitation of being in polynomial time. So you can also solve this hard instance.

[01:06:30] You build many instances from a short seed. You start from maybe a logarithmically long seed. You take many subsets of this, like N. You will need n of them. I ask you for the value of the solution to each one. Each one is hard, but we want to say that they are simultaneously hard. You cannot tell them apart. There is no correlation that you can detect between them, even though they are extremely correlated.

[01:07:00] So, we have methods to do that as well. The connection you embed, having a hard problem to a limited observer, means there’s some uncertainty about the answer. That’s the key. That’s a starting point.

Ryan:

[01:07:14] So when you say that the quality of the randomness is a function of the computational power of the observer, one thought that comes to mind is, imagine I have maybe infinite compute. Then nothing is truly random. Is that accurate to say? Like, you know, if I had infinite compute, the weather is not random. Nothing is random.

Avi:

[01:07:42] Okay. So it points to the fact that you have to really be careful about what question you are asking about randomness. If you want to apply this to probabilistic algorithms, then it’s really important that you are limited computationally. If you have infinite compute time, you will be able to distinguish a distribution with full entropy from one coming from a generator with low entropy.

[01:08:19] So you could do that. But one thing you cannot do with infinite compute is something that’s information theoretic. There are many other things we need to do with random bits than just run probabilistic algorithms. For example, we want to generate passwords for our cryptography for security systems. The very demand is that they are random, right? I mean, Shannon’s theorem tells you the quality of your password is as good as the entropy in it.

[01:08:58] So the notion of a secret rests on the true randomness of the bits. If I ask you to produce a random bit, or you ask me to produce a random bit, whether you have infinite computational power or not doesn’t matter. To generate this random bit, it has to be random. Your demand is not that it will succeed in some test; you really want the probability of heads to be half and tails to be half.

[01:09:34] Then this definition or this task of producing randomness is not related to your computational power. Lucky or unlucky, I don’t know. For us, many times we do have implicit or explicit tests for this randomness. You say, “I can break this system or something.” Then you can maybe use pseudorandomness. But if your very demand is to have a random event, then you need to produce a random event.

Ryan:

[01:10:08] I saw in my research that there are ways to create higher quality randomness by aggregating weaker sources. How does that work?

Avi:

[01:10:22] Well, it’s another theory I talked about before, the theory of pseudorandomness. There’s a whole different theory. They are related in nontrivial ways, but it’s called the theory of randomness extraction or randomness purification, maybe. And this is exactly what you asked about. We imagine that the world may not give us perfect randomness, but we have all these events we cannot predict, like the weather, various quantum phenomena, or sunspots. We cannot predict the stock prices, right? Otherwise, the market will be.

[01:11:04] But these are not, even though these are unpredictable events, they are somewhat predictable. I mean, the weather tomorrow is more likely to be similar to the weather today than the opposite. So there are correlations. If you sample this weather, for example, there are also biases. I mean, sometimes in the spring it’s more, or in the summer it’s more likely to be hot than cold.

[01:11:33] So there are biases and correlations. These are called weak random sources, and there’s a mathematical quantification of how weak they are. Basically, we talk about the amount of entropy in them. You have N bits; potentially, you can have entropy N, but maybe they were generated from a generator or they came from some source. It has some entropy in it, but you have no idea where. Maybe half of them are fixed and half of them are chaotic, and you don’t know which half.

[01:12:10] Maybe they are each half, three-quarters zero and one-quarter heads or tails, have a virus that also has lots of entropy, but not full. The correlations may be even more complicated. You can imagine all sorts of situations like this, and you mathematically model it by saying, okay, there are N bits. It’s a probability distribution of N bits which has some entropy, let’s say the square root of N.

[01:12:39] You don’t know where, and you want to use it in a probabilistic algorithm. So it’s a basic question: can we use physical events, weak sources coming from nature, maybe in probabilistic algorithms? It’s certainly not obvious. I mean, just feeding them to the algorithm as is is not going to work. There are very simple algorithms that will succeed on perfect randomness and will fail, making an error.

[01:13:14] But the theory of randomness purification wants to take this sample of n bits with some amount of entropy, which is not full, and massage it to create from it maybe a shorter string that is of higher quality. Ideally, it would be perfect random bits. So, ideally, you can imagine that if I have n random bits and I have entropy root N, maybe I can massage out of there square root N, or maybe just the fourth root of n bits that will be essentially uniformly distributed.

[01:13:59] Then if I have an algorithm that just needs this amount, that’s what I feel. Turns out you cannot do that; it’s impossible. But what you can do is not produce one string of length, let’s say, the square root of N, but you can produce N to the hundred of them, okay? And what you are guaranteed is that this randomness purification device is an official algorithm that takes any distribution with entropy in it and produces many blocks, polynomially many blocks, of the length roughly the entropy.

[01:14:41] What you would hope to be perfect and what you are guaranteed is that 99% of them are truly perfect. So you have many, 99% of them are as good as perfect. And there’s 1% that is bad. You don’t know which it is. But this already solves your problem because you can run your algorithm on each one of them and take a majority vote. Doing this is extremely complicated. There are many ways of doing it. There are variants of various assumptions you can make about the entropy.

[01:15:21] Maybe you have several weak sources that are not related to each other. There are various variants, and this theory is very well developed. But we know that the statement I made essentially is accurate. If you have one source with some entropy in it, you can generate polynomially many samples of the length of almost the entropy, and most of them will be perfect. So that’s as useful as having one.

Ryan:

[01:15:55] Yeah, I vaguely remember skimming a bunch of papers, and one of the papers had this figure with a two-dimensional array of numbers. I think they were drawing rectangles or something like that. Is that one of the ideas?

Avi:

[01:16:11] So it’s a notion of pseudorandomness, which is related to one particular variant of this extraction problem where you have several weak sources, not just one. One is the hardest because that’s... Yeah, but maybe you have a few that are independent, and each of them has some entropy in it. I think what you are referring to is this sum product theorem. I cannot draw the pictures of that, but you don’t really need to.

[01:16:45] So let’s just think about this problem. You have three sources of randomness; they are each weak, so each has just a little bit of entropy in it. All you want to produce is one source with somewhat larger entropy. You are losing the number of sources, but you are gaining entropy. If you can repeat this, you eventually get to full entropy. So that’s the idea. What combination of which sources would you take?

[01:17:17] It’s not obvious. And the solution? Yeah, there certainly is one. Maybe you mean this one. There’s a paper I have with Barack and Impagliazzo about this particular problem, and we are using a result in what’s called arithmetic combinatorics. I can explain it very simply. I think that we learn about addition in second grade and about multiplication in third grade, maybe. The only motivation for multiplication is that it’s a way to shortcut repeated addition.

[01:17:56] And that’s all we talk about. Then if you go to college, you learn that sums and products are the basic operations of fields. You can do it with integers, rational numbers, complex numbers, and finite fields. And so you remember these are the basic operations. But how do they relate to each other? It’s not clear. Maybe there are many ways to ask this question. But one way to ask this question that was suggested by Erdős and was solved for the first time by Erdős and Semerédi, and then in a much more applicable form we really need by Bourgain and Katzentau, said the following.

[01:18:38] Just think of a set of integers and add all pairs. So you start with, let’s say, K integers and add all pairs. How many new integers do you get? This you should think of as entropy increase. If you get many more somehow. So you have K integers. Well, it depends on what they are. I mean, if they are the first K integers and you add all pairs up, you get the numbers between 1 and 2K. That’s not much larger; that’s a factor of 2 larger.

[01:19:11] If you want to increase entropy, you want to get some power of K bigger than one. Of course, if you take a random set of integers, you’ll get K squared. They’ll all be distinct because K squared integers. So somehow you want to say, it would be great if, for any set, when you take all pairs, you group. But it’s not true. By this example of the interval, you cannot take sums; you can take products, but also products don’t always increase.

[01:19:48] If you take 1, 2, 4, 8, you take a geometric progression, you multiply all pairs, you just get twice as many as you started with. So that’s also not good. The magic in this sum-product theorem is that sums and products are orthogonal to each other. Somehow, if one of them fails to grow a set, then the other will grow this set. This is an amazing result. In this context, you think of the outcome of your sources as numbers.

[01:20:31] You multiply the first pair and add them to the third. You mix the sum and product. This guarantees growth. This is not obvious, but if you do this, it guarantees that you grow. The number of different values you get is significantly more than K. Maybe it’s K to the 1.5 or something. Entropy does increase, and that’s the source. You need to prove it. Distributionally, it’s not just the size.

[01:21:03] The entropy is not just size; it’s more than that. Anyway, this is behind one solution to one of the problems about purification. It does not solve the one source problem. For this, you need other tools.

01:21:20 — Zero knowledge proofs and their significance

Ryan:

[01:21:20] I saw that you had done some work in zero-knowledge proofs. I was wondering if you could explain what a zero-knowledge proof is. Maybe we can talk about its significance.

Avi:

[01:21:30] Sure, zero-knowledge proof. It’s pretty amazing that it became a household word. I mean, it was in the domain of theorists for a long time. The original definition of zero-knowledge proof came in a seminar paper by Goldwasser, Micali, and Rakoff, which also contains the definition of interactive proof. So even before zero-knowledge, they conceived of the notion that proofs don’t have to be written down like in mathematical papers.

[01:22:06] People can have a discussion, a randomized discussion, in which I try to convince you of something I want to prove to you. Something like, I know the proof of the Riemann Hypothesis, or I can solve this Sudoku puzzle, and we can do it interactively with the random. We are randomized. In particular, you, the verifier of my claim, are allowed to be randomized. So it’s like in randomized algorithms. But now there’s a prover.

[01:22:37] It’s not just an algorithm; it’s a prover, like in NP. Someone who knows the solution. But the convincing has this interactive form. This model was suggested in this paper and also in a paper of Babai, parallel at the same time from different motivations. Basically, it offers a generalization of the notion of NP. NP is just when I send you a message, and it convinces you; you verify it and it convinces you.

[01:23:09] Now, we allow an interaction and we allow a small chance because you are randomized. We allow a small chance that I will convince you of a false claim. But this error can be reduced, like in probabilistic algorithms. You want one in a billion? You can get one in a billion, whatever. Anyway, so there’s a notion of an interactive proof. In the Goldwasser-Micali-Rothblum paper, they suggested another notion of interactive proof, which is more restricted.

[01:23:46] You want to prove something, but the zero-knowledge proof means that you want the verifier to learn nothing, absolutely nothing about the proof except that it’s true. Now, this sounds really totally ridiculous. I mean, if you think about the last time you convinced somebody to change their mind about anything without providing them any knowledge, any new things they didn’t know, say, what are you talking about?

[01:24:17] Such proofs don’t exist for anything. There’s no zero-knowledge proof for anything. They were not thinking about convincing someone of a political opinion; they were thinking about cryptography. Of course, they’ve created the foundation of cryptography in many other papers, but they are imagining cryptographic protocols in which you have secrets. You use these secrets in your computation, and the protocol tells you to do things that, if you don’t, then you’re violating the protocol.

[01:25:00] So the others don’t want you to cheat. They want to make sure you perform the right operations on your secrets. For example, you are supposed to pick a public key by multiplying two prime numbers. If you multiply three or something else, then you are violating the protocol, and maybe security is not guaranteed. So I would like to convince you that the number I give you is actually a product of two primes.

[01:25:30] I certainly don’t want to give you the two primes. What I really want to convince you of is that I computed this number by multiplying two primes. You learn from this interaction absolutely nothing except that you are convinced with very high probability that I did multiply two primes and not any other number, and I didn’t do anything else that I shouldn’t. You can think about lots of other cryptographic protocols where people are doing things like multiparty computation, very complex things.

[01:26:06] They are confused with their secrets and don’t want to reveal them, whereas the others want to make sure that they did what they should. There are many applications for this idea. The only problem is that it sounds ridiculous because it seems impossible in mathematical terms. If someone came here and said that they proved P versus NP, I would want to see the proof and then tell me, okay, I’ll prove it to you in zero-knowledge proof.

[01:26:40] How can anybody convince me that they have a proof of P versus NP? I come out congratulating them and amazed by them, and nevertheless, I know absolutely nothing about the way they proved it. I just know that they did this. So it really sounds ridiculous. And yeah, certainly one of my favorite papers, maybe my favorite, is the zero-knowledge proof paper, which came a year later, co-authored with Omer Goldreich and Silvio Micali.

[01:27:17] Well, we showed that it’s not only not ridiculous, it’s universal. Namely, anything that has a mathematical proof also has a zero-knowledge interactive proof. Anything like P versus NP or that I multiply two primes, or anything that you can prove revealing your secret, you can prove without revealing your secret and be convinced beyond any reasonable doubt. So that’s possible.

Ryan:

[01:27:46] What’s the intuition behind that? Let’s say I have a proof for P versus NP, and we want to convert it to a zero-knowledge proof. How does that work?

Avi:

[01:27:55] How does this work? Well, first of all, it assumes cryptography. We assume we have some one-way functions. We assume that some problem, like factoring integers or discrete logarithms, or any number of one-way functions believed to be one-way, exists. One-way functions, if people don’t know, are functions that are easy to compute in one direction but hard to invert. For example, multiplying numbers is easy. Finding the factors of a number, which is the inverse problem, is believed to be hard.

[01:28:32] Of course, we can never know any hard problems. That’s the pre-first entry question. We don’t know, but we believe many problems are hard, and this belief is actually shared by the whole world because the entire world is using cryptographic systems that rest on this assumption. All electronic commerce assumes this. So, we assume we have one-way functions. One-way functions allow you to create basically garbled message commitments.

[01:29:06] I have a number in my head. I don’t want to tell you what the number is. It’s a number between 1 and 100. I can write down another number that looks like a random number to you. On the one hand, you have no idea. I claim this encodes my secret, okay? This commitment scheme, which is built very simply for one-way functions, guarantees two properties. A) You cannot tell what my secret is, even though you can see this other number I gave you.

[01:29:48] And on the other hand, I cannot change my mind about my secret. It really commits me to it. So I can later provide you with a certificate that I was thinking about 17, and I can only do it for 17. I cannot do it for any other number. In fact, a very simple example is really using factoring in some sense the product of two primes. This number commits to its factors.

[01:30:27] It uniquely defines its factors, right? The only problem is this is not exactly random products of primes. Yeah, but you can do it, so you can do this. Okay? So now that we have to take commitments, that are possible, that’s very important. I’m describing very high level; I’m hiding lots of things. Even the definition of zero-knowledge proof, the formal definition, is quite intricate. The intuition is obvious, and I said it.

[01:30:58] But actually, formally defining it is non-trivial. At a high level, I’ll give you some idea about how a zero-knowledge proof looks. Now we have to think about what theorems I am proving to you. It was very important for us to figure out what problem to think about. Even though today you can do it for others, we were thinking about theorems of the type where, in the graph, we can color it in three colors.

[01:31:27] It’s also a formal mathematical statement; either it’s doable or not, right? We simply focus on proving zero-knowledge claims of this type. The graph we both know, I claim I can color it with three colors. You don’t believe me. You want to be convinced of this fact, and you don’t want me to cheat you. You don’t want me to be able to provide an interactive proof. Moreover, I want it to be zero-knowledge.

[01:31:58] I don’t want you to have the slightest idea of what my coloring is or what anything you didn’t know before is. Okay, so roughly the way it works. The honor is proof for this particular set of claims that are certainly not all claims. They are very structured claims. We look like the following. We iteratively repeat the following procedure. I put commitments for the coloring on every vertex. I tell you, you know, I don’t tell you it’s green, red, or blue, but I put a commitment to one of them.

[01:32:38] On each vertex, you will choose at random an edge of the graph and ask me to open these two envelopes to decommit. You will check that the colors you see are in this set. They are not gray or orange; they are either red, green, or blue, and they are different. This should give you some slight advantage or support to the belief that maybe I’m not cheating you.

[01:33:13] Of course, it’s a very limited one because I can cheat you in some corner. We have to establish two things. We will repeat this again and again. We have to establish that it’s a proof, an interactive proof that I cannot cheat you. And we have to argue that zero-knowledge part. Okay, so let’s do one at a time. The correctness that I cannot fool you is very simple, is the following.

[01:33:49] If the graph is not three-colorable, whatever I commit for there is one place which is an arrow. Either it has the wrong color and not allowed color, or two colors are equal. You have some non-trivial chance of catching this with your random guess. It’s one over the number of edges. It’s not so small. It’s a graph with a hundred edges. It’s one in a hundred if thousands.

[01:34:16] If we repeat it, not a thousand, but 10,000 times, the probability that you don’t catch me in any of them drops exponentially to zero. So, of course, this establishes that I cannot fool you. The zero-knowledge proof is a more serious problem because if I keep using the same coloring, you will ask me about this edge and this edge, and then eventually you’ll know the coloring of all colors.

[01:34:50] Here’s something that’s really special to the coloring that is being used. If I have one coloring of the graph, I really have six because I can permute the names; I can replace red and green, and it’s another valid coloring. So what I do in each one of these iterations is not use the same coloring, but use a random one of these possible six. They are all legal, and I just use one of these six.

[01:35:21] What’s the advantage of this? When you open a pair of vertices under this distribution, one of the six, what you will see, if I do have a coloring—and that’s the only thing I want to stress—is that we only have to establish zero-knowledge proof if I really can prove the theorem. So if I do know the solution, the proof, I want to protect my knowledge. What happens when I reveal to you two of these colors on the adjacent vertices of the graph?

[01:35:52] It will be true in this distribution. When you are given two vertices, it will simply be two different random colors. This is what you get by all permutations in this experimentation. What did you learn from this? Nothing. You could have picked two random different colors. You didn’t need me for that. So you didn’t learn anything. In each iteration, you learn nothing. This is the idea of the proof.

[01:36:20] Well, this is the end of the proof that I can show there are zero-knowledge proofs for all graph coloring, three-coloring problems. What about all the Riemann Hypotheses and the P versus NP problem and factoring and all these other things? I want to claim to you and prove with zero-knowledge. Here it’s very simple. You use NP-completeness. This is an NP-complete problem. Anything that can be proved is really something in NP, right?

[01:36:52] This is the definition. So you reduce it to three-coloring, proving it in zero-knowledge. The main point about reductions in NP is that they don’t only convert the yes/no answer in a consistent way—if this is solvable, then this is solvable. But actually, if you have a witness, if you have a proof for, I don’t know, whatever, it will become a legal three-coloring of the graph you generate. So if you have a proof, you can also have the three-coloring.

[01:37:26] It’s very important that the reduction also provides translation not just between the instances, but also between the proofs. NP-completeness theory gives you for free that if you solve the problem for Hamiltonian cycle instances, you solve it for any. You can prove anything in zero-knowledge proof.

Ryan:

[01:37:47] So with the zero-knowledge proofs, you can never be 100%.

Avi:

[01:37:52] No.

Ryan:

[01:37:53] Okay, but practically, I mean exponentially approaching.

Avi:

[01:37:58] You can make the completeness 100%. Namely, if I do have a proof, you will be. Yeah, there’ll be no error in your proof. But there is a slight possibility that the graph is not three-colorable, and you will not catch me. You can make this exponentially small. You cannot make it zero.

Ryan:

[01:38:18] When I think of cryptography and one-way functions, there’s this idea of quantum computing that’s kind of changed complexity theory.

Avi:

[01:38:27] bit and big time.

01:38:30 — Quantum computation and why it matters

Ryan:

[01:38:30] And I wanted to ask your thoughts on how quantum computing, or that model of computation, is changing complexity theory. What are the big takeaways?

Avi:

[01:38:42] Okay, let me say what it is. First of all, quantum mechanics is a theory of nature that seems to be universally accepted. So, like with randomness, we can ask, why not enhance computers with this physical knowledge? We allow our computers to operate in the ways that quantum mechanics dictates, namely, manipulate bits in superposition using unitary operations, whatever this means.

[01:39:16] But according to the rules of quantum mechanics, really strange things happen. I’m sure many people know about the interference patterns. Basically, you are working with probability theory. With negative numbers, events cannot aggregate; they can only cancel each other. Okay, so it’s a model of computation.

[01:39:51] It’s a generalization of Turing machines. In fact, it’s a generalization of randomized Turing machines. It’s very easy to see that a quantum computer is at least as strong as a probabilistic computer. How do you see it? You just measure the quantum bit before you start. You measure them. This is what I mentioned about the photons in the beginning. If you have some basic superposition on a quantum bit, if you measure it, you get a random bit.

[01:40:23] So because of that, quantum computers are at least as strong as probabilistic computers. Okay, so it’s a model of computation. It was suggested in the 1980s. Richard Feynman, along with others, suggested letting algorithms use these quantum mechanical operations, mainly originally for simulating quantum systems rather than building bigger parallel systems, like we simulate other things, such as turbulence. I don’t know what the power of quantum algorithms is, again, let’s say running into polynomial time or efficient ones.

[01:41:08] Can they do more than efficient classical ones, deterministic or probabilistic? It wasn’t clear for a while. There were a few examples that were very stylized but were not for concrete natural problems we care about. Then in 1994, Peter Shor sort of created an earthquake or avalanche. He found quantum algorithms that are efficient, that factor integers and also compute discrete logarithms, the two most basic underpinnings of all security systems that exist.

[01:41:48] And this sets the world on fire. Right. So lots of people try to do a lot of things. Of course, people want to have these algorithms they implement; maybe they want to break other people’s cryptosystems. Since then, billions were invested by companies, by governments, by lots of people trying to build the technological infrastructure. This is extremely complicated. Holding bits in superposition is extremely complicated.

[01:42:25] These things are called noise. When people built classical computers like von Neumann, noise was one of the serious problems because the bits were really in vacuum tubes, and they had to contend with it. In fact, it built a nice theory of classical computers that have errors. They have to cope with errors; some of their components can be faulty. But today’s hardware has no errors to speak of.

[01:43:02] We don’t, there are error-correcting mechanisms, but we don’t need them for classical computing. Intel chips don’t have error correction in them. Hardware is very reliable. When you move to quantum, there is this decoherence noise in quantum mechanics. Everything depends on everything. The world can influence the computation in your laptop, and protecting from this is very hard.

[01:43:33] There are quantum error correcting codes, and that’s part of the solution. That’s only one of the problems. Just holding bits in superposition is hard, and there are many hard technological issues, and there’s progress on that. So this is one line of huge investment. The other line comes from cryptography because, of course, everybody should be worried. Forget quantum computers.

[01:43:59] If tomorrow, I mean really tomorrow, somebody finds even a classical factorization algorithm in polynomial time, I think there’ll be chaos in the world because nobody can do any transactions since most security systems still rely on factoring. Of course, we want to change this. We want to change the underlying assumptions of security. The whole revolution in cryptography was that we risk cryptography on computationally hard assumptions.

[01:44:38] And with this, we build all the wonderful public key systems and all the magical things you can do by assuming that players are computationally limited. But now, if the adversary is a quantum computer, suddenly they can break this assumption. You want to find other mathematical problems, computational problems that are somehow hard even for quantum computers. So that’s a whole field, this change in complexity theory and cryptography.

[01:45:19] There is an army of people who are just trying to invent problems. I maybe should stress one-way functions are easy to find. Most problems, most processes in nature are not easily reversible. You make an omelet from an egg. Reversing this doesn’t take the same amount of time; that’s the usual example of a physical one-way function. But trapdoor functions, like factoring problems, are what you can use to build public key systems.

[01:45:55] And that’s the most really basic thing for electronic commerce. We don’t know many. So we know this. I mentioned factoring and discrete logarithm. In the 90s, Itay and Roark created another type of trapdoor function from problems on lattices in high dimensions. I will not describe it, but it’s another problem. Then people refined and got a few more of similar nature. It turned out when Peter Shor discovered this, people immediately tried to solve other problems with quantum computers.

[01:46:31] In fact, even today we cannot solve too many other problems with quantum computers. These are special. The special thing about them is that somehow you can reduce them to finding periods in a signal. Periods are like fluorescence form. A Fourier transform turns out to be in exponential space. But you can somehow perform a Fourier transform with quantum computers using interference with lattice problems and problems related to it.

[01:47:02] Some call it learning with errors. Similar to this, to this day, nobody has found an efficient quantum algorithm for them. So there’s not even a theory. Forget building quantum computers; we don’t know how to solve them. The world is not just complexity theory. The whole physical world of security systems and also governments like the NSA is supporting or asking the world to produce assumptions that may be resilient to quantum attacks.

[01:47:44] And so this is a huge change. There is a major change in the interaction between computer scientists and physicists, which grew tremendously since its discovery. It’s rich in many ways that are not the influence of algorithmic thinking on physical theories, including today’s theories in quantum gravity, black holes, and so on. The impact is immense. We have new sources of problems, new models, and new complexity classes, et cetera, et cetera.

[01:48:19] And there’s a fantastic interaction. It’s also enormous complexity theory in that it turns out you can discover quantum algorithms and maybe then dequantize them in special cases. Like I mentioned factoring before, de-randomizing. Sometimes it’s a way, it’s a road to discover new algorithms. There are connections between problems. It’s extremely rich. But one of the most amazing consequences is that we, being what we are, make up models and study them.

[01:49:05] And the interactive proofs I mentioned before, it turns out that even before quantum, they were generalized to interactive proofs not with one prover, but with many provers, which seems weird but turns out to be very important in itself because it led to this PCP theorem. Then people said, okay, let’s allow quantum verifiers, quantum provers, and see what they can do. So one amazing result, which is about five years ago, is the acronym MIP* = RE.

[01:49:40] Of course, we have acronyms for all these complexity classes: MIP*, RE. You may have seen them, maybe you didn’t, and they look weird, but I’ll tell you what they say. It says that there is a really weird proof system with quantum provers trying to convince efficient verifiers. And it turns out, you know, you ask for which problems can they do it? Forget their own knowledge, just convince.

[01:50:12] And it turns out they can do it for the halting problem, for problems that are not computable. But things that are not computable are verifiable by efficient verifiers if the provers are quantum and they are entangled. It’s a weird proof system, very weird, but it does something that looks totally ridiculous. Things that are uncomputable by any classifiable component are verifiable in this interactive probabilistic sense by efficient verifiers.

[01:50:51] So what I like to say in talking about this is that it seems the best hypothetical reaction of anybody who hears this is, “Okay, you complexity theorists, you play in your sandbox and you build all these sandcastles and make up all these models that have nothing to do with anything just because you can.” Once you get weird enough, you get weird enough consequences.

[01:51:20] So one message which I think is very powerful is that this result has absolutely fundamental impact on mathematics and physics. It turns out that it implies, and this was already done in the initial paper, resolution of well-known conjectures in mathematics and physics. It turns out that, weird as it is, it’s a new mathematical technique to solve problems that nobody had any idea about. Famous problems, important problems that fields were dedicated to, are resolved by this result, by the techniques of this result.

[01:52:05] So you ask about the impact of quantum computing. The impact spans many generations with different motivations and developments, but all of them following the methodology of complexity theory, understanding the power of computational models, proof systems, and so on, has magically led to such a consequence. We are just at the beginning of this, right? This type of proof technique is now being explored and used, and there are more results using this type of technique to address long-standing problems.

[01:52:44] Pretty amazing.

Ryan:

[01:52:45] That’s incredible. Yeah, I mean, in complexity theory there are these Venn diagrams of all possible problems. The implicit assumption in this picture is that these are decidable problems.

Avi:

[01:53:02] Yeah, yeah.

Ryan:

[01:53:03] In some of these diagrams, there’s a little in the corner. Undecidable.

Avi:

[01:53:09] Yes, unreachable.

Ryan:

[01:53:10] Yeah, right. And it’s just, I can’t, I don’t even understand how you could verify an undecidable problem.

Avi:

[01:53:19] It’s a 200-page paper. It builds on ten years of understanding, which involves a lot of development in quantum algorithms and quantum proof systems, but also relies on techniques from classical proof systems related to coding theory and various algebraic methods used to prove, for example, the PCP theorem, as I mentioned. It’s a huge body of work, and on top of this, they wrote this 200-page paper in which they had to develop many more tools.

[01:53:56] And yeah, there you have it.

Ryan:

[01:53:58] Wow. I mean, when someone produces 200 pages of such complicated work that makes such an outrageous claim, how does that get verified by people?

Avi:

[01:54:12] I mean, it’s a very good question. This tool is for any mathematical result that is complicated. Of course, proofs of P versus NP and the Riemann hypothesis are generated very frequently, but some of them were generated by very serious people and took a long time to refute in these two cases. Others, like the Perelman proof of the Poincaré conjecture, one of the Clay Millennium Prize problems, was another one like that.

[01:54:53] And this took several years to verify and fix small bugs in. Books expositing this proof were written, and eventually, this particular paper is for some reason still in the referring stage of the Annals of Mathematics now, probably almost six years; it will be done. It’s robust in the sense that already new papers were written with this type of tools, in fact giving somewhat alternative proofs of the same statements and stronger statements in order to resolve other mathematical conjectures.

[01:55:29] But yeah, it’s a serious thing. In fact, the original paper had a bug, like the original paper with Fermat by Wiles. Fermat’s Last Theorem also had a bug, and it was fixed. This was very early on, just when it came out, and it was fixed very quickly. So any mathematical claim—your question is very important. In general, mathematics is a mathematical truth that is a social construct.

[01:56:04] That’s what mathematicians believe. Maybe we are moving into a world in which we have lean proofs. We have formal ways of writing proofs, and you can maybe formally verify them. But anyway, I think that this community is quite certain about.

01:56:24 — Math vs computer science

Ryan:

[01:56:24] Yeah, in complexity theory, it’s very mathematical in nature when I read the papers, but obviously it has implications in computer science. What’s the relationship between mathematics and computer science to you?

Avi:

[01:56:39] Well, the theory of computer science, my field, complexity theory and algorithms, is really lucky to have these two parents. It lives in both spaces. It is a mathematical field in the sense that most of what we generate are theorems and proofs in the mathematical sense. On the other hand, it relates to computation because the kind of questions we ask ourselves, the problems we are trying to solve, are partly motivated by trying to understand computation.

[01:57:19] So you can think of theoretical computer science very much like you can think about analysis, algebra, and topology. There are notions in algebra where you are trying to understand equations or systems of equations. In analysis, you are trying to understand, very generally speaking, inequalities and continuous spaces. Right? So there are these notions you are trying to understand. Computation is one of these notions where you are trying to understand what you can produce by an evolution of simple local steps through a given environment.

[01:57:59] The input can be a sequence of bits, or it can be the DNA of a person, from which you produce proteins using computation, or you produce a new baby using the evolution of a fertilized egg, or the weather. Computation is sort of everywhere, and we are trying to understand it in ways that are different as we care about resources—how much resources are exerted in any such computation. We are trying to model it, understand the algorithms, and let the underlying natural phenomena inform our understanding.

[01:58:48] So we live in both spaces. We have a lot of input of problems and models from industry technology systems and so on. And we have all the mathematics, so we have the rigorous side coming from and the aesthetic side, I would say, like creating models of proof systems because they are nice, because they are interesting, not because they are implementable necessarily.

[01:59:21] So this living in these two spaces is extremely beneficial to this theory of computation.

Ryan:

[01:59:32] You mentioned earlier that some people might have the perspective of complexity theory that it’s kind of solving problems for problem’s sake. I read in another interview you did that you were never motivated by application. My natural thought was, what motivates you to solve all these really tricky, complicated problems?

Avi:

[01:59:54] Okay, so let me say one thing about the beginning of your question. It says about solving problems. I think that what we do more, much more, is modeling things. Modeling things then gives rise to problems that you want to understand. So I think the modeling part in the theory of computation, modeling various forms of computation, is a large part of theoretical computer science and manifests itself in particular in cryptography.

[02:00:23] Well, there are models of adversaries in distributed computation. There are many. I think modeling, just asking questions, and creating definitions is a very central part of the field. It happens more than in mathematics, maybe because they are more ancient and definitions happened earlier. But I want to stress this; it’s a very important thing. For example, we discussed randomness, and our definition of randomness is totally different, which gives rise to very surprising and interesting practical theorems.

[02:01:06] Okay, now that we said that, what motivates me or what motivates other people? Of course, everybody has their own motivation. I am a product of computer science education. All my undergrad, grad school, postdoc, and everything were in computer science. I certainly regret not taking many more math courses because then I wouldn’t have to learn these later in life. But that aside, I think that the focus on computation, which is what computer science education gives you, is different. Different systems and different issues are, of course, algorithms in databases and algorithms in programming languages, and in fact, famous algorithms in other fields.

[02:02:02] There are models and algorithms. Computation is something that I’m interested in intrinsically. Over the years, I’ve realized that computation, you know, the field expanded to interact with all the sciences, and that computation happened everywhere in various forms, in physics, biology, and so on. It makes it a fundamental object of study. For me, given the way I was brought up, I guess it is particularly fascinating in itself.

[02:02:34] I know from experience that lots of the proofs we develop, not many, but probably with much higher proportions than in other areas of math, are impactful in the world, in real systems, in security systems, in blockchains, in the real world. But this particular side of seeing exactly how you translate your algorithm into a system or your zero-knowledge proof, I predicted when we came up with the zero-knowledge proof that this would never be implemented, because the protocol that I described more or less to you is very costly.

[02:03:20] I mean, you want to prove that I’m producing a product of two primes, and I convert it to a map, and then I do all this complicated procedure many times. It doesn’t seem that anybody would ever use such a thing. But yeah, I was wrong. I didn’t realize how motivated people can be and how important applications of zero-knowledge proofs can be in the real world, not in the theory of protocol design. They did simplify, maybe using more assumptions, different assumptions.

[02:03:54] There’s a recent breakthrough of a postdoc here, Ilango, you may have heard, because it gets some publicity in Quanta and so on, where he introduced into the assumptions of cryptography not just hard computational problems, but also things of the nature of Gödel’s theorem, that something is unprovable in some mathematical proof system like formal frameworks, whatever mathematicians use. Somehow using that, they can get zero-knowledge that’s non-interactive.

[02:04:28] You can get. Yeah, it’s an amazing thing. It gets richer. What I want to say is that, again from experience, I have 45 years of experience in this field, which has been fascinating. I mentioned to you in the email that there are extra benefits to this field. I think the community is amazing. Not just that it has so many brilliant minds, but young brilliant minds are entering the field all the time. It’s also very lively, interactive, and collaborative.

[02:05:14] Yeah, many of my best friends are also colleagues, so it’s fantastic. The theoretical understanding of many things that we may ask ourselves for aesthetic reasons that are mathematical, we generalize something for generalization’s sake, which may not have a counterpart in industry. In fact, Peter Shor‘s work on quantum algorithms raises the question, why do that? There are no quantum computers.

[02:05:49] Maybe let’s wait until somebody builds one and we understand why. So, no, the answer is no. You should try to understand whatever is natural for you to understand. This field has already, maybe it was not clear 40 years ago, but it is clear now that lots of theoretical understanding is not just a product of mathematical results, but because it is about computation, it is meaningful.

[02:06:25] I don’t know often enough in the real world. There are many other examples besides cryptography and quantum computing, including theory. The field has revolutionized coding theory mainly because of the PCP theorem tools needed for it. There are lots of examples, and this eventually went into systems, like memory schemes. Look, I’m interested in P versus NP.

[02:06:57] This is a question about impossibility, right? This is a problem. We still haven’t gotten closer to this, at least in the years I’ve been at it. We are not much closer to showing hardness for anything. We don’t know that multiplication is harder than addition. It’s a basic question. So this kind of question by itself is certainly not practical. I mean, it’s not clear. Well, it is a little clear because if you find R functions, maybe you can substantiate cryptography on a theorem rather than on an assumption.

[02:07:35] But anyway, there are questions in the field that don’t directly relate and will not directly relate to implementations of any system, but they are all part of trying to understand what efficient computation can do and what it cannot. Or when can you minimize resources of some type and when you cannot. And how do different problems relate to each other. This basic methodology of the field has created some wonderful edifice.

[02:08:10] And I would say that we are still in the embryonic stage of understanding computation.

02:08:16 — Major breakthroughs and his experience

Ryan:

[02:08:16] You mentioned a few times in this conversation that some people made a major discovery, which kind of broke some assumptions that were maybe 50 years old. Researchers make these advances, and I know in your career you’ve made a few as well. The time span between breakthroughs is sometimes decades.

Avi:

[02:08:38] a major discovery sometimes because a lot of

Ryan:

[02:08:41] people’s careers, like maybe if you’re just an engineer, you got your shipping projects every year. But in your case, let’s say you discovered P versus NP or something like that.

Avi:

[02:08:56] Yeah.

Ryan:

[02:08:56] How do you feel when you make those discoveries, those sparse discoveries in your career?

Avi:

[02:09:02] It’s great. Yeah, it happens. The big ones happen. The biggest, rarely I think of science—I mean realistically, science and math is a community of ants working. Most progress comes from these conferences, theory conferences, and folks, and then we have all sorts of satellite conferences. There are hundreds of papers in them, satellite, I mean, focused on concrete areas and learning theory equipped.

[02:09:33] So I know online algorithms. There are many, many thousand researchers working in this, I don’t know, maybe. And they are producing many papers a year. Most of my papers are, you know, we make a little progress in understanding something. Usually, it’s not a big problem; we discover a variant of a technique or we can strengthen a bound on some. I find this essential.

[02:10:05] I think this is true in science in general. Bigger understandings come more early, and they send a shockwave to everybody who learns them and uses them. But of course, those bigger understandings don’t have to be a resolution of 50 years of papers. It can be something that, when Barrington discovered his algorithm for counting that I mentioned, he was trying to actually prove that it’s impossible.

[02:10:39] And it was not like somebody asked this question; the answer was obvious. It’s impossible. Right? So when you have an insight of this type, it’s amazing. Whenever you realize something really new for you and for the community, it’s a phenomenal result. It’s rare. I tell all my students and postdocs that, yeah, it’s not, you know, you don’t work for the million dollar. The million dollar is nothing, of course, for some of these problems.

[02:11:12] But yeah, you work because you enjoy the practice of it. You enjoy thinking about the problems. In fact, most days in the life of a mathematician, unlike a systems person, you go in the morning, you come back in the evening, and you fail to do what you wanted to do. You just couldn’t. You just thought more. And I think that it’s really important. It’s not for everybody.

[02:11:45] Clearly, it’s not satisfying if this happens every day and every week. It’s not for everybody, but it’s for the people for whom this activity of trying to think, of throwing ideas, of failing, but learning from the failures, is meaningful. People don’t realize that often you learn. The fact that you failed is not just a wasted day. There is something that you gain from it, maybe subconsciously, that will help you later.

[02:12:17] So unless you enjoy this activity, maybe this field is not for you. Yeah, and then you see something, and even if it’s small, sometimes it’s very satisfactory.

02:12:31 — Advice for his younger self

Ryan:

[02:12:31] Last question for you. I think people in the field might be curious because you have so much experience. If you could go back to the beginning of your career when you just started becoming a researcher, is there any advice that you’d give yourself?

Avi:

[02:12:48] I think I was lucky enough not to think that I would change anything. I think I was lucky enough, and many people are lucky in this field because the field is so accommodating to young people. It’s completely leveled; there are no hierarchies. People would go to conferences and meet. I remember my first conference meeting Dick Harp, who was in the field and invented, you know, did all these NP-complete things.

[02:13:23] And I was a first-year graduate student, and somebody introduced me to him. He immediately asked me what I was doing, and I said I proved that some problem was NP-complete, and it was actually interesting. It amazed me. I was lucky to have phenomenal mentors as an undergrad. As a graduate student, I was lucky to have unbelievable collaborators over the years. You learn from all this.

[02:13:52] So it’s not very informative advice to tell people to be lucky. I think this environment exists. One piece of advice I give everybody is that in the early stages, they should work on things they enjoy the most. When you are discovering your talents as a researcher, one direction may be these are all problems. If you solve them, then you. But maybe those aren’t your problems; maybe it’s not your affinity to them.

[02:14:28] You have to experiment with the type of problems. It’s good to have a variety of areas and papers to read in these areas or problems to understand. It’s always good to start with a few to better understand your capabilities. Often, what you like most is what you are better at.

The conversation was a lot of fun and I’m glad a lot of his takes are refreshingly not “doomer” takes. Below are the top three takeaways I had from the conversation to save you time.

You can find the full conversation on YouTube, Spotify, Apple Podcasts. The transcript is on Substack if you prefer to skim.

Brought to you by:

WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code

Takeaways from the conversation:

1) Career advice for the AI era - His take was that software isn’t about syntax or algorithms. It’s all about conceptualizing problems and coming up with clean solutions for them. And to build that muscle takes experience. He urged that people shouldn’t stop exercising that muscle or you’ll atrophy be left behind. Use AI but also make sure you aren’t being passive in your learning.

The other major point he had was that using Claude Code isn’t that hard if you are a good engineer. The value isn’t in memorizing the details and learning all the latest AI tools. The important part is building things and solving problems that matter. He said you should just ignore Twitter for the most part and focus on what actually matters.

2) Fixing broken team incentives - The problem we discussed is when a team’s identity, mission and name all revolve around a system they own. What happens is these teams end up trying to protect the system rather than doing what is best for the company.

The example fix James gave is when he was at Dropbox, he worked on a huge migration to move off of AWS. The resulting team was named after the system they built. He went out of his way to rename the team the “Storage team” instead.

The reason this was so important is he felt that the direction of the team should be oriented around the problem they are solving for the company. Otherwise, imagine if moving back to AWS turns out to be better for the business. The team named after the existing system would have natural incentive to battle doing the right thing. He called this phenomenon “system bias”

3) Simple systems are the goal - To the untrained eye, simple systems can seem obvious but actually designing simple systems is much harder than building complex ones. And the key James mentioned is that simplicity reduces operational burden. Simple systems are easier to keep running and debug when they break.

I asked him for a concrete example and he shared how Dropbox managed the metadata for where files are actually stored. All they did was have a cluster of 1000 MySQL nodes that stored the block ID and its location. Many people would say it wasn’t sophisticated but all the alternative proposals would ruin observability and simplicity of querying this data.

The idea of complexity being incentivized in larger tech companies frustrated him. To him, the goal is to solve the problem not to check off the box for complexity.

I am really excited about some of the future guests that are coming on the show. I love the interviews with the people who have a piece of computer science history in them.

Also, I’ll have some AI research folks on soon too. Many people have requested I bring them on so people can learn how to future proof their careers.

Lastly, I interviewed an award-winning complexity theory professor from MIT and asked him some Leetcode questions. His optimal solution was asymptotically faster than what Leetcode says is “optimal”

Thanks for reading,
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

0:53 - Systems work during his PhD

13:05 - Dropbox technical deep dive

21:57 - Why Dropbox migrated from AWS

36:40 - How to do massive migrations

44:31 - Simplicity vs complexity in promos

49:23 - What technical teams should be focused on

1:00:25 - Doing the right thing vs promo hypothetical

1:08:13 - Why he dipped into management sometimes

1:11:36 - Why you shouldn’t lead by example

1:23:23 - How to mentor Senior Staff+ engineers

1:27:30 - Career advice for the AI era

1:37:21 - Why he started his own company

1:46:05 - The most technically challenging work of his career

1:48:10 - How he got involved in Silicon Valley

1:52:16 - Career regrets

1:55:54 - Top technical book recommendation

1:56:36 - Younger self & permanent underclass advice

Transcript

0:53 — Systems work during his PhD

Ryan:

[0:53] Here’s the full episode. First off, your PhD thesis was huge. It’s 156 pages. I didn’t know theses were that long; I thought papers were maybe 10 pages or something like that. It’s its own book.

James:

[1:12] It’s very similar to Spanner ultimately. What’s funny is I did that work; it came out a little bit before Spanner, so I got a citation in the Spanner paper. But then Spanner came out, and everyone kind of forgot about my paper.

Ryan:

[1:27] You know, about the research. If you could kind of describe the problem that Granola was solving and how it solved it, those types of things. We can talk about that.

James:

[1:38] Yeah, absolutely. So my interest in my career has been two parallel threads. One has been abstractions in general, the idea about how to build simple models for complex problems. I find that a very difficult and intellectually stimulating design exercise—how to design APIs, basically. The other has been large-scale transactional systems. I’m a big fan of transactions. A transaction is doing a bunch of things at once, and I think transactions are just one of the most incredible abstractions we’ve invented because they allow us to manage probably the most difficult problem in computer science, which is concurrency.

[2:22] Granola was an algorithm for how to do distributed transaction coordination, particularly for transactions that I described at the time as one-shot transactions. I’m not actually sure if that was a standard term at the time or whether I made it up. A one-shot transaction means you send some code to the server and say, “Run this function, all these reads, all these writes on two, three, whatever nodes at once.”

[2:55] And how to make sure this commits atomically across multiple shards in a distributed system. That was my PhD thesis, and my master’s thesis was on Byzantine fault tolerance, a consensus protocol in the presence of malicious nodes. It focused on how to achieve agreement in state across multiple parties when there are malicious entities. I guess the most well-known work I did as part of grad school was on a paper called View Stamp Replication Revisited.

[3:27] And that was a paper on redefining a protocol called View Stamp Replication Revisited, which predated Paxos. It’s a very similar algorithm. Paxos, Raft, and View Stamp Replication are all based on virtual synchrony; they’re all basically the same thing. That was a paper I wrote at the time, which ended up being influential to a few really great companies like Tiger Beetle.

Ryan:

[3:53] You mentioned transactions, and I saw in the paper there’s this idea of an independent transaction. If I’m understanding correctly, a lot of the efficiency in a distributed system is lost by needing to get consensus and to vote for consensus. In your research, you found a way to avoid needing to vote to reach consensus. How did you do that?

James:

[4:20] Yes, I mean a lot of people think about performance maybe from the wrong angle because I think of performance as a factor of just this raw horsepower, how fast are your disks, how fast is your network, etc. No one has particularly faster disks or memory than anybody else. What really matters to performance in a large-scale system is eliminating points of coordination. So it’s how to allow systems to progress without having contention between parties.

[4:53] Where you’re basically reducing parallel throughput to serial throughput across large numbers of transactions. In Granola, we had this idea called an independent transaction. Again, that was the terminology at the time where there were two entities basically processing pure functions. They have independent state and they’ll both come to the same conclusion as a result.

[5:16] And all they need to do then is serialize those transactions. They have to decide if it was to happen atomically across multiple nodes, what timestamp should it get. Granola, the paper, was mostly about how to exchange these timestamps very efficiently. So, how to have multiple parties each propose a timestamp and then choose the maximum of these, basically so you could safely serialize the transaction.

[5:42] And there’s a lot more complexity that goes into this. But really the focus there was about how to maximize throughput in a large distributed system without resulting in the alternative, which is two-phase commit and two-phase locking. Two-phase commit with two-phase locking is basically the standard approach for when you want to have multiple nodes agreeing on the same thing, where they both agree to lock their state, not process any other data, and then commit a transaction.

[6:09] Two-phase commit can be quite low performance because you’re blocking basically the systems for the duration of the transaction. It can be high risk too because you are taking a dependency on another node; you’re basically blocked waiting for another node to return. And so that was what Granola was. Now, it’s funny you asked about Granola because I forgot about it.

[6:33] That’s so far in my history, I kind of forgot about that work. I even forgot the phrase “independent transactions,” to be honest. It’s great to hear you bring it up again. But I guess all this stuff does feed into all the work you do later on in interesting ways.

Ryan:

[6:48] When I think about distributed systems, I think about you’re at a big company and you got a bunch of machines. But when you were at MIT building this thing, how did you build and test it? Were there spare machines that the college had, or was this in the cloud?

James:

[7:05] Yeah, this was a long time ago. Now I’m showing my age. This is pretty early in the days of Amazon Web Services. We had a rack of servers in our office that we could use. I was fortunate enough to be at MIT where we could afford a rack. There was also a service called Planet Lab, which was a big communal set of nodes that academics could use to run tests. But Planet Lab was a communal system.

[7:34] And so it was continually having problems. It’s just a free for all. For the longest time, I would just sleep next to my desk and wake up whenever. Planet Lab was free. Because you need to run some benchmarks these days, you just spin something up on Amazon Web Services. But then I would sleep next to my desk, and I would wake up in the middle of the night or at random times and check Planet Lab status and then kick off a benchmarking job.

[7:58] Now this was right at the cusp of when, in some respects, Google ruined systems research. I say that with affection and respect to Google because before then, most of the research was coming out of academia. A lot of distributed systems research was done on smaller scales, and a lot of the value proposition was the ideas. So, like, hey, I wrote a paper, here’s an interesting new idea, and yeah, it’s all in theory.

[8:28] And so the value that came out of it was the idea. Around this time, you started seeing papers out of Google, Amazon, and these various companies where it wasn’t necessarily a paper about an idea; it was a paper about a system. Hey, I built this giant system, it has a whole bunch of features, some of which are interesting, some of which are not, and by the way, it powers Gmail or whatever. That kicked off a pretty interesting transition because at that point, program committees reviewing papers started to expect to see realistic benchmarks that, frankly, grad students were not able to produce at least at the time.

[9:08] I mean, I wasn’t running Gmail on my system, and I think it did, in some respects, obscure intellectual ideas. I’m more for papers about systems. But I think there’s also value in a paper about an idea. It’s not just, “We built this big thing, and it works, and it’s up to you to figure out what’s interesting about it.” Instead, it’s, “Here’s a new thing.” There’s a paper, an old paper just called Leases, where someone invented the idea of a time-based lock.

[9:37] And it’s just a paper called Leases. That was a cool era of systems research. I think a lot of it now has shifted towards industrial research, where people are building stuff for practical purposes. Whereas I think it’s hard for academia to compete on pragmatism. Academia is really a great place to do impractical work.

Ryan:

[9:59] When you look back on getting the PhD in academia, being enabled to completely explore an idea versus going into industry, maybe working on Spanner at Google or something like that, which path do you think would be better and why?

James:

[10:18] I think that’s a bit of a misconception a lot of folks have about PhD programs. A lot of students are high-achieving students in college, and they think that the PhD program is college. It’s like, hey, I like learning things, so I’m going to go do a PhD, but it’s not. A PhD is training to be a researcher, and for most people, they shouldn’t do that. If someone wants to be a professional software engineer, they probably shouldn’t spend their time training to be a researcher.

[10:46] But with a really important caveat, I think there’s a really interesting and challenging developmental experience you go through in the PhD program at a top university, at least. You reach a certain point in time when you have a problem that you’re facing that no one else in the world knows the answer to. You have a problem that you are the world expert on, and you can’t chat about it with folks, but you can’t ask your advisor because your advisor doesn’t know either.

[11:12] Right. So you face these difficult challenges where you can’t read a book, you can’t ask Claude. And so you’re forced to go through a quite difficult and, frankly, emotionally challenging experience of being unsure, being uncertain, and learning to think for yourself. I think that is really valuable for all engineers. I think that was really, really valuable in my career.

[11:39] I do see a lot of folks early in their career think that all knowledge comes from reading or absorbing it from someone else, or that there’s a right way to do things. But I think being in a PhD program or being faced with really demanding, open questions does train your mind to be comfortable with that discomfort. I feel a little bit lucky that I went through grad school without large language models existing because I had to deal with this uncertainty.

[12:15] There was no crutch to help me out, which I think was very valuable.

Ryan:

[12:18] So, I mean, a lot of, if you want to be a researcher, go do a PhD. If you want to build stuff, go build stuff. Since leaving academia, I’ve done a lot of work that I guess one could call innovative. It advanced the industry in certain ways and did novel stuff. But our goal was never research. Our goal was just to solve problems.

James:

[12:22] I mean, a lot of, if you want to be a researcher, go do a PhD. If you want to build stuff, go build stuff. Since leaving academia, I’ve done a lot of work that one could call innovative. It advanced the industry in certain ways and did novel stuff. But our goal was never research. Our goal was just to solve problems.

[12:47] And I think that’s the difference. In academia, your goal is to advance knowledge. In industry, your goal is to solve problems. I gravitate more towards just solving problems, and I find that a more comfortable environment within which to work.

13:05 — Dropbox technical deep dive

Ryan:

[13:05] Going to your time in industry, at Dropbox, you became the most senior engineer at the company. Looking through all the projects that you did, I had a series of just technical curiosities. I saw this idea early in your career about multi-homing, and I wasn’t familiar with that concept. What is multi-homing? What’s the problem it solves?

James:

[13:27] Yeah, I mean, multi-homing is the ability to have data in two locations, two homes. Multi-homing can be valuable for a variety of reasons. One is called primary-secondary or lazily replicated multi-homing, whereby all writes commit authoritatively in one region and get replicated to a secondary region. This is normally used for business continuity. One thing we did at Dropbox was to ensure that if the entire West Coast blew up, which hopefully wouldn’t happen, Dropbox would keep running because the data would be replicated in other regions, but with a window of vulnerability, with a window of time where there may be some data lost.

So that is, you know, that’s kind of primary-secondary replication or multi-homing. There’s something else called active-active multi-homing where there is truly an authoritative copy of the data in multiple locations. Basically, when you, for example, write to a system, you don’t externalize that write as having succeeded until it has landed in all the regions. For example, in the storage system at Dropbox, the block storage system, we truly had a multi-region replicated system where we could take down an entire region, say take down the region in Ashburn, Virginia, where everyone’s data centers are, and there’d be zero downtime for the company and the data was still safe in multiple regions.

[14:13] So that is kind of primary-secondary replication or multi-homing. There’s something else called active-active multi-homing, where there is truly an authoritative copy of the data in multiple locations. Basically, when you write to a system, you don’t externalize that write as having succeeded until it has landed in all the regions. For example, in the storage system at Dropbox, the block storage system, we truly had a multi-region replicated system where we could take down an entire region, say take down the region in Ashburn, Virginia, where everyone’s data centers are, and there’d be zero downtime for the company, and the data was still safe in multiple regions.

[14:54] I think multi-homing is something a lot of engineers aspire to work on because it seems like the right thing to do. But the reality is for most companies, it is not. Not because there’s very high cost, but because there are very high latency costs for this. The speed of light is fixed. If you have to synchronously write data across multiple regions in the United States, you’re going to have, say, 60 milliseconds in the commit path of your protocol, which for most applications is not tenable.

[15:32] So there is a lot of desire; a lot of engineering teams reach out for advice on how to adopt active-active multi-homing for their company. I would normally say don’t do it. Frankly, if US East is down, if Amazon is down that day, that’s okay. If Amazon’s down that day, your company will be down. That’s a shame, right? But by avoiding that complexity, you’re going to be able to move much faster and build a much better product.

[16:00] And so in my mind, systems are all about trade-offs and making the right ones. I would generally recommend most people do not make the trade-off to have partition tolerance or multi-region availability. Even though for a company like Dropbox, yes, that does make sense. When your job is storing data and you have several exabytes of it and hundreds of millions of customers, I think that’s when it starts to make sense.

Ryan:

[16:26] So it’s just another term for, I guess, data replication and having it available in other regions. I imagine also, I mean, the cost of storage is a concern. How many replicas would you keep for something like, let’s just say I stored something in Dropbox? It’s my document. Is that on the order of one or two, or are there multiple?

James:

[16:47] No, it’s on the order of many. Many. If you were to store a file in Dropbox, I modeled the storage to be, as we advertise, at least 12 nines of durability internally. The models look around 24 nines of durability. That means the data is secure with 99.9999% where there’s 24 nines. Right? Which means, at least according to the model, the universe will be extinct before any data is lost.

[17:20] And the way that is done is by a combination of what’s called erasure coding. Erasure coding is how you take several blocks of data and combine them together with an encoding scheme and spread them around in different locations. At that scale, at Dropbox scale, you’re taking things into consideration, like putting data in different racks, different rows of a data center because they’re on different power feeds.

[17:41] You’re taking into consideration different eras of hard drives and different manufacturers of drives because they could have correlated failure patterns and then replication across regions. So you could be looking at 27 fragments, for example. That doesn’t mean you’re storing 27 times the data. It’s kind of encoded in many regions. But the replication schemes get really quite sophisticated at that point.

[18:06] And we actually had our own custom encoding matrix we had developed. It’s called a Vandermonde matrix, where you take a bunch of data and combine it together to produce outputs. We would plug in some variables like how much do disks cost? How much does network bandwidth cost? Because there’s a trade-off. You can either store, if you don’t want to lose your data, more copies on more disks, or you can store fewer copies.

[18:33] Anytime a disk fails, you re-replicate it really fast. Re-replicating really fast costs disk cross network bandwidth. These kinds of variables go into this equation. It ends up being an extremely complex field of endeavor, but it’s kind of abstracted away into a part of the system that doesn’t leak into anywhere else.

Ryan:

[18:53] So with erasure encoding, my document in Dropbox is fragmented into a bunch of different chunks of data and loaded potentially from many different machines.

James:

[19:03] Yes, absolutely.

Ryan:

[19:05] I mean, the first thought I have is now I might be waiting there, and one of the 27 machines is slow, and I can’t look at the whole document. So how do you prevent against that?

James:

[19:17] It’s actually faster than not replicated. Because if you imagine, I’ll pick a simplified example. Imagine this is not the encoding scheme Dropbox uses, but to reconstruct a file, you have to read six out of nine fragments. So if you read six out of nine fragments, you can just ask all nine and reconstruct and return the data as soon as you’ve heard from the first six. It’s actually faster.

[19:46] And you can construct these encoding matrices such that it’s actually faster to have a ratio-coded data than not.

Ryan:

[19:53] I see. Okay. So you can oversubscribe, over-request, and then you complete on a portion of them being received.

James:

[20:01] Yes. Now in practice, it was a bit more complex than that. We’d often have a copy and a single disk to provide fast access. We’d often try to make sure that you could serve your data out of a region close to your home region. So we’d make sure that your data was mostly served with low latency. But if that region had failed, you could reconstruct it from the remaining regions. So there’s a lot of talk about building your own infrastructure.

[20:28] And you can save money by moving off the cloud. Almost definitely you can’t unless you have very small requirements, very fixed requirements, or very heavy investment. Because if you want to compete with Amazon, if you want to build a more efficient storage system than Amazon, you have to have a supply chain team that’s working with Western Digital and Seagate, constantly negotiating on prices of disks and buying shipments at certain times, along with capacity teams and data center teams.

[21:00] There’s a lot of work that goes into optimizing this because ultimately our desire was to use the disks to 90, 95% of their disk size to maximize storage efficiency. To put it this way, that was, you know, I guess a ballpark figure of a $1 billion project. And it was, as far as I know, the largest ever data migration in history at that time.

[21:28] And so, an extremely large engineering project with very high technical investment. Yes, if you have that scale and you have the engineering team to do it, and you’re willing to keep innovating, if you’re willing to keep optimizing and investing effort in it, then yes, you can do it. But I think the real— I mean, the cloud has been an incredible innovation. Most people are not experts at this, and most people should not be experts at this.

[21:55] Most people should focus on their applications.

21:57 — Why Dropbox migrated from AWS

Ryan:

[21:57] When you say that most people shouldn’t, my immediate thought was, one of the big projects you worked on at Dropbox was migrating away from Amazon Simple Storage Service. So why did Dropbox migrate away from S3?

James:

[22:12] Yeah, that was a desire at the company for a long time, from even before I was there. I started at Dropbox in 2010, and I spoke to Drew, the Dropbox founder, about this project. There was a desire to control the destiny of the company from a strategic perspective. At the time, this was before Dropbox reshaped itself as being more about collaboration.

At that time, it was in the file sync and share category. Owning the file system was really valuable to the company. Ultimately, we saved a huge amount of money. And this was before the company went public. We really drove massive cost efficiencies through the project. But it was hard, in a way that I think would be very difficult to emulate without a huge investment.

[22:36] At the time, it was a file sync and share category. That was the market sector, and owning the file system was really valuable to the company. Ultimately, we saved a huge amount of money. This was before the company went public. We really drove massive cost efficiencies through the project. But it was hard, in a way that I think would be very difficult to emulate without a huge investment.

[23:03] And I do think there is a benefit to an organization from having hard problems to solve. If you have a company with extremely hard technical challenges, you can attract engineers who like working on those hard technical problems. When they’ve solved those problems, they cycle off and work on different parts of the system. After we all worked, we had such a great team.

[23:28] It was a very, very small engineering team. After we shipped the storage system reliably, we all went off, and Jamie redesigned the sync protocol, the desktop client, and I worked on the file system and the distributed databases. There’s value to a business in having that level of technical investment. But it’s like having a baby; you have to raise the baby. You can’t just build a system like this and say, “That’s it, we’re done.”

[23:59] You own it, and you have to keep investing in it.

Ryan:

[24:03] Did S3 do any counter negotiation before you set out to leave them? Did they say, “Oh, we’ll cut you a deal?”

James:

[24:12] I guess I’m allowed to talk about this now. It was a long time ago. For the longest time, I don’t think they were particularly aware that this was happening. But the data center folks talk, and certainly it was noticed that Dropbox was buying up a lot of data center space. So yeah, we had, obviously at the scales that we were at, we were negotiating very good rates with Amazon Web Services.

[24:38] We weren’t paying sticker price. We were paying very, very, very good discounted rates. But at a certain point, they weren’t able to meet our cost efficiency because when we launched the system, it really was more efficient than Amazon Simple Storage Service. And that’s for a variety of reasons. One was that we were using kind of new experimental disks called Shingled Magnetic Recording. We were the first ones, I think, to use these disks at scale.

[25:02] And two, we had a very tight understanding of our workload. We were able to design the system specifically optimized for our workloads, whereas S3 has to design the system for everybody. It got to the point where Amazon would not have been able to offer us a more competitive deal because we had a more efficient system. I wouldn’t recommend another company do this right now, but I think at the time it certainly made sense for us as a company.

Ryan:

[25:28] Can you give an example of a tight understanding of your workload leading to something you could do that Amazon Simple Storage Service couldn’t?

James:

[25:36] Yeah, absolutely. So, for example, I know that, well, I’ll try not to leak any confidential data, right? But when you upload a file to Dropbox, there is a pattern of access. Typically, people access the file very quickly shortly afterwards because you’re sharing it with someone, or maybe Dropbox is processing that file to generate an image preview, and then it decays at a certain rate. We understand in general the average block size, and we also understand the access pattern.

[26:11] So we could do things like, at a certain point, we had these two clusters. One was designed for temporary storage that was kind of storage inefficient but access efficient. It was very cheap to read and write to, but it was inefficient to store. Data would get written to there first, and then in the background, it would get moved in bulk to this colder storage system. This cold storage was far more static.

[26:40] And so it was able to have more efficient algorithms and be written to in bulk. If it went down for writes, that was no problem because it wasn’t in the live path. We were able to trade off again; like systems are all about trade-offs. We were able to trade off the live data write path from the long-term read path. That was one of many examples where knowing the size of your data, where it’s accessed from, how frequently it gets accessed, and how long it takes to delete that data, you can really tune a system to your workload.

[27:13] You know, stuff like looking at even things down to knowing how much power to put in a rack. You have a rack of hardware; there’s a power distribution unit, a PDU. At the top of that rack, it has a circuit breaker which can handle a certain number of amps. We would have to figure out how many amps are required for that rack based on access patterns. There were times where we got it slightly wrong.

[27:37] There was a time when we got maybe a bad batch of hardware, and disks were failing too frequently. As a result, we were re-replicating the data more regularly. I was getting messages from the data center team saying, “Hey, we’re running the racks really hot right now.” That’s the level of optimization you can make when you get to that scale. But again, that’s multi-exabyte scale—million hard drive scale.

Ryan:

[28:03] Yeah. When I was reading about this migration, I saw somewhere in the migration you initially started with Go, and then the racks or something that you’re running on, the hardware itself was out of memory too much or requesting too much memory. And then you migrated to Rust.

James:

[28:20] So initially, the prototype was in Python, if you can believe that. To be fair, Python is actually pretty efficient for I/O. I think people give Python a bad rap for I/O-bound workloads. It’s pretty good at I/O, but obviously not great for concurrency, not great for memory management, and very hard to refactor. It was critical the system was correct. So we migrated everything to Go, and we built most of the storage system in Go.

[28:47] This is before Go was in general availability. We built the system in Go. Go’s a great language for concurrency, a great language for proxies. It’s really well designed for servers that move from one place to another. At a certain point, though, we would have, let’s just pick a number, let’s say a million. Let’s say we have a million nodes in the system, and every node has some amount of memory, some amount of disk, some amount of sheet metal in the chassis.

[29:18] Right. We would itemize all these things. You’d have a pie chart of how much money is spent on all the things. It would include items like sheet metal and screws. So you’re trying to optimize the storage. A big problem for us was the amount of memory these nodes were using. Not just the memory that we were using, but the unpredictability of it. With Go having a runtime, in a storage system, an out of memory error is pretty bad because if a node runs out of memory and restarts, that looks like a disk failure.

So that looks a lot like a disk has failed and has to be re-replicated. A batch of nodes running out of memory can lead to cascading failures throughout the system. I remember a time it was a band; it might have been De La Soul, I can’t remember. There was a band that released an album on Dropbox, which caused a big spike in load to a few files. It was very high bandwidth and it caused the machines to run out of memory.

[29:54] So that looks a lot like a disk has failed and has to be re-replicated. A batch of nodes running out of memory can lead to cascading failures throughout the system. I remember a time it was a band; it might have been De La Soul, I can’t remember. There was a band that released an album on Dropbox, which caused a big spike in load to a few files. It was very high bandwidth, and it caused the machines to run out of memory.

[30:24] So those machines, those disks oomed. As a result, no problems; the system went to try to recover that data from a whole bunch of other replicas. Now all of a sudden, you’ve taken one amount, one fire hose worth of load coming in, and you’ve turned this into seven fire hoses worth of load coming. Because now you have to do a more expensive reconstruction operation, right? So now you’ve 7x the load.

[30:47] And these are the kind of cyclical behaviors that can lead to something called congestion collapse. Congestion collapse is when workload to a system crosses a threshold where it all kind of collapses. Designing against congestion collapse is really one of the hardest parts of Magic Pocket, which is the name of the storage system. Ultimately, we switched to Rust for the storage nodes themselves.

[31:13] And this, again, this is before Rust was in GA. So that was a bit of a risky move. But we rewrote the switch to Rust, which coincided with getting rid of the file system entirely on the disks and directly addressing the disk heads. There’s an instruction set, I think it’s called ZBC, Zone Based Block Control, something like that. There’s an instruction set for accessing disks that we were using. The disk manufacturers gave us the draft specs of these new disks, and we were operating off the draft specs and directly controlling the disks.

[31:45] And so all that product was tied up together into a product called Discotech, which was the disk technology project. Ultimately, if you look at that pie chart, congestion collapse stopped and reliability improved. But if you looked at the pie chart of where all the money was going, it really shifted to be almost all disks.

Ryan:

[32:16] Metal, et cetera, the cascading failure you mentioned was there. That happened, and then there was a postmortem.

James:

[32:23] I don’t think we needed a postmortem. I think we knew as it was happening. When I was getting paged in the middle of the night on these things, it was pretty evident. This is, again, an argument in favor of the cloud. Imagine you’ve spent hundreds of millions of dollars on a storage system, and it’s out in production on physical hardware.

[32:47] You can’t just go and put new memory chips in every one of them. I mean, you can, but you have to pay people to come in and swap them out. That’s a tricky place. This is what I love about the industry. Some of the more challenging moments on Magic Pocket were stuff like, in one week, just a weird coincidence, two trucks crashed that were delivering servers.

[33:13] So there are two trucks showing up to deliver racks, and they both crashed. I don’t know, the drivers were okay. So we lost capacity for two weeks. We lost capacity for more than two weeks, probably six weeks. What do you do? What happens is the equivalent of your disk filling up on your laptop, except it’s a million disks, and you can’t tell the customers to go away; you can’t delete their files.

[33:43] A lot of tricky capacity work. Ultimately, what that led to was trying to build in all these protections against the unknowns, making sure we had the right amount of buffer planned out for anything bad that could happen, making sure we designed the system so there couldn’t be congestion collapse and so there wouldn’t be memory spikes. At one point, there’s a process called FMEA, which is a threat modeling process where you have a big spreadsheet and you kind of write down every bad thing that could possibly happen, and then you know how bad it would be if it happened. Existential risk does someone die? If there’s a fire in the data center, all these kinds of things get put into this spreadsheet, and you kind of do a bit of a pre-mortem to figure out all the potential failure modes and then design around them. I love that work. I really, I don’t know, I do like the firefight. I can’t say I like getting paged because I’ve spent my whole life on call, but I do like that rubber-hits-the-road stuff.

[34:21] Existential risk does someone die? If there’s a fire in the data center, all these kinds of things get put into this spreadsheet, and you kind of do a bit of a pre-mortem to figure out all the potential failure modes and then design around them. I love that work. I really do like the firefight. I can’t say I like getting paged because I’ve spent my whole life on call, but I do like that rubber-hits-the-road stuff.

[34:52] I like that. Wow. There’s congestion collapse and there’s no one that can help you. You’ve got to think through this problem.

Ryan:

[35:01] When I worked on infrastructure at Instagram, we had this concept of defcon knobs, which are basically these configs that you could flip to gracefully degrade your system while still operating it. But maybe in the case of Dropbox, you store fewer replicas, so you take on a temporary increase of risk in losing data because you need to. Did you have something like that, and did you flip it in that type of case?

James:

[35:29] Never. When it came to durability, we had absolute zero, non-negotiable standards; there was no room for negotiation on user durability. We had those knobs for background processes, for CPU and memory, for example. If there was a spike in load, you could turn off background processes and then turn off the test load on the system. Eventually, we built this system called Trampoline, and what Trampoline did was save us a ton of money.

[36:05] If we ever got too close to the threshold, we would just start writing data to Amazon Simple Storage Service because it’s right there. You can run your capacity way closer to the edge if you’re willing, under worst-case scenarios, just to dump 30 petabytes on S3 and then move it back when it’s done. Now, that didn’t happen very often. We would do it to test it. We would do that just to make sure the system worked. But yeah, being able to have an escape hatch for worst-case scenarios was really nice.

Ryan:

[36:35] I see. So S3 is kind of like elastic storage.

James:

[36:39] Exactly.

36:40 — How to do massive migrations

Ryan:

[36:40] When I looked at this project, it’s such a massive migration, and my first thought is how do you coordinate this whole project without breaking the system as it’s running? What are your thoughts on doing such a large migration without breaking things?

James:

[36:58] Yeah. Now, in terms of the engineers that built the initial version of the system, it was a handful—three, four, five, six engineers. It wasn’t a team of a thousand; it was a very small team. So how do you build such a large system with a small number of people? And then how do you do a high-risk migration? One of the things you do is keep things simple. You try very hard to build cleanly abstracted, simple systems so that their failure modes are very understandable.

[37:33] And so they’re decoupled, so they don’t have congestion, collapse, et cetera. Focusing on simplicity gets tricky, by the way, with people doing agentic development. They’re not the best at building simple systems. Simplicity is still the domain of human beings for now, but there’s a big focus on simplicity. The other was this very thick layer of validation checks during this migration.

[37:58] And in fact, when we did the migration off of Amazon Simple Storage Service, we had something called the dark launch where we would be moving data off of S3, but we’d keep it in both locations. We had to demonstrate to the Dropbox founders that we would have to keep this system running with no incidents, no downtime, and no data loss for six months before we would delete any of the data from S3.

[38:26] So we have double routed, and there’s one point in time, halfway through this process, where a bug got through to production. Nothing bad happened with the bug, but it was like bugs slipped through our multiple layers of the release process, etc. I went to the VP and I said, “Hey, a bug made it through to production, and we’re going to reset the launch clock. As a result, it’s going to launch later, and it’s going to cost us some amount of money, let’s say double-digit millions.”

[39:05] And they were like, great, thank you, that’s good, I trust you. That was the cool thing. I mean, Dropbox had a lot of incredible cultural values, but that was like, okay, cool. If you’re prioritizing user safety, that’s the right thing to do. No one was mad about that. It was almost like they were proud, you know? It was just like, yes, you’re operating in accordance with the principles of this company.

Ryan:

[39:31] Okay. So it was double writing both systems for a while and then some subset of data.

James:

[39:34] for a while and then some subset of data.

Ryan:

[39:36] Yeah, okay. And then you switched over reads once.

James:

[39:39] We were sure that the system was durable and we had all the validators running in production 24/7. We were just migrating as fast as we could. I think at some point we got to 764 gigabits per second of peering bandwidth between Amazon servers and ours. Certainly, someone on the network team over there noticed that there was that much data moving out. At one point, I got a slightly nasty email from someone saying it was super weird.

[40:08] I think the phrase “super weird” is like, it’s super weird that you’re doing so many reads and not that many writes. I didn’t respond to that email. But to be fair, Amazon was a great partner for Dropbox. So Dropbox still uses Amazon Web Services. There was no concern that they would do the wrong thing by us as a company. I mean, we only had an excellent experience with Amazon Web Services. But yeah, it was a moment for us.

[40:36] It probably strained the relationship somewhat.

Ryan:

[40:41] You mentioned simplicity and intuition; it makes sense. Do you have a concrete example, though?

James:

[40:48] Yeah, I’ve got a concrete example for you. Maybe it shows the difference between academia and industry. The storage system is a giant distributed system with files stored in various locations. You need a mapping from the file to where it lives on these disks. All we did was have a cluster of 1,000 MySQL nodes, a big giant database. It was indexed by the block ID and indicated that this block is on these disks.

[41:21] And that’s pretty simple. It’s not sophisticated. Every time we’d hire someone out of academia or maybe from other companies, they would say, “Oh, this is not very sophisticated because you could use a Patricia trie or you could use a distributed hash table, and that would map a block to a set of locations.” I think that’s optimizing for the wrong thing because the really nice thing about dumping a list of files and the locations in a giant database is that it is written in one location.

[41:57] If I want to validate what happened, if I want to check all the data is where it’s meant to be, I just walk over the table and check. We had services constantly walking over the table and checking. Whereas if it was a distributed hash table or some giant complex data structure, it’s very hard to validate. Designing for validation is very important. Designing for understanding is very important.

[42:19] It’s not about getting a system to work. It’s what do you do when it doesn’t work. Right. And so having a very simple boundary, that’s a very basic example. There are more sophisticated examples that take more time to explain. But something like that is a lot of engineers will feed, a lot of engineers will want to do interesting work, will want to advance in their career. They want to be seen as an intellectual problem solver.

[42:49] And so the tendency can be to design complex systems. My argument is always that simple systems are way harder to design than complex systems. Simplicity is so hard. And I think to the untrained eye, a simple system can seem obvious. The best compliment you could ever get about anything you design is when people say, “Oh, isn’t that the obvious way of doing it?” It’s the same as Convex; people say, “Oh, isn’t that just the obvious way of structuring?” Great, because it wasn’t obvious when we did it. No one else was doing it. Everyone thought we were idiots. If after the fact people think it’s obvious, then you really nailed it. But I think it requires an understanding that simplicity is the hardest thing in systems. And because simplicity is scalable. Yes, simplicity is scalable in terms of numbers of queries per second.

[43:20] What’s that? That’s just like the obvious way of structuring. Great. Because it wasn’t obvious when we did it. No one else was doing it. Right, right. Everyone thought we were idiots. If after the fact people think it’s obvious, then you really nailed it. But I think it requires an understanding that simplicity is the hardest thing in systems. And because simplicity is scalable. And yes, simplicity is scalable in terms of numbers of queries per second.

[43:48] But what I really mean about scalability is you can take a simple system and have it run for five years, have people work in it for five years, have all sorts of features added to it, and have requirements changed because the company realized the product didn’t work the way it wanted to work and it wants to change things. It still stands the test of time, whereas a complex, over-optimized system will not.

[44:11] I think that’s the tough thing about distributed systems design, especially LLM-augmented distributed systems design. Just because something works doesn’t mean it’s maintainable over a long period of time, doesn’t mean it’s understandable, and doesn’t mean it’s cleanly architected and abstracted. That stuff’s really very hard.

44:31 — Simplicity vs complexity in promos

Ryan:

[44:31] Absolutely. And I agree with you. I think it’s the long-term beneficial thing to do. One unusual thing, though, in the industry that I’ve seen is the incentive system for engineers. I mean, you mentioned the desire for an engineer to want to be seen as someone who can do something difficult. There’s that, but there’s also the incentive system of promotions. I’ve had many friends whose promotions were rejected because their work wasn’t complex enough.

[45:01] And so that kind of forces complexity, which is kind of unusual. I wanted to know what you thought about that.

James:

[45:08] Yeah, I mean it almost angers me. I just like it so much. Partly why I started my own company. I think the ideal for anyone is to be doing work where you’re being appreciated for solving the problem. If we get philosophical, this is what it was like going back to the farming days, right? There was no incentive to make it really complicated to milk a cow because the goal is to milk the cow, and then the reward is you got milk.

At a startup, that’s the same thing. The goal is to build the system, have it work, have the users like it, have it grow, and everyone gets rewarded and celebrated for solving the problem. It gets hard to scale that. So at large companies, you end up with so many layers of organization that people end up building alternative incentive structures.

[45:42] Right. I think it sounds so silly, but at a startup, the goal is to build the system, have it work, have the users like it, have it grow, and everyone gets rewarded and celebrated for solving the problem. It gets hard to scale that. So at large companies, you end up with so many layers of organization that people build alternative incentive structures.

[46:08] Right. It’s like I’m so far away from whatever the hell we’re trying to do over here that my goal now is to get all green check marks on my OKR plan. But who cares about your OKR plan unless it solves the problem? The thing that really drives me insane is when people try to chase artificial goals. I understand that if you’re in a company like this, you may have no choice in the matter.

[46:39] But what I want to tell people is there is a better way. That better way may not be available to you; you may not have job opportunities near where you are, for example. But if you do have the ability to work at a company where you are being appreciated for problem solving, that will make you so much better as an engineer. I see this when I interview people.

[47:07] If I do a deep dive with them, they’ll say they built a system, and I’ll ask, “Why did you build it?” and they’re like, “I don’t know, the VP told me to.” I’ll ask, “How’s the system used?” and they’re like, “I don’t really, I think ads use it.” I’m not sure. This is a caricature, but I think it’s very, very hard to do good engineering in that environment. You can do competent engineering, but the best engineering comes from a deep understanding of why. This is something we just drill into the team here at Convex; the team embodies this so strongly at Convex: everything exists for the why. Don’t build a fancy load balancer unless it’s needed.

[47:32] I’m not sure. This is a caricature, but I think it’s very hard to do good engineering in that environment. You can do competent engineering, but the best engineering comes from a deep understanding of why. This is something we just drill into the team here at Convex. The team embodies this so strongly: everything exists for the why. Don’t build a fancy load balancer unless it’s needed.

[47:57] Turns out we do need a fancy load balancer. We’re building it right now. But you should always start with, why are we doing this? What’s the point? I feel for people stuck in environments that are not like this. But you know what? Try to fight the system a little bit. I do see a lot of nihilism, a lot of defeatedness sometimes amongst junior engineers, a lot of this cynicism, like, what does it matter? Who cares? It’s just a big organization and nothing matters. But I think it does matter. If I think of the happiest times in my life, it’s been dedicating myself to a cause, trying really hard, and trying to do the right thing. I felt good when I went home, not trying to get promoted, just trying to do the right thing and then assuming I’m going to get promoted. If not, go somewhere else.

[48:26] Who cares? It’s just a big organization and nothing matters. But I think it does matter. If I think of the happiest times in my life, it’s been dedicating myself to a cause and trying really hard to do the right thing. I felt good when I went home, not trying to get promoted, just trying to do the right thing and assuming I’m going to get promoted. If not, I’ll go somewhere else.

[48:53] I know it does sound quaint when I’m saying this, but I think it’s possible to do this, and especially possible if you surround yourself with people like this. If someone is in a big company and they’re feeling frustrated by politics, look around and see if there’s a team of folks who just seem to want to do the right thing, just seem to want to do good stuff. I don’t think that’s selling out. I think that’s being true to yourself. That’s what real engineering is. Not trying to make a complicated fancy thing to get promoted. Just build the coolest thing that solves the problem.

[49:13] I think that’s being true to yourself. That’s what real engineering is. Not trying to make a complicated, fancy thing to get promoted. Just build the coolest thing that solves the problem.

49:23 — What technical teams should be focused on

Ryan:

[49:23] This really reminds me of something you had written. I thought it was really good writing. In the writing, there was this idea of system bias. You have this quote you’re writing. It says, here are some examples: the team is spending six months to improve performance by 10% when it was completely fine to begin with, or the team is trying desperately to force their tooling on clients who don’t need it.

[49:49] Or the team is riding their outdated system to the grave like the captain going down on the Titanic. I’ve definitely seen examples of all those types of things in industry. And so, yeah, I think it was in the context of your writing about what you should orient your team around. Not systems, but actually missions. And maybe that’s a way to fight system bias.

James:

[50:20] Yeah, I mean, one of my jobs at Dropbox wasn’t the most fun job, but it might have been one of the most impactful jobs. It was shutting down projects. You know, looking around and being like, huh, that thing over there that has had 60 people working on it for two years doesn’t seem to make a lot of sense to me. And then I, you know, it wasn’t a hostile thing, but I’d go and chat with a team and I’d say, hey, what are you all doing?

Do you believe in what you’re doing? Does this make sense? And the team in private would say, I don’t really know. But inertia is so strong. You know, this whole desire to not get in trouble, to just keep doing what you were previously doing is so strong, and talented people can end up doing things that don’t make a lot of sense. One of the things I said in that article is it’s kind of a cheesy story, but when we started building the storage system at Dropbox, the team was called the Magic Pocket Team because that was a silly code name for the system we built.

[50:44] Do you believe in what you’re doing? Does this make sense? The team, in private, would say, “I don’t really know.” But inertia is so strong. This whole desire to not get in trouble, to just keep doing what you were previously doing, is so strong, and talented people can end up doing things that don’t make a lot of sense. One of the things I said in that article is it’s kind of a cheesy story, but when we started building the storage system at Dropbox, the team was called the Magic Pocket Team because that was a silly code name for the system we built.

[51:17] The team was oriented around building that system, but as soon as we shipped it, I renamed the team to the Storage team. That actually took a bit of work because you had to rename all the email addresses, the channels, the repos, and the things. It seems like a waste of time. But my argument to the team was that the responsibility of the storage team is not to advocate for Magic Pocket, the storage system. It’s to solve the needs of storage for the organization. Because who else in the company knows more about storage than the storage team? If there was a point in time where S3 was a better idea, it would make sense to move back. Or maybe there was a different kind of storage system that meant to use. Right. It’s the job of the storage team to advocate for moving back. Right. And so I’ve seen this before.

[51:46] It’s to solve the needs of storage for the organization. Because who else in the company knows more about storage than the storage team? If there was a point in time where Amazon Simple Storage Service was a better idea, it would make sense to move back. Or maybe there was a different kind of storage system that meant to use. Right. It’s the job of the storage team to advocate for moving back. Right. And so I’ve seen this before.

[52:12] You have a team called the Puppet team. People don’t really use Puppet that much anymore, but for the job manager, Puppet versus Chef. They’d be kind of advocating for their team’s thing when really a team should be oriented around what problem they solve. They should not care about the system that survives. Because if you are on the Magic Pocket team and someone says we should move back to S3, that’s pretty threatening to your identity and your career.

[52:44] But if you’re on the storage team and it turns out it makes sense to move back, I don’t think that’s the case. But then that’s an exciting new product for you to own. I think it seems like such silly management philosophy, but I think it’s really, really important to orient a team and an identity around solving a problem and not owning and defending a system. You just see this in big companies. Inertia is so strong.

[53:07] Inertia is so strong.

Ryan:

[53:10] Yeah.

James:

[53:10] And you see people doing things they don’t believe in because that’s just what they do.

Ryan:

[53:14] If inertia is so strong, how did you fight it and close down all those projects?

James:

[53:20] I think I got lucky insofar as I was there pretty early on and I worked hard enough. I was putting in probably 16-hour days at the start. I’m not advocating for that, but I was dedicating my life to the company, and I think it became pretty obvious to people that I cared. This is a guy over here that really wants to do the right thing and cares about the company.

[53:42] At that point, you build up enough confidence, capital, that you feel comfortable saying things. I wasn’t afraid for my career; I was afraid of the wrong decisions getting made. So I felt psychologically comfortable making observations. Then at a certain point, there were a lot of engineers. I would mentor many of the staff plus engineers at the company.

People would sometimes get grumpy because they felt we weren’t doing the right thing or that something was inefficient. Every engineer listening to this has a story like this; they’re annoyed about some inefficiency at the company. My response was generally, do you think we should solve this problem right now? Because if we should, let me know the team to take some engineers off and the product to shut down, and I can redirect resources so we can solve this problem right now.

[54:16] And people would sometimes get grumpy because they were like, “Oh, we’re not doing the right thing over here,” or “This is inefficient.” Every engineer listening to this has a story like this. They’re annoyed about some inefficiency at the company. My response was generally, “Do you think we should solve this problem right now? Because if we should, let me know the team to take some engineers off and the product to shut down, and I can redirect resources, and we could solve this problem right now.”

And they’d be like, “Oh, well, we shouldn’t shut down any other stuff.” I’m like, “Cool, well, we just have this many engineers right now. If there’s anything lower priority, let’s stop doing the lower priority thing and do the higher priority thing.” Oftentimes the answer was, “Oh no, nothing else is lower priority.” Then the answer is we just have to accept it. Just have to accept it, right?

[54:42] And they’d be like, oh, well, we shouldn’t shut down any other stuff. I’m like, cool, well we just have this many engineers right now. If there’s anything higher or lower priority, let’s stop doing the lower priority thing and do the higher priority thing. This thing. Oftentimes the answer was, oh no, nothing else is lower priority. Then the answer is we just have to accept. Just have to accept, right?

[55:06] There’s no point in being angry or upset that we’re not doing the right thing all the time. I think there’s a dimension to “you break it, you bought it.” I don’t think, when I said part of my job was shutting products down, it was just going around causing problems. Ideally, it’s about solving problems, like, “Oh, this product is not going in the right direction, let’s redirect it and do this alternative thing.”

[55:28] And so I think the thing that helped me, I guess, was having a sense of ownership that instead of complaining, I just wanted to go fix problems. That was part of the culture. I mean, when I started at Dropbox, the infrastructure team was, I don’t know, seven, eight, nine people. We’d have someone join from Google, for example, and they’d say, “Well, someone should go build.” I can’t do anything without this logging framework. I couldn’t possibly do anything with this logging framework. I’m like, “Well, we’re going okay without it.” And then someone needs to build this thing. Everyone would be like, “Well, who is someone?” Right? Because it’s just us. It’s just us. We build it or we don’t build it. They’d pretty quickly come to understand, “Oh wait, it’s just us.”

[55:55] I can’t do anything without this logging framework. Couldn’t possibly do anything with this logging framework. I’m like, well, we’re going okay without it. And then someone needs to build this thing. And everyone will be like, well, who is someone? Right? Because it’s just us. It’s just us. We build it or we don’t build it. And they’d pretty quickly come to understand, oh wait, it’s just us.

[56:17] There’s no other idiots out there. We’re the idiots, right? So that was, I don’t know, I loved that time because it was just a time of accountability. Life gets easier and harder when you realize that everyone else is not an idiot. When you realize that everyone else is just dealing with their own stuff, right? I do not think someone will have good luck going around complaining about stuff and just saying this is a dumb idea and being negative.

I think people will have—everyone wants problems solved, though. So if you’re someone in an organization who is willing to put their head up and say, you know what, I think this thing over here is a bad idea, but here’s a different idea and I’m willing to own it and put the effort behind it, I think that’s a recipe for success.

[56:50] I think people will have—everyone wants problems solved, though. So if you’re someone in an organization who is willing to put their head up and say, “You know what, I think this thing over here is a bad idea, but here’s a different idea, and I’m willing to own it and put the effort behind it,” I think that’s a recipe for success.

Ryan:

[57:11] From that article, you had a great quote or a great question to think through this. It said you went to everyone or a bunch of people, and you would say, if we could be spending these resources working on any project at the company right now, would this still be the best use of time? I feel like it frames exactly what you just described really cleanly.

James:

[57:32] Yeah, I mean, I guess this is like maybe a trick for being a tech lead or a manager. Nothing’s a yes or no question. It’s a prioritization question. It’s not like, should we redesign the database? I don’t know, maybe, I guess. Is it the most important thing to do right now? No. Cool, let’s not do it. I think it’s much easier to have those conversations than to think about it as a yes or no.

[57:54] I often hear, “Oh, my VP won’t let me do blah.” Okay, well, it could be that your VP is uninformed, but it’s probably not. It could be your VP has a different set of priorities. Maybe their VP knows that you need to ship these features, and if you don’t, then the company is going to struggle. I don’t know. But to really frame it around prioritization, the way I think about the career ladder for engineers is that some people—maybe it’s not common—think that becoming a senior engineer means you get better at programming.

[58:27] But I don’t know, I think my programming abilities went down from level four onwards. I think once I got to level four, that was peak programmer for me. Then I probably went downhill from there, and I got wiser, whatever. But I think the real thing that happened is the scope that I cared about increased. At a certain point, you’re just thinking about what matters most at the company for the next five years.

[58:52] And I don’t think a junior engineer in their first year on the job should try to do this because you probably don’t yet have the wisdom, insight, or knowledge to make a good assessment. I think it would be a mistake to try to come up with redirecting company strategy. But as you grow, the real key part about growing is that the IC 6, 7, and 8 engineers are not necessarily the best programmers. They’re just getting better at having a broad perspective in decisions within a company.

[59:22] They’re just getting better at having a broad perspective in decisions within a company.

1:00:15 — Doing the right thing vs promo hypothetical

Ryan:

[1:00:15] You mentioned this idea of doing the right thing and then, you know, the byproduct, you also get promoted. I think that’s the dream, you know, do the right thing, get promoted.

[1:00:25] But in reality, oftentimes people would have to make the trade-off. Imagine a two-by-two matrix of doing the right thing, doing the wrong thing, getting promoted, and not getting promoted. Obviously, do the right thing, get promoted—great. Do the wrong thing, don’t get promoted—obviously bad. But I’m curious about the other two quadrants. Which one would you have picked when you were earlier in your career?

[1:00:50] Let’s say I came to you, I said, hey, you can do the wrong thing, but you’re going to get promoted, or you can do the right thing and I guarantee you’re not going to get promoted. Which one?

James:

[1:01:00] Absolutely the second one. I’m going to say a really tacky thing, right? There are a set of engineers who at a certain point just make infinity money. The amount of money they can make, you know, going to work at Anthropic is a huge amount of money. Right? And so there’s a certain point where it just doesn’t matter anymore. The money is whatever. But if you really want to maximize long-term income, I don’t think you should try to do that.

If you get to a certain level of experience, skill, and seniority, you’ve made it; that’s it, you’re done, the money’s fine. I think there’s this desire amongst junior engineers early in their career to be kind of over-optimizing for promotion and salary, etc., as opposed to investing in themselves.

[1:01:31] But if you wanted to maximize long-term income, if you get to a certain level of experience, skill, and seniority, you’ve made it. That’s it, you’re done; the money’s fine. I think there’s this desire among junior engineers early in their careers to be kind of over-optimizing for promotion, income, and salary, etc., as opposed to investing in themselves.

[1:01:59] They probably don’t want to hear me say this because it’s easier for me as an old guy to say this, right? But I think, for the longest time at Dropbox, there was a time at Dropbox where we just didn’t have our level system. I was tech leading the team and I was making the least amount of money on the team because I was at some point a technical manager. I saw everyone’s salary, so I was making the least money, but whatever, right?

[1:02:20] It all worked out in the end, right? Now that was a happy story. But I do really think I benefited so much from working with the best people. If you have two choices—making 20% more money now or working with the best people in the world—just be around the best people because that’s going to set you on the ship to success. Now, not everyone wants to do that. Not everyone wants to go on that. There’s no shame in just wanting to have a regular job and just be chilling and getting paid. Fine, there’s nothing wrong with that. But if you do want to maximize your career growth, then the way to maximize that is to maximize your skills. Hopefully, you’re not so cynical to think that there is no correlation between talent and compensation.

[1:02:49] Not everyone wants to go on that. There’s no shame in just wanting to have a regular job and just be chilling and getting paid. Fine, there’s nothing wrong with that. But if you do want to maximize your career growth, then the way to maximize that is to maximize your skills. Hopefully, you’re not so cynical to think that there is no correlation between talent and compensation. I don’t. If you think that’s the case, okay, I don’t know what to say, but there is a correlation between talent and compensation and growth. So my advice to people early in their career is to land at the best company with the best people doing the most important problems. Your life will be great because we are so lucky as engineers. Where else can you work solving problems for a job, solving puzzles for a job, and getting paid so well? Engineers get paid so well compared to most jobs.

[1:03:19] I don’t. If you think that’s the case, okay, I don’t know what to say, but there is a correlation between talent, compensation, and growth. So my advice to people early in their career is to land at the best company with the best people doing the most important problems. Your life will be great because we are so lucky as engineers. Where else can you work solving problems for a job, like solving puzzles for a job and getting paid so well? Engineers get paid so well compared to most jobs. The real privilege that we get is to work on something cool. That’s what I advocate for.

[1:03:53] The privilege, the real privilege that we get is to work on something cool. And that’s what I advocate for.

Ryan:

[1:04:02] I love the idea you mentioned earlier. That’s kind of counter to the common opinion I often hear. The common opinion I hear is that someone working at a big company is in this machine, and it’s kind of defeatist. They’re going, “I ship this thing, but I hate this” or “I don’t believe in this.” I really liked your perspective on doing the right thing and basically giving a damn that someone outside of you is doing the right thing too and expanding your sense of ownership.

[1:04:35] And I want to know what is your motivation for that? Because there are so many other people who are faced with the same inputs, and they come to a very different conclusion. So what motivates you to actually do the right thing when the machine doesn’t necessarily incentivize you to do so?

James:

[1:04:51] Yeah, well, I think the machine does incentivize you to do so, but I don’t think it’s visible immediately. I think people who do less job hopping really do grow the most. Because I would say very strongly, if you’re not in a job for three years, you’re not going to see whether your decisions were good. You can get more money as a junior engineer, but you cannot become a very talented senior engineer without being around for long enough to own the consequences of your decisions.

[1:05:23] It’s like playing a basketball game and leaving before the game’s over. You’re just not learning. I do think there’s an actual structural incentive towards staying in a job now. If you’re in a bad job, leave the bad job. Right? But there’s a structural incentive to be able to stay long enough to have an impact. But also, I don’t know, I guess it’s easy for me to say. I joined the tech industry when it wasn’t a very lucrative field.

[1:05:50] It wasn’t like it is now, but I just like it. The reason I left academia was because I wasn’t confident I was making the right decision. So what would happen, how I was in academia—I’m not saying everyone was like this, but you don’t know what to do. So you have to make up a problem to solve. You make up a problem, and then you make up a solution, hopefully a good one.

[1:06:18] And then you write a paper where you try to convince everyone that it was a really good idea. I hated it because I didn’t want to convince people. I just wanted to build it and see if it was good. I just wanted to build it and ship it and see; I didn’t want to play pretend and argue about whether it was good. That’s what makes me feel good. I don’t know. Obviously, if you’re an engineer who’s living paycheck to paycheck, you should maybe ignore what I’m saying.

[1:06:50] I don’t want to come across as unempathetic to anyone who’s really struggling financially. But if you are not struggling financially, the best thing you can do for your quality of life is to enjoy what you do every day. I don’t know, you could have a fancier car or you could enjoy what you do every day. I love cars. I’m a motorhead. But I promise that enjoying what you do every day is going to have a much bigger impact on your life.

[1:07:22] Personally, I don’t enjoy going to work and working on products that I believe in. Jamie, my co-founder, is the same way. We really get along well in this respect. We can only put up with doing things we actually care about and believe in. I think that’s the luxury—it’s more luxury than taking a first-class flight.

[1:07:49] That’s more luxury than going to a three-million-star restaurant. The luxury is you don’t get up and have to do a terrible job. You get up and go to a place where you like your coworkers, and you solve cool problems, and you go home and feel proud of yourself. If you can construct your career that way—and it’s not easy, it requires very active effort—then you’re going to have a good life.

1:08:13 — Why he dipped into management sometimes

Ryan:

[1:08:13] I saw in your career journey it said that you’re an occasional manager, and you seem like someone who really enjoys the technical aspects of things. So how did you decide to occasionally dip into management?

James:

[1:08:27] I feel like most people should not want to be managers, and I sometimes think it’s a bit of a red flag if someone wants to be a manager too much. Right. Because being a manager is a hard job. I think most people should go into management because they need to, because it’s necessary. What happened every time I went into management—I’m in a management role now as well—was because you have to.

[1:08:50] Right. But someone needed to manage the team, and so I do genuinely enjoy accountability. I like responsibility; I like having weight on my shoulders, I suppose. I went into management several times, but as soon as I had the opportunity to get out, as soon as someone else would come in and manage the team, I would bounce out of that role back into engineering. Now, going into management is tremendously educational.

[1:09:22] Being a manager really lets you see the world in a different way. You realize companies are more complicated than you thought. You realize the engineers are more complicated than you thought. You realize everyone on the team is going through something, and you realize, oh wow, now I understand why that thing happened. So being a manager is very educational. One thing I would say, I would strongly caution people against is going into management too early.

[1:09:44] In their career. I see this happen a lot with well-intentioned people who want to push others into management as a means of career advancement. I think it’s doing people a disservice. You should let folks take their time in an organization, in a career journey. I don’t think you should be going into management under most circumstances in the first three years of your career.

[1:10:08] I think you should ideally get to staff engineer before you do that. That’s not going to happen for everybody. But ideally, you take the time because if you go into management before you’re an excellent technician, it will limit your ability to influence strategy later in your career.

Ryan:

[1:10:24] How does that play out?

James:

[1:10:25] It plays out with people who are stuck or have roles as people managers where they see their job as making a team happy or maybe coordinating a team or dealing with all the day-to-day challenges of people on a team. But they’re not organizational leaders. They’re not pushing the team towards excellence. They’re not asking, “How can we reframe what this team is about? How can we help influence technical strategy?”

[1:10:59] And again, people management is a fine job. But I think the best companies are ones where all the managers are very technical, so they’re able to make sure the company’s doing well at a certain point. If you’re not a particularly technical manager, you’re not going to be able to evaluate the work of your team. You’re not going to know whether your team is even doing well. I guess maybe you might contribute to some of the stuff you were saying about cynical organizational attitudes where my manager doesn’t understand me.

[1:11:29] Maybe your manager doesn’t understand you if they haven’t spent enough time developing technical skills and struggling.

1:11:36 — Why you shouldn’t lead by example

Ryan:

[1:11:36] Yeah, there’s another piece that you wrote I thought was really good about leading by example, and you actually say it’s bad to lead by example. But there’s a quote in there that says modern tech workers can be an anti-authoritarian bunch at the best of times. Let’s say you’re a tech lead or manager. How do you strike that balance so you don’t lose credibility as a tech lead?

James:

[1:12:03] Yeah, there are command and control companies where the manager just tells people to do things, and they just do them. Those typically are not excellent companies, and not every company needs to be excellent. But if you want to have a company where your team is really innovating, where the engineers feel very personally responsible for making high-quality decisions, you have to have them believe in what you’re doing.

[1:12:30] And so, at Convex, I’m the founder and CTO. I can’t really go to someone’s desk and say, “Do this thing.” Now, they might do it just because they like me. There’s a good chance they would do it, but not because I’m the boss. They’d be like, “Well, why?” And that’s because that’s our culture. We don’t just do things because we’re told. If I want someone to do something, I’ll spend time with the team talking about why we’re doing it, where the market’s trending, where the gap in our product, what, you know.

[1:12:57] And then once we’ve really well articulated why something matters, people are just going to do it anyway. Sometimes you have to have hard conversations, but I think about it in terms of conflict in an organization. There’s this kind of hierarchy of the values you have and then why, what, and how. Engineers are very often debating the how. They’re often debating what algorithm we should use for this and whether we should use this container service or that container service. These are kind of like the implementation details. Most times when I see organizational conflict, it’s because well-intentioned people are debating as best they can about how to do something, but they don’t agree on why we’re doing it. If one team thinks the most important thing we can do right now is get more features out to expand our customer base, and the other team thinks the most important thing we can do is increase reliability because there’s a risk to the business.

[1:13:24] And should we use this container service or that container service? These are kind of like the implementation details. Most times when I see organizational conflict, it’s because well-intentioned people are debating as best they can about how to do something, but they don’t agree on why we’re doing it. If one team thinks the most important thing we can do right now is get more features out to expand our customer base, and the other team thinks the most important thing we can do is increase reliability because there’s a risk to the business, they’re both very valid perspectives. However, it’s going to lead to them doing very different things. Within an organization, I’m a strong believer that everyone needs to have 100% alignment on why. The stuff we argue about or debate or talk about ad nauseam is why. Largely, I just trust the team to do the right thing. I think that’s the case for any tech lead. If you want to have credibility on a team, if you think that you can just tell someone to do something and they’ll listen to you because you’re the senior guy, no, they won’t.

[1:13:54] They’re both very valid perspectives, but it’s going to lead to them doing very different things. Within an organization, I’m a strong believer that everyone needs to have 100% alignment on why. The stuff we argue about or debate or talk about ad nauseam is why. Then largely, I just trust the team to do the right thing. I think that’s the case for any tech lead. If you want to have credibility on a team, if you think that you can just tell someone to do something and they’ll listen to you because you’re the senior guy, no, they won’t. They won’t listen to you. If they listen to me, it’s because I’ve come in and explained it in a way that resonates with them. Again, one of my jobs at Dropbox was to resolve situations. Someone would say, “Oh, that team’s an idiot; they won’t do blah.” Then I’d go talk to that team. That team was not an idiot. We talked through it, and they would decide to do the project. It wasn’t because they were scared of me.

[1:14:30] If they listen to me, it’s because I’ve come in and explained it in a way that resonates with them. Again, one of my jobs at Dropbox was to resolve situations. Someone would say, “Oh, that team’s an idiot, they won’t do blah.” Then I’d go talk to that team. That team was not an idiot. We talked through it, and they would decide to do the project. It wasn’t because they were scared of me.

[1:14:51] I hope they weren’t. At least it was because I took the time to figure out what their motivations are. That’s something you really have to learn. If you’re a tech lead, you don’t really have that much authority over people, and you have to encourage them and get them to believe in what you’re doing. If you are leading through authority, you’re not going to have a culture where good ideas arise from within the organization.

[1:15:17] People are just going to do what they’re told.

Ryan:

[1:15:19] Influence without authority is huge. Even big companies like Meta technically don’t have titles; everyone’s just a software engineer. Is that also how Convex is run?

James:

[1:15:32] I mean, we’re a very flat organization. There are people in tech lead roles. I don’t completely buy that everyone’s a software engineer thing. It’s like, you know, Anthropic. Everyone’s a member of technical stuff because it’s kind of like a wink-wink thing. You kind of know, it’s like no one’s mentioned the title, but you kind of know if the most senior person in the company comes to your desk, you’re probably going to notice. I think there’s no point in playing pretend. People know. But I will say that you should have an organizational culture where you don’t do something just because a senior person says something. Now, I do think that reputation matters. I certainly think that if a very experienced person at a company comes and tries to explain something to you, you probably should listen to them because they probably have some wisdom.

[1:16:03] People know. But I will say that you should have an organizational culture where you don’t do something just because a senior person says something. Now, I do think that reputation matters. Like, I certainly think that if a very experienced person at a company comes and tries to explain something to you, you probably should listen to them because they probably have some wisdom. There might be something to be learned there. You should be open to it. But ultimately, even the most senior person, I don’t think should be leading through authority. They should use the benefit of their experience to be able to articulate, to win the hearts and minds. And that’s how your engineering culture is so important. And if you want a culture of ownership and innovation and drive and enthusiasm, and people trying to do the right thing, not get promoted, you have to have a culture where everyone believes in what they’re doing, and no one’s going to believe in what they’re doing.

[1:16:27] There might be something to be learned there. You should be open to it. But ultimately, even the most senior person, I don’t think should be leading through authority. They should use the benefit of their experience to be able to articulate, to win the hearts and minds. And that’s how your engineering culture is so important. If you want a culture of ownership, innovation, drive, and enthusiasm, where people are trying to do the right thing and not just get promoted, you have to have a culture where everyone believes in what they’re doing, and no one’s going to believe in what they’re doing.

[1:17:00] If the senior principal engineer comes to the desk and says, “I’m not going to tell you why, but you have to delete this database and do this other thing,” that’s not an empowering statement. What the empowering statement is, “Hey, let’s spend some time together to talk about where this product’s trending and how this is probably not going to work out and how there might be a different way we can solve this problem.”

Ryan:

[1:17:20] On the topic of tech leadership, in that article I mentioned, the title was “Don’t Lead by Example.” I think that might be confusing for people. Can you explain why you think you shouldn’t lead by example?

James:

[1:17:33] Yeah, leadership by example is a very passive thing to do. Engineers are passive people at the best of times. I think that’s stereotypically a little bit part of that personality. The very concrete example is when I first started becoming an engineering leader, I was trying to lead by example. I wanted to demonstrate the behaviors I wanted everyone else to have. Very specifically, with regards to on-call and people getting paged, I wanted people to have high ownership. I wanted people to jump on issues as soon as they happened. So I would do it. I’d be the first one to respond to a page. I would always be writing up the reports. I would always be jumping on all the bugs and stuff, really falling over myself to show how I wanted people to be. But from their perspective, all they see is that the lead is just doing all these jobs, and they don’t know. They’re like, “Oh, maybe that’s James’s job,” or “Maybe James knows how to do it, and I don’t know how to do it.”

[1:18:05] I want people to jump on issues as soon as they happen. I would do it. I’d be the first one to respond to a page. I would always be writing up the reports. I would always be jumping on all the bugs and stuff, really falling over myself to show how I want people to be. But from their perspective, all they see is that the lead is just doing all these jobs, and they don’t know. They’re like, “Oh, maybe that’s James’s job,” or “Maybe James knows how to do it, and I don’t know how to do it.”

[1:18:37] Turns out I didn’t know. I was just kind of figuring out, or maybe he likes doing those things. At a certain point, being a leader is about understanding human psychology. I don’t think you can just act a certain way in front of people and wait for them to copy you. I think you should act with integrity and values, and you should own the values of the team.

[1:19:01] But you sometimes have to explain stuff; sometimes you have to tell people very, very specifically. There’s a trajectory almost every high-achieving leader goes through. They become a tech lead and care so much that they become a micromanager. They review every line of code and are involved in every decision, and at a certain point, they become the bottleneck for the team.

[1:19:26] So this person is so overwhelmed; you might have gone through this yourself. They’re so overwhelmed, and they’re like, “Wait, my team doesn’t even seem busy right now. And I’m so busy. I’m reviewing all this code. I’m doing the strategy, what’s going on?” Then a manager will come along and say, “Hey, you’re micromanaging. You got to let your team have more ownership.” So the tech lead says, “Okay, sure, whatever.”

[1:19:49] I’ll let them own stuff. And they just take their hands off the wheel. Then the team falls apart because you can’t just stop doing the things you’re doing. You have to go and have a conversation with people. I see leadership as this kind of slider between oversight and accountability. When someone’s new to the team, when they’re very junior, they’re in a mode of oversight, like you’re checking their work. But at a certain point, you have to dial down the oversight and, very importantly, dial up the accountability. Instead of saying, I’m not going to look at what you’re doing anymore, you say, okay, cool, you’ve got this project. Let me know when it’s going to get done. Next Thursday.

[1:20:17] Right. But at a certain point, you have to dial down the oversight and, very importantly, dial up the accountability. So instead of saying, “I’m not going to look at what you’re doing anymore,” you say, “Okay, cool, you’ve got this project. Let me know when it’s going to get done. Next Thursday.”

Ryan:

[1:20:34] Cool.

James:

[1:20:35] All right. What’s the plan? This is going to happen. How are you going to know it’s correct? Great, great, great. It’s on you. I expect you to do that. Let me know if there are any issues. The ownership relationship is explicitly on them. What you want to do is basically encourage ownership within teams. You can’t go from owning something to not owning something and expect that to develop.

[1:21:00] You have to go have those conversations. You have to go and give people accountability. Now, what I have found, even though that can feel like an awkward conversation, most people genuinely like accountability. Most people like to own their work. Most people like to say, “Hey, this is on you. We’re all going down with the ship, right? I’m not going to leave you high and dry.”

[1:21:23] I’m the tech lead. I still take responsibility for this project. That’s how you can develop people within your team. A tech lead shouldn’t be about you as the boss and the team as the people who do the work. It’s about this kind of flow where you’re the more experienced person, typically maybe the more organized or more strategic person, and you’re working on developing your team members so they can take your job and then you can do something else.

Ryan:

[1:21:55] That slider you mentioned, what if you give someone accountability and they blow it? Do they go back down the slider to where you start micromanaging?

James:

[1:22:06] You have to recognize it. Right. So this is growth. I mean, this is growth. Right. It doesn’t always work. I had another article about paper cuts or something. So we’d call these paper cuts. Some decisions don’t matter that much. You get them wrong, and maybe it sets you back a week. You get them wrong, and maybe something’s a bit suboptimal. These are great candidates to give people accountability for because they do it.

[1:22:33] And if it doesn’t work, they get to experience it, and it will feel bad. I guess that’s good. It’s good to feel bad. You shouldn’t be demoralized, but it’s good to try something. It doesn’t work, and you’re like, oh, wow, that didn’t work. Then you feed that back into your little LLM in your brain, and you get better next time. Right? That’s a growth experience.

[1:22:56] I think what you don’t want to let people do is lose their arm. Paper cuts are fine, but I would not let someone, a junior person especially, design the replication system at Convex. You have to have safeguards. One of the arts as a CTO, a leader, or tech lead is figuring out the right level of altitude for how to know whether you can trust someone to take on ownership for something.

1:23:23 — How to mentor Senior Staff+ engineers

Ryan:

[1:23:23] You mentioned that, because you were the most senior engineer at Dropbox, at some point you had to mentor other very senior engineers. When I think about mentoring a junior engineer, it’s relatively straightforward patterns. But when I think about, let’s say, needing to mentor a senior staff engineer, maybe even a principal engineer, how do you mentor someone like that who’s already so polished and knows how to take ownership?

[1:23:51] Yeah. How do you mentor someone so senior?

James:

[1:23:53] Yeah. I mean, there’s in two ways. One is there is still just a whole bunch of commonality. Everyone goes through the same problems. They don’t know how to deliver harsh feedback. There is a bunch of just standard stuff people are working on. But I do think that at a certain point everyone needs to become the best version of themselves. Sounds so cheesy, right? But there is not an archetype for what. According to me, there’s not an archetype for a senior principal engineer. You’ve got to be your own brand of engineer, right? So you might be. For me, I’m like the strategic kind of collaboration, simplicity, abstraction engineer. And my coding just fell off a cliff. You know, some engineers are like the deep science fiction, you know, the super hardcore science, you know, problem-solving engineer.

[1:24:18] According to me, there’s not an archetype for a senior principal engineer. You’ve got to be your own brand of engineer, right? For me, I’m like the strategic kind of collaboration, simplicity, abstraction engineer. My coding just fell off a cliff. Some engineers are like the deep science fiction, the super hardcore science, problem-solving engineer.

[1:24:42] So everyone’s going to find their own brand. Oftentimes, what I’ll be working on is a combination of two things. One, finding their strengths and helping them be more spiky—helping them really excel in the area that makes them special. At the same time, I’m almost always working on the personal side of being an engineer, like the organizational and personal understanding, people understanding the why. I don’t know, we’re so mathy as an industry. We think that somehow, even silly things, like for example, I have to tell so many senior engineers that you can’t change someone’s mind in a meeting.

You just can’t do it. You can disagree with someone, and they’re going to be mad or whatever, right? But to change someone’s mind involves going through a complex series of neurological processes that don’t happen live in front of ten people in a meeting. If you force someone to agree with you about them being wrong, they’re just going to go along with it, and their ego is going to get bruised and whatever.

[1:25:23] You just can’t do it. You can disagree with someone, and they’re going to be mad or whatever, right? But to change someone’s mind involves going through a complex series of neurological processes that don’t happen live in front of ten people in a meeting. If you force someone to agree with you about them being wrong, they’re just going to go along with it, and their ego is going to get bruised.

[1:25:52] So, one thing, you know, firstly, meetings generally aren’t for decision making. Most of the time when you identify a disagreement, I would point out the disagreement and why I don’t agree or where the problems are, provide enough information, and then let it sit. Let them go and reflect on it and come back and have a conversation a week later after they’ve thought about the why.

[1:26:16] Right. And so this is a lot of very psychological, but it makes no sense to be like, well, they should be able to change their mind. I mean, well, they won’t, right? That team should do this. Well, they didn’t, right? You know, people should like my API? Well, they don’t, right? I think it’s like that whole, it’s almost like a capitalist attitude towards interpersonal behaviors.

[1:26:41] Right. It’s like capitalism rewards success or impact. It doesn’t matter how well-intentioned you were. If you didn’t manage to convince, you could be the smartest person in the world. But if you can’t change someone’s mind, then you are kind of useless. Right? And so I think a lot of seniors still struggle with this. It doesn’t matter how smart you are, it doesn’t matter how right you are; are you effective?

[1:27:07] And a lot of being effective at that level is about uncertainty, project management, and simplicity, but also kind of interpersonal dynamics because most hard problems happen with a team. Not always, but generally when you’re at that level, you’re going to have to have 10 or 20 people working with you to get something done. And that’s a whole different ball game.

1:27:30 — Career advice for the AI era

Ryan:

[1:27:30] On the topic of career advice, the industry’s changed a lot in the last five years because of all these agentic tools. I wanted to know if you thought there is any career advice that has majorly changed in the last five years. Is there something that you used to say five years ago that you don’t say anymore, or vice versa?

James:

[1:27:52] No, I don’t know if I’ve changed my perspective much, but I think the industry has changed this perspective dramatically. I mean, let’s just be honest about it. There is incredible demand in Silicon Valley for senior engineers, and it is getting harder for junior engineers to succeed and grow for a variety of reasons. Why is there demand for senior engineers? Well, because large language models can’t do everything.

[1:28:17] Well, the architecture and simplicity in design are still the domain of human beings, despite what you might hear on Twitter. Every company, including the labs, is desperately hiring senior engineers. But junior tasks are getting a little bit commoditized, and that worries me because I do think that learning—wisdom is kind of facts put into practice and then synthesized. People would argue that it’s easy to learn now because of ChatGPT; you can just go ask it a question about how two-phase commit works or what’s the difference between snapshot isolation and serializability, and it will give you probably a pretty good answer. But I think growth as an engineer does require wisdom, and wisdom only really happens when you synthesize it, in my opinion. I’m still very bullish on young people. We’re hiring junior engineers at Convex, and I’m very excited about it. I love working with junior engineers who are really hungry to grow.

[1:28:56] Or what’s the difference between snapshot isolation and serializability? It will give you probably a pretty good answer. But I think growth as an engineer does require wisdom. Wisdom only really happens when you synthesize it, in my opinion. I think I’m still very bullish on young people. We’re hiring junior engineers at Convex. I’m very excited about it. I love working with junior engineers who are really hungry to grow.

[1:29:28] But what I would say is train your mind. Do not listen to anyone who tells you that there is an advantage to having less knowledge. I’m not sure if you’ve seen people say these ludicrous things, like, oh, maybe in the future, not knowing engineering will be an advantage because you won’t have biases and you’ll just use Claude. I think these are ludicrous statements. Software engineering is an intellectual discipline that helps you think. The best software engineering is not about knowing syntax and it’s not about knowing an algorithm; it’s about being really good at conceptualizing problems, breaking them down into building blocks, and coming up with clean solutions. That requires experience. It’s like doing weights with your mind. Just like if you went to the gym and it never hurt.

[1:29:56] The best software engineering is not about knowing syntax, and it’s not about knowing an algorithm. It’s about being really good at conceptualizing problems, breaking them down into building blocks, and coming up with clean solutions. That requires experience. It’s like doing weights with your mind. If you went to the gym and just picked up really light weights, you’re not growing. Also, if you went to the gym, picked up a heavy weight, and let the robot pick it up for you, you’re also not really growing. You have to do the reps. It’s tough, but find a way to stress your brain every day.

[1:30:17] If you went to the gym and just picked up really light weights, you’re not growing. Also, if you went to the gym and picked up a heavy weight and thought, “Okay, cool, I think I can do it,” but let the robot pick up the weight for you, you’re also not really growing, right? You have to do the reps. It’s tough, but find a way to stress your brain every day. Find a way to.

[1:30:47] To avoid. Now, obviously, agentic coding is here, right? Obviously, that’s. I could never tell someone to never use a coding agent, because that would be silly. But I can say that it is easy to fall into a passivity trap, where you’re just being passive about your learning. I do think you need to spend some time in the intellectual wilderness of not being able to solve a problem and struggling.

[1:31:16] I would say if you’re running into a new problem, try to think of a solution yourself and then go check with a large language model. It’s going to be hard for you. It’s almost like, for example, I’m not very good at reading anymore. I got to be honest, I find it hard to sit down and read a book because my brain has been fried by the stimulation economy, right? I find that I go home and I’m tired, and I watch a YouTube video. I don’t tend to go home and read a novel. I should be better at that, but it takes discipline to do that. I would say similarly, it’s getting harder to solve a difficult problem without reaching for help. It’s getting harder and harder every day to be faced with a very difficult intellectual problem and not be like, well, I’ll just do a Google search or I’ll just ask Claude.

[1:31:45] I find it hard. I go home and I’m tired, and I watch a YouTube video. I don’t tend to go home and read a novel. I should be better at that, but it takes discipline to do that. I would say similarly, it’s getting harder to solve a difficult problem without reaching for help. It’s getting harder every day to be faced with a very difficult intellectual problem and not think, “Well, I’ll just do a Google search or I’ll just ask Claude.”

[1:32:10] You’re whatever, I’m still doing the work. I don’t know. I would really encourage people to practice using your brain every day.

Ryan:

[1:32:19] If I was just playing devil’s advocate or thinking from the junior engineer perspective, I might think, well, the proof that Claude can do this work today means that I don’t need to know it today or tomorrow or in the future. So why even build that skill in the first place? Also, these agentic tools have positive trajectories too.

James:

[1:32:43] Yes. So firstly, the agents are not particularly good at a lot of parts of engineering currently. Probably everyone should agree Claude is not good at designing distributed systems protocols right now, for example, or managing a 3 million line code base. There are parts of engineering where engineers are valuable. And like I said, you know this because the labs are all hiring engineers.

[1:33:10] No matter what they say, they’re still hiring engineers, right? Desperately hiring engineers. Really, really aggressively hiring engineers. So engineers still have value. And maybe they won’t in the future, but I doubt it. I really do think there’s a role for human ingenuity in engineering. And so here’s the trade-off I would pose to people. I mean, there’s kind of three paths. One, get out, get out of engineering, go mow lawns and do whatever if you want.

[1:33:41] That’s the most nihilistic attitude. I don’t believe in that. I really love engineering. I believe that there’s promising roles for human beings in engineering. So the second is to realize that there’s value right now for humans in engineering. And maybe one day AGI will be here. Let’s say in a year’s time AGI will be here. And there’s two people. One person just gave up on problem solving right now and they’re just feeding the machine. They’re running 17 coding agents in parallel, accepting everything Claude says, and they’ve given up.

[1:34:15] Right. And one person said, you know what, I still want to have an active role in my learning. I want to understand what it’s doing. I want to think hard about problem solving, right? Played out one year, AGI arrives. Who is going to be better placed for the future, right? The person who’s been training their mind, right? You don’t—engineering is not a means to an end. It’s a mechanism for improving your mental processes.

[1:34:41] It’s like, I still do whiteboard coding interviews, and by the way, Anthropic still does whiteboard coding interviews, just in case you were wondering. They don’t say, just use Claude. So I still do whiteboard coding interviews with candidates. Candidates are getting worse at coding. That’s absolutely true. But I don’t know any better vehicle for evaluating someone’s intellectual capacity for problem solving than seeing them solve an engineering problem.

[1:35:08] It’s a great mechanism for doing so. If you were doing a math degree, a big part of doing advanced mathematics is proving theorems. By the way, the theorems are already all proven. Part of the exams is proving a theorem that has already been proven. You might say, well, what’s the point of proving that theorem? It’s already been done. The point is that the act of proving that theorem improves your mind, and then you can go and do more innovative stuff. I would say, sure, you may not. I still think that there is a very promising role for human beings in engineering. Maybe not in coding, but coding and engineering are very different things. Even if you don’t believe me, you can either give up now, but.

[1:35:31] It’s already been done. Well, the point is that the act of proving that theorem improves your mind, right? And then you can go and do more innovative stuff. So I would say, sure, you may not. I still do think that there is a very, very promising role for human beings in engineering. Maybe not in coding, but coding and engineering are very different things, right? But even if you don’t believe me, you can either give up now, but.

[1:35:56] Or you could just keep trying to improve your brain, and you’ll be better off anyway. Imagine if you’re wrong. Imagine if you take the defeatist view. Imagine if you say, “Well, AGI is coming next week, why bother doing anything?” And imagine you’re wrong. Oh my God, you just got off the ride. You get off the ride, and there’s so much cool stuff. I mean, this is a cool time for engineering, right? This is a really cool time. There’s so much cool stuff happening. And I hate this talk about how AI means humans don’t have a role anymore. AI means no one’s going to have a job. AI means we’re not going to do anything new. What I like is, hey, look at this. All this new cool stuff we can build. Look at the ways we can make people’s lives better.

[1:36:19] This is a really cool time. There’s so much cool stuff happening. I hate this talk about how AI means humans don’t have a role anymore, that AI means no one’s going to have a job, or that AI means we’re going to do nothing else new. What I like is, hey, look at this—all this new cool stuff we can build. Look at the ways we can make people’s lives better.

[1:36:42] And to be honest, I really wish my peers and my cohort would stop it with the real doomer, human elimination kind of narrative. I think there is a little bit of a psychological anchoring. As part of Convex, we didn’t start Convex to eliminate jobs; we started Convex to make it easy for people to build cool stuff.

[1:37:10] I think the more we as an industry get behind, let’s do everything we can to make it possible for people to do more cool things. I think it’s a really exciting future we have ahead of us.

1:37:21 — Why he started his own company

Ryan:

[1:37:21] I wanted to talk about what you’re working on now at Convex and why you quit Dropbox to build Convex.

James:

[1:37:27] Dropbox is a great place. At a certain point, I was there eight years. I don’t know if I outgrew the company, but I’d been the most senior engineer for a while. At a certain point, I wanted to grow and do my own stuff, so I left the company very amicably and started Convex. Why Convex? Convex started pre-agentic era because my observation was that the real differentiator in the success of a project, especially a large project, is the quality of the abstractions, the quality of the design, the quality of the architecture. Systems that thrive over time and are extensible are systems that are architected well. In my mind, the most difficult challenge in engineering is distributed state management. How do we store state reliably and reason about it, modifying it concurrently with other users? We designed a platform for application building based on our experiences building large-scale systems.

[1:38:08] Systems that thrive over time and are extensible are systems that are architected well. In my mind, the most difficult challenge in engineering is distributed state management. How do we store state reliably and reason about it while modifying it concurrently with other users? We designed a platform for application building based on our experiences building large-scale systems.

[1:38:34] So Convex is a transactional database where the transactions are written in TypeScript; they run as stored procedures. TypeScript stored procedures are serializable. There’s automatic reactivity, so what the client sees is a consistent view of what’s on the server. I would say Convex is a very designed platform because Convex is designed to be very composable and fit together well. We designed Convex for developers to use.

[1:38:59] In particular, we wanted to make it so that application developers were able to build complex full-stack applications. That was the goal of Convex. Now all of a sudden, a coding agent development came along. That’s been really interesting for us because it turns out that what humans find hard is also what agents find hard. The coding agents are not particularly good with large code bases.

[1:39:24] They’re not particularly good at reasoning about action at a distance, race conditions across services. They’re not particularly good at simple architectures over time. And these are the things that Convex gives you as a developer. So the idea now, and almost everyone using Convex is using Convex because they have the coding agent doing their front end. But they need a backend abstraction, a higher-level abstraction than something like Amazon Web Services or something like hosted PostgreSQL, which makes their problems go away.

Ryan:

[1:39:56] So this is a layer of abstraction on top of those types of primitives that makes it easier for an application developer.

James:

[1:40:11] It ties into a lot of the stuff I said earlier about making problems go away. Frankly, I watched the interview you did with Barbara Liskov. Barbara was my advisor in grad school, and we worked a lot together on abstraction and the value in clean designs that minimize complexity. AWS is a fine tool. PostgreSQL is a fine tool, although none of the mainstream databases are that great, frankly.

[1:40:36] But they’re fine tools, but they don’t make problems go away. Right. And so the idea of Convex is a higher level set of abstractions that you can use and not reason about state management, not reason about concurrency, not reason about scheduling, not reason about transactions, not reason about polling and data sync and type safety and all those things. So Convex is a, if you think about, you know, the history of engineering over time, the abstraction floor raises.

[1:41:05] You know, when Barbara Liskov was first starting, she was using punch cards. I don’t know if she mentioned it to you, but when she started as a programmer, she never heard the word “programmer” before. That was the first time she heard the word. And then you went from punch cards to having proper operating systems, and then languages like C, and then higher-level languages, and then you had cloud computing.

[1:41:31] And over time, the abstraction floor raises, and you largely forget about what’s going on beneath the surfaces. Most people don’t think about how Amazon Simple Storage Service is implemented. I do, but that’s what I used to work on. Most people just use it, and it stores your data and gives it back. That’s great; that’s a successful abstraction. But I do strongly believe that the world is and has been overdue for a new abstraction one level up the stack.

[1:41:55] And especially now that people are doing agent development, they don’t want to own a PostgreSQL instance. They don’t want to think about Kafka versus RabbitMQ. They don’t want to think about what set of tools to use. They want it just to work so they can focus on building their application.

Ryan:

[1:42:11] When you talk about the abstraction, there’s obviously a lot of stuff going on behind the scenes in Convex and the technical side. What is it that Convex is building behind the scenes that you’re most excited about and why?

James:

[1:42:27] Basically, Convex is a new operating system in some respects. We have the primitives: queries, mutations, actions, subscriptions. What I think is cool is how we built this. We have our own distributed database that tracks read ranges and write ranges and does very efficient subscriptions over WebSockets, et cetera. So that’s the current operating system set of primitives.

[1:42:56] But Convex is getting much larger workloads now and much more interesting workloads and more high-performance workloads. So we’re in the process of developing a slightly lower-level API for doing very efficient background processes, singletons, and APIs like fork, like operating system primitives. I’m pretty excited about launching these and how much faster it’s going to make various Convex components like the workflow system.

[1:43:28] And to be honest, the thing I find exciting every day, challenging every day. I still find Convex very hard. I struggle every day. I don’t find my job easy. I feel confident at my job, but it’s not easy. Designing the new API for this is super hard. I can’t just go ask Claude; it’s not going to give a good answer, right? Because it’s innovation, it’s new ideas. I really enjoy it.

[1:43:56] I find it stressful sometimes. I find it challenging and tiring, but I also find it exciting. I would encourage engineers to try to find this kind of stuff to work on, where it’s like you’re on that edge of, “I’m really liking this, but also it’s a bit tricky.”

Ryan:

[1:44:16] You mentioned fork, and in operating systems, I’m familiar with it. You take the existing process and kind of split it. What’s the idea of fork in a distributed system?

James:

[1:44:27] So Convex almost never has scale issues with regards to live traffic. Live traffic is typically bound by user-facing interactions, people clicking on stuff, running a website, acting on a website. Every now and then, someone will come to Convex and want to kick off a million background jobs to do some background processing. It’s a big workload. You can programmatically trigger huge workloads, right?

And so one of the things we have to scale is these background workloads, and a lot of them involve things like scheduling. There are a lot of workloads in Convex that would be very efficient if you had a background singleton process to perform things like aggregates. I’ll give a very silly example, right? If you’re building an election on Convex, a voting system, and every vote is a new row in the...

[1:44:50] And so one of the things we have to scale is kind of these background workloads, and a lot of them involve things like scheduling. There are a lot of workloads in Convex that would be very efficient if you had a background singleton process to perform things like aggregates. I’ll give a very silly example. Let’s say you’re building an election on Convex, a voting system, and every vote is a new row in the table. You want to show a tally of the votes. One way of doing this is having a bunch of background processes or cron jobs adding these things up. One way is doing a table scan, which is the obvious way to use PostgreSQL, which doesn’t scale. The other is to have a background job, which, if there are new votes, adds them all up and keeps a tally. If there are no new votes, it goes to sleep and waits on a condition variable to wake up again when there is a new job to perform.

[1:45:19] In the table, you want to show a tally of the votes. One way of doing this is having a bunch of background processes or crons adding these things up. One way is doing a table scan, which is the obvious way to use PostgreSQL, which doesn’t scale. The other is to have a background job that, if there are new votes, adds them all up and keeps a tally. If there are no new votes, it goes to sleep and waits on a condition variable to wake up again when there is a new job to perform.

[1:45:45] And so these are the kind of primitives that we’re working on right now. Most people won’t even know they exist, but they allow us to build this very high-performance primitive for scheduling, aggregates, background aggregations, et cetera. I’m pretty excited about the next generation of workloads we can support.

1:46:05 — The most technically challenging work of his career

Ryan:

[1:46:05] As a result, when you reflect on your career, it sounds like you’ve done a lot of gnarly technical work across your PhD. Dropbox seemed like pretty intense systems work, and Convex is also doing a lot of cool stuff. When you look back on your career, what was the most technically stimulating work you’ve ever done? Why was it hard, and what did you learn from it?

James:

[1:46:29] There were certainly times in grad school where we were formally modeling consensus protocols and stuff. I’d be on the phone with Barbara Liskov on weekends, talking through and trying to reason about this in our heads. That was pretty intellectually stimulating and fun. But I think the stuff I found most stimulating was working on a very large storage system with a team where things were going wrong.

[1:46:53] Where the rubber hits the road, that’s where I find, and this is every day at Convex. The rubber hits the road. We have a compaction process that runs in the background, but it’s running into issues. We might have to redesign it using partitioning, etc. I feel most intellectually stimulated where there’s a really clear constraint in front of me.

[1:47:16] And that to me is engineering. If I don’t actually know what the definition of engineering is, I’m just going to make it up in my mind. Engineering is science with constraints. It’s like, how do you solve problems in the presence of resource constraints? I’m not particularly interested in constraint-free environments. That’s art. I like craft and engineering. The more visceral and difficult the constraints, the more fun that is for me.

[1:47:47] And I’ve been lucky enough to, whether it’s luck or intention, I don’t know, but I’ve always placed myself in those environments. You know, let’s go get on the hardest team and own the hardest problem and then put the effort in to survive.

1:48:10 — How he got involved in Silicon Valley

Ryan:

[1:48:10] This question might be a little bit off topic, but I know you were a consultant for the TV show Silicon Valley. I love that show, and I gotta hear, how’d you get involved with that?

James:

[1:48:21] Yeah, that was a lot of fun. A lot of folks might not know this: I had nothing to do with season one. Many TV shows don’t know whether they’re going to survive as a series. Mike Judge, who wrote Silicon Valley, is also known for Beavis and Butt-Head and Office Space. He started his career as a software engineer at, I think, Lockheed or something. So he actually was a software engineer, which a lot of people don’t realize.

[1:48:48] And so Silicon Valley was like a throwback to the kind of work he did. If anyone’s seen the movie Office Space, you would get this. That’s really a dystopian cubicle era tech industry film. They did season one of Silicon Valley, and then it was very popular, so they got picked up. They had to figure out what to do for season two, but they didn’t know what to do because they had written a storyline that gets to the point where there’s a compression algorithm, and then what happens?

[1:49:18] And so they needed to find an expert on compression. I guess ostensibly that was me. I don’t know whether I was an expert on compression; I guess I was an expert on storage at least. They came to the office, and we just chatted. It was so much fun. I was pretty heavily involved in the show. A lot of it was storyline. So first, like, yeah, sure, what would you do with the compression algorithm?

What would you design for a storage system? Coming up with story ideas that were technically accurate, you’d be surprised to know how much they care about accuracy. A lot of people I know can’t watch that show because it’s just so creepily accurate. They find it so cringy. Partly why Silicon Valley can be so cringy is because it’s real. Those stories are almost.

[1:49:47] What would you do? Could you design a storage system? Coming up with story ideas that are technically accurate, you’d be surprised to know how much they care about accuracy. A lot of people I know can’t watch that show because it’s just so creepily accurate. They find it so cringy. Partly why Silicon Valley can be so cringy is because it’s real. Those stories are almost...

[1:50:14] Almost. Maybe not everyone. So many of the stories in Silicon Valley are just real stories. They went to a bunch of companies, just farmed everyone for stories of crazy things that happened in the tech industry, and they wove them into the series. All the characters are based on real people and real archetypes. But they also cared very much about technical accuracy. So I would also do technical consulting.

[1:50:37] And then they’d say stuff like, “Oh, we’re building a data center in our house. What should the rack look like? And what should the diagram on the wall look like?” Part of me wanted to say, “Oh, well, it doesn’t really matter. No one’s going to care.” But they were like, “No, no, no, it matters.” They really cared to get it right. So, yeah, I love the show, but it can be hard to watch just because of, oh my God, how real it can feel. Yeah.

[1:51:01] How real it can feel. Yeah.

Ryan:

[1:51:05] Are there any Easter eggs where you look at that and go, that’s unusually accurate? Or, you know, that system diagram actually is very spot on.

James:

[1:51:15] I can’t. I don’t think I can even say them because there were stories of early Dropbox. There were stories of having ideas ripped off by other companies and being tricked into having meetings with folks, only to find it may be the competitor’s team there to steal the information. A lot of those stories are real. And so there are people who watch Silicon Valley and think, oh, wow, that was something I went through.

[1:51:43] And probably it was because it was about that situation.

Ryan:

[1:51:48] That’s such a cool experience. Did you get paid for that, or was it just for fun?

James:

[1:51:52] Yeah, that’s a complicated question. I got paid because I had to get paid; it was like a Hollywood union thing. I didn’t want to get paid because it made my visa more complicated. So I’ve got a green card now; I’m all good. But there was something where they had to pay me $400 anyway. I made a grand sum of $400 off that show.

1:52:16 — Career regrets

Ryan:

[1:52:16] Looking back on your career, is there any regret that comes to mind that maybe other people can learn from?

James:

[1:52:23] Yeah, I mean, I think I underinvested in my personal life, to be honest. People don’t probably say that much because I think you can be all about growth, and sure, I could have grown more. I could have dropped out of grad school, say, three years in or four years in. I probably would have learned just as much. I could have taken a job at Dropbox two or three years earlier and made a lot more money.

[1:52:48] Everyone who has been in the industry long enough has been offered the chance to co-found several billion-dollar companies. Everyone has a story about the times they could have been a billionaire several times over. I don’t really regret those. Yes, it’s been a lot of sacrifice, to be blunt. I’ve been on call my whole career. I’ve carried a laptop almost every day. There are many dinners, parties, and events I’ve had to skip, and there are people in my personal life who have suffered as a result.

I really appreciate these people; they love and care about me, and they know that I have a passion for this, so they accept me for who I am. I think this is old person talk, but everyone has to decide how much they want to really drive their career because there’s a trade-off. Absolutely. I made a tremendous amount of sacrifices in my career, and I really prioritized building as probably the number one.

[1:53:21] And I really appreciate these people. They love and care about me, and they know that I have a passion for this, so they accept me for who I am. I think this is old person talk, but yeah, I think everyone has to decide how much they want to really drive their career because there’s a trade-off. Absolutely. I made a tremendous amount of sacrifices in my career, and I really prioritized building as probably the number one.

I mean values first and then building. But if I went back in time, I would have had a good life, but I probably would have done more vacations and just had a bit more of a balanced life. I really do think. I see that with all the 996 stuff and this kind of performative photos of being in a bar with a laptop, and I’m like that’s not real.

[1:53:52] I mean values first and then building. But if I went back in time, I would have had a good life, but I probably would have done more vacations and just had a bit more of a balanced life. I really do think—I mean I see that with all the 996 stuff and this kind of performative photos of being in a bar with a laptop and stuff, and I’m like that’s not real.

[1:54:16] That’s not real. I mean, sure, I was working more than I do now back then, but I probably still work more than I did then. But I don’t do it as a checkbox; I do it because I really want to be doing stuff. I would caution people against that hustle culture. That’s not, firstly, like, you’re only young once; you should have fun. You know, I’ve got quite a few grays in here.

[1:54:46] But that’s acting. I mean, focus on solving problems, work hard, be passionate. Yeah.

Ryan:

[1:54:55] When I studied your career, there are mentions of you working 16 hours a week in various places. But you like being on call, you like fires, you like ownership; all those things are a recipe for working obscene hours.

James:

[1:55:13] Yeah. I go home and I’m tired, and I wind down by building stuff. I’m lucky enough to have a little workshop at home, so I go home and make things with my hands. You become an infra person; you become an engineer in all aspects of your life. But yeah, I would just say the cool thing is doing cool stuff that humans use.

[1:55:41] The cool thing is not working long hours, the cool thing is not showing off that you were running a coding agent all night. Who cares? The cool stuff is enjoying doing important things.

1:55:54 — Top technical book recommendation

Ryan:

[1:55:54] Do you have a best technical book recommendation for people?

James:

[1:56:00] I have to be honest, I haven’t read almost any technical book. I was in academia for a long time, so I read a lot of papers. Learning is awesome, reading is great, but balance it, right? Read something and then go put it into practice. Most of my career has been about doing. Sure, I did a PhD, so I guess that is like the academic side. But after that, most of my learning has been by doing because there’s nothing like being faced with a real problem to really develop as an engineer.

1:56:36 — Younger self & permanent underclass advice

Ryan:

[1:56:36] Yeah, I think if I answered the question, I’d probably say the same. I think learning by doing is really where it matters most. And then the last question for you is, if you could go back to the beginning of your career and give yourself some advice, what would you say?

James:

[1:56:51] I’d say, it’ll be okay. Don’t sweat the small stuff as much. It’s hard because my whole engineering brand is about caring about details, and I love design, so being obsessive is a little bit part of my DNA. But I think I would go back and say careers are long. I mean, there’s just any story anyone reads about a 22-year-old billionaire, blah, blah, blah. Just ignore that story. That’s not real. That’s not repeatable. That’s not normal. And it’s not that healthy. And it’s not that good for the people either. You probably won’t, ideally, max out your growth for 20 plus years as an engineer. I’m still learning all the time. I’ve been an engineer for several decades. So my advice would be don’t sweat it. There’s time to grow. And I do think that, again, this is a little bit of a modern phenomenon, but there is a feeling right now, oh my God, AGI has come and better max out my growth in the next three months.

[1:57:26] That’s not real. That’s not repeatable. That’s not normal. And it’s not that healthy. And it’s not that good for the people either. You probably won’t, ideally, max out your growth for 20 plus years as an engineer. I’m still learning all the time. I’ve been an engineer for several decades. So my advice would be don’t sweat it. There’s time to grow. And I do think that, again, this is a little bit of a modern phenomenon, but there is a feeling right now, oh my God, AGI has come and better max out my growth in the next three months.

Well, guess what? You ain’t going to do it. It’s not going to happen. You can’t max your growth out in the next three months. It won’t happen. You don’t have to be running 17 agents at the same time. All you gotta do is orient your career around learning every day, getting better at what you’re doing, and trying to solve things in the most simple ways.

[1:58:04] Well, guess what? You ain’t going to do it. It’s not going to happen. You can’t do it. You can’t max your growth out in the next three months. It won’t happen. You don’t have to be running 17 agents at the same time. All you gotta do is orient your career around learning every day, getting better at what you’re doing, and trying to solve things in the most simple ways.

Ryan:

[1:58:26] I mean, on Twitter, I see this take on the idea of a permanent underclass, where if you don’t make it in time for AGI, then you’re going to be part of this permanent underclass.

James:

[1:58:39] Yeah. I mean, look, it’s a bit challenging economic times for a lot of folks. I don’t want to be unsympathetic to people who are having financial difficulties. At the same time, it’s just not an instructive attitude. There’s not much you can do with that information other than feel bad about yourself, and I don’t. Someone will argue back.

[1:59:00] No, what you can do is get really good at using Claude. Well, guess what? It’s not very hard to use Claude. Sometimes people tell me, “Oh my God, I’m finding it hard to keep up with all the new models.” The new model drops, and I don’t even know what the new models are. I use them, but I forget the latest model because it doesn’t matter. Right? Like there was this thing called Ralph.

[1:59:23] Right. I guess it’s still RALPH. It’s like a loop thing. I don’t really know what RALPH is, and I haven’t heard anyone mention RALPH in the past few weeks, but it was the biggest thing on Twitter for like a month. And it just doesn’t. This is noise somehow. Sometimes I feel like it’s like tech tabloids. It’s like people think that they’re learning by somehow knowing, like listening to what Jensen said today or like, oh my God, Boris said that Claude writes itself.

[1:59:53] I don’t think people—look, that’s like—it’s just like reading about Beyoncé, but you’re a nerd, and so you’re reading about Jensen, right? But it doesn’t matter, you know? Growing—like, you don’t have to know. It’s okay. The new coding agent could come out, and you could miss it. And then next year, if it turns out it’s the big one, you’ll just use it. It’s not hard. I haven’t seen any skill so far.

I guess there’s some skill, but it’s not a hard skill. If you’re good at engineering, you can figure out how to use Claude code, right? Or open code. So just watch out for tech tabloidism. It doesn’t matter. Just be building stuff. Just do real work.

[2:00:19] I guess there’s some skill, but it’s not a hard skill. If you’re good at engineering, you can figure out how to use Claude code or open code. Just watch out for tech tabloidism. It doesn’t matter. Just be building stuff. Just do real work.

Ryan:

[2:00:37] I love that mindset. Yeah, well, thank you for your time.

James:

[2:00:42] I’m on Twitter, too. I’m part of the thing, but just ignore me.

Ryan:

[2:00:51] Oh, God. All right, well, thank you so much for your time, James. I really appreciate it. This was a lot of fun.

James:

[2:00:56] Ryan, it was great. Thank you.

In addition to language design questions, I asked him for his personal anecdotes since I felt only he could tell those stories. Below are my top three takeaways to save you time.

You can find the full conversation on YouTube, Spotify, Apple Podcasts. The transcript is on Substack if you prefer to skim.

Brought to you by:

Cursor3: a unified workspace for building software with agents. I used it to build tools for the podcast and shared the process in this episode’s ad read.

WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code

Takeaways from the conversation:

1) How Bell Labs did research - When I asked Bjarne about how project selection went, he talked about two ways of doing research.

In one way, you have a well-designed project that is carefully chosen by management. Then the project is staffed with many researchers. This was not the Bell Labs way.

The other way was to hire the best people on the planet and don’t tell them what to do. Each year, the researchers would be asked to explain what they did in 9pt font or larger on a single page. If you can’t explain what you’re doing concisely, it probably isn’t that interesting.

If the work was interesting enough, then you could continue to work on it. Even though this was unusual, the results speak for themselves as much of the technology we take for granted today was invented at Bell Labs.

2) Negative overhead abstraction - Coming into the conversation, I assumed that more abstraction meant you’d lose performance but Bjarne corrected my naive assumption.

A simple example is to look at C vs C++. Because C++ encodes more information in the language itself, the compiler has more it can use to optimize the final result. Therefore, the abstractions in C++ can be used to get better performance actually.

3) There was a garbage collection API in C++ - I had always assumed that where there was C++, there was manual memory management. I was wrong.

In 1995, Bjarne introduced a standard interface for garbage collection due to user requests. However, C++ developers tended to prefer manual resource management so the garbage collection API never got much use. Later, this garbage collection interface was removed from the standard. This is probably why you don’t hear too much about this.

By the way, thanks for the input last week on where to place ads. It looks like most people prefer the ads grouped together instead of multiple shorter breaks. I would have picked the same personally. I factored in that feedback for this episode.

Looking at other podcasts, is it seems like some creators like to lump their ads together at the beginning instead of in the middle of the episode. I can see arguments for both sides:

1. Beginning → get the interruptions out of the way so the episode can be interruption free

2. Middle → reading ads before delivering value could make people bounce

Curious your thoughts on what you’d prefer:

Thanks for reading,
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

0:50 - The origin of C++

8:46 - What Bell Labs was like

17:24 - Dennis Ritchie

24:00 - When to build a programming language

31:59 - Bootstrapping a language

33:58 - C++ is not object-oriented

37:32 - Discussing type systems

46:20 - Memory safety

49:26 - Standards committee anecdotes

1:09:40 - Adding automatic garbage collection to C++

1:18:25 - Template instantiation is Turing complete

1:21:57 - Abstraction and performance

1:28:51 - AI writing code

1:35:54 - His motivation

1:39:18 - Famous quotes

1:46:48 - Reflecting on building C++

1:49:12 - Top C++ book recommendation

1:50:59 - Advice for his younger self

Transcript

0:50 — The origin of C++

Ryan:

[0:50] What is the origin story behind C++?

Bjarne:

[0:54] Well, let’s start from the real beginning. I got a job at Bell Labs, which is a really great place in New Jersey. It’s not like that anymore, but at the time it was the best applied math, applied engineering place in the world. I looked around at the great people who were there; they built Unix. They did a lot of the theory behind it, and I realized I had to do something important; otherwise, it didn’t belong.

[1:30] So I decided I was going to build a distributed Unix because it was clear that computers were getting better, networking was getting better, so we needed one of those. If I had succeeded, we would have had Unix clusters ten years earlier or something like that. But of course, I couldn’t do it. That’s not a one-person job. The first thing I realized was there wasn’t a language in the world that could do what I needed.

[2:05] It needed two things: low-level access to hardware, such as memory managers, process implementations, process schedulers, network drivers, and device drivers. Then it needed high-level features that indicate, well, there’s a module here on this computer and there’s a module there on that computer, and here’s the communication protocol they’re using, things like that. There are lots of languages that could do either, but none that could do both.

[2:43] The obvious language for the low-level stuff was C because Dennis Ritchie and Brian Kernighan were down the hall, and I said distributed Unix because I was in the home where Unix was invented and still being built. For the high-level languages, there were a fair number, but they were all too slow and they couldn’t manipulate hardware. But I learned to use Simula. I knew Kristen Nygaard and Ole-Johan Dahl, who invented object-oriented programming and Simula.

[3:22] And so I decided I had to merge these two. The way that was practical was to take the class concept from Simula and stick it into C so that it could run much faster and be used for systems programming. At the same time, I made the type system a bit more regular. User-defined types and classes were handled the same way as built-in types. That’s basically the start of what neither C nor Simula could do, which gets us to generic programming.

[4:03] Eventually, many years later, I had to add overloading. We have always had overloading. You can add two integers, you can add two floating-point numbers, you can add a floating-point number to an integer with a plus. That’s a single name, right? So I had to generalize that to be able to have a unique set of rules for both built-in and user-defined types. That’s where it came from.

Ryan:

[4:36] In one of the lectures that I saw that you gave, you talked about rewriting a simulator in BCPL. Is that the distributed Unix work?

Bjarne:

[4:46] No, that’s before that. I went to Cambridge, England, to get a PhD, and at some point, I decided I needed a simulator of software on a distributed system to do the PhD work on distributed systems. Of course, the idea of distributed Unix three or four years later came out of the same way of thinking. What I did was I wrote a really nice simulator in Simula. Simula is very good at that. It’s misnamed because it was a general-purpose programming language, and its bad name didn’t help it at all.

[5:37] But anyway, I wrote this simulator and I wrote little examples, test cases, etc. It all worked nicely. Then I tried the first real run full scale and I took the department’s mainframe and used it for a very significant time. Well, PhD students can’t do that. The chemists and the astrophysicists would never accept it. So I was kicked off the machine, and it was clear that Simula.

[6:19] I could write the program in Simula very well, but I couldn’t afford to run it. So I took the ideas and moved them to a little-used experimental computer, which was the CAP computer, which had hardware protection and capabilities and great stuff for hardware. It was somewhat unusual, so the astrophysicists couldn’t use it. They weren’t computer scientists as such, but I could. The only problem was I couldn’t run Simula there because Simula was never ported to that kind of machine; it was being used on mainframes and it was proprietary, and everything was wrong in the context of the CAP computer.

So I basically rewrote my simulator in BCPL. BCPL is a language that will make C look like a high-level language and has only one data type, the word. It was a very painful exercise, but once I’d done it, my program ran, I guesstimated about 50 times faster. I got my data and I got my PhD. So that was good. But I was convinced I would never again attempt a problem with tools that inadequate as I had tried on the mainframe in Cambridge.

[7:13] So I basically rewrote my simulator in BCPL. BCPL is a language that will make C look like a high-level language and has only one data type, the word. It was a very painful exercise, but once I’d done it, my program ran, I guesstimated about 50 times faster. I got my data and I got my PhD. So that was good. But I was convinced I would never again attempt a problem with tools that inadequate as I had tried it on the mainframe in Cambridge.

[8:07] And so I had a list of things that my ideal language should have, and, well, C didn’t have all of that, but it came closer than any other language that existed. C++ came out of there.

Ryan:

[8:25] Yeah, in that lecture you said something like that. Writing that program in BCPL was so difficult you lost half your hair debugging.

Bjarne:

[8:35] That’s almost exactly true. And I lost the other half getting C++ going over the years. But anyway, it worked.

8:46 — What Bell Labs was like

Ryan:

[8:46] You mentioned Bell Labs, and I think there’s a lot of curiosity about that topic just because it’s such a legendary place. When you graduated from your PhD and were thinking about where to work, what was Bell Labs known as at that time?

Bjarne:

[9:04] Bell Labs was the place to go if you wanted to do practical engineering at a large scale, at sort of world class. I think it was easily the best. We probably had twice as many computer scientists as MIT at the time, things like that. The Computer Science Research Center had great people, and some of them had come from Cambridge. One day, in my last year in Cambridge, one of the people from Bell Labs came along to give a talk.

[9:54] And the tradition in England and in the computer lab is, after a day’s work, you go to the pub and you chat with other people to see what has been going on. He said, “Well, when you need a job, give us a buzz.” So I did, and I flew over to New Jersey on my own tab, actually. My later boss, Sandy Fraser, a great guy working with networking, told me that I’d come at a wrong time. They didn’t have any jobs. This is not what you want to hear when you’ve just flown over the Atlantic. Anyway, the next day I gave a talk to a development group, not the research group, and then they changed their minds and took me up to the research group, and I worked there for the next couple of decades.

[10:34] This is not what you want to hear when you’ve just flown over the Atlantic. Anyway, the next day I gave a talk to a development group, not the research group, and then they changed their minds and took me up to the research group. I worked there for the next couple of decades.

Ryan:

[10:53] What was the interview process like?

Bjarne:

[10:56] You just talk to some people. I remember having a long chat with Dennis Ritchie, for instance, and I talked to people doing networking mostly. There wasn’t an interview process as such. They hadn’t actually hired anybody new for five years. So no, they just did it by the seat of the pants.

Ryan:

[11:27] So it’s kind of like the belief and credibility that other people say that you have. Like Dennis Ritchie talked to you and he knew that you knew what you were talking about?

Bjarne:

[11:38] Sandy Fraser and such. They just talk to you, see what you know and don’t know. At the end, they go to the director and say, in this case, we’ve got a good guy; can you let us have him? I, of course, didn’t know anything about that. I wasn’t there. I was out talking to somebody in California, and I get a phone call from the director saying, would you like to come and work here a week later?

Ryan:

[12:10] What gave you the conviction to fly on your own tab to go? And there wasn’t even a promise of a job yet?

Bjarne:

[12:17] No. Well, it was the best place in the world, right? Do you need any more? If it worked, it was the best. And if it didn’t work, so what? You can’t succeed at everything.

Ryan:

[12:35] Today, there are more industries that have a stronger pull than at that time. Would it have been IBM or something? That would have been the best industry.

Bjarne:

[12:43] I talked to IBM; they weren’t as good as the Bell Labs Computer Science Research Center. I was up at Yorktown Heights, and I talked to the researchers and the young researchers. I just didn’t think they were doing the right stuff and not in the right way. They were much more controlled and directed than the researchers at Bell Labs.

Ryan:

[13:12] At a place like Bell Labs, how does project selection go?

Bjarne:

[13:19] I think still there are two philosophies about how to get good research. The one is that you have a well designed project chosen carefully by management and higher management, seriously funded and maybe you do put 20 or 30 people at the problem and you solve it and you have something great. The other philosophy is you hire the best people you can find and don’t tell them what to do.

[14:01] My job was described as do something interesting in a year’s time, tell us what you did, and if we like it, we’ll extend. We will give you the same deal next year. By the way, the way you tell us is you write one sheet of paper using more than a nine-point font or more. Because if you can’t say what you did fairly briefly, you probably haven’t done something interesting enough, very unusual. So there they were actually worrying when they built Unix because it eventually involved five or seven people, and it was getting too big for that model of the world of individuals doing interesting things.

[15:03] Very different. I would say that on average, this fairly anarchic organization did better than the well-organized one. Most of the things you’ve heard of from Bell Labs came out of there. In another part of the building, they were doing hardware things. So fibers, as we use them today, came out of there. A lot of the wireless technology came out of there. The charge-coupled devices that are our cameras came out of there.

[15:43] They tried to do videophones and couldn’t get it to work because hardware hadn’t grown up to it, but they were trying to do it. The system of cells for cell phones came out of not that building, but another building. For Bell Labs, it was just a great place. The computer science people tended to talk to people doing other things. I remember when I was doing simulations, I was helping somebody build a simulator for some networking stuff.

[16:18] A lot of early C++ had to do with doing things like what happens when a network get overloaded, how do we handle the overload protocols. And in this particular case, they did a good job. And they called me back and they had a slightly bigger problem. They wanted to simulate the computer traffic of Manhattan. Even then, my answer was no, we don’t have the compute power to do that.

Ryan:

[17:02] Back then, though.

Bjarne:

[17:04] Now they probably could, but of course the computer traffic has become much more. So maybe they can’t; I don’t know. I don’t have the numbers now. Then they gave me the numbers. That was why I declined to help them because it was impossible.

17:24 — Dennis Ritchie

Ryan:

[17:24] I saw somewhere when I was doing research that you said you had gotten lunch with Dennis Ritchie once a week for like 16 years or something like that. And you know, he’s also a very legendary name. And I was curious, you know, if there’s anything that you, you learned from him or anything that impressed you about him that maybe influenced you or C++.

Bjarne:

[17:49] He was a great guy and we talked about a lot of things. He never said anything rude or negative about C++. Actually, in his Hubble paper, he points to C++ as the obvious successor to C. So all of these C versus C++ language wars are ridiculous. They should never have happened, and they certainly didn’t happen because, well, I knew Dennis. We were not fighting. I still know Brian Kernighan. I was talking to him this Friday.

[18:33] We’re good friends. And yeah, language wars are silly. Dennis helped me design Const, for instance, for C++. It used to be called Read Only and Write Only, but the C guys couldn’t handle two words and they were too long. So we got what we got. But that’s one specific thing I remember Dennis being helpful with. He was a bit worried. Worried about overloading because you had to look at the declarations of functions before you knew what the meaning of a call was.

[19:21] But that’s a very reasonable way of thinking. It just happens that it works. Anybody who writes see today are using my handiwork essentially all of the time, because the modern syntax for function definitions and function declarations and the call semantics came out of the early C++, early serial classes work of mine. So when people start ranting, they should remember that they’re actually using my handiwork every day.

Ryan:

[20:04] You had written these almost like historical accountings of the history of C++, like maybe three really long papers. I think for some conference I forgot the exact. And so in there, there was one anecdote about Dennis Ritchie. It was about this concept where he proposed to the C Standard committee this idea of a fat pointer where it also has. It stores its size as well. You mentioned the C Committee didn’t approve.

Bjarne:

[20:43] Dennis did see, but he didn’t take part in the Standards Committee. I’ve even heard people from the C Standards Committee say, no, Dennis isn’t a C expert. He’s never come to meetings. Very strange attitude. But anyway, we knew the problem about buffer overflow and range errors. The obvious solution is to use what Dennis called a fat pointer, which is a pointer with its number of elements it points to attached to it.

[21:24] But that’s two words. And that’s probably— no, that is why it wasn’t used in early C++, because then they had 48 kilobytes of memory. When I had this discussion with Dennis, I think we had a whole megabyte. When I started with C++, we had 256 kilobytes. I knew that we were going to get a megabyte. We were talking about it, and he called them fat pointers. Today in C++, they’re called span.

[22:04] And the span came out of my work with others on the C++ core guidelines. We needed something like that. We couldn’t provide the degree of control and safety that we needed. So we built span, and it came into the standard a bit later. But some of these ideas are very old.

Ryan:

[22:30] When you put together really impressive people, like the world’s greatest people, you look around and see how great the others are. Even though each individual is great, just the greatness of others can give people this feeling of imposter syndrome that’s not necessarily founded. Is that something that you ever felt or saw at Bell Labs?

Bjarne:

[22:52] Definitely, maybe I still have a bit of it, but certainly when I came to Bell Labs and saw the names on the doors and I had read the papers, they had created the fields I like to work in. Yeah, I thought I have to up my game. I have to do something bigger and better than what I had imagined. Also, you talk to them and you learn things. I learned a lot over lunch where there were a bunch of them talking about what they’re doing and why they’re doing it.

[23:30] There are some places that have that effect on people and have the density of talent. Cambridge University was one of those places. I learned a lot there. The doors were always open. It was almost a policy that we kept the doors open because of Howells.

24:00 — When to build a programming language

Ryan:

[24:00] And when we talk about a programming language and just generally about language design, if I wanted to build a programming language today, what are all the pieces that I’d need to create to make a programming language?

Bjarne:

[24:17] Well, that’s a relatively easy question to answer. Everybody asks that question, and I think it’s the wrong question. What you need is a problem that needs a solution. A lot of people just want to build a language that is better at what they are doing now and what they particularly are doing. Most of the time, that can be done reasonably well with existing languages. If you build a very specialized language, that’s fine.

But if we’re talking about more general-purpose languages, you’re then building something that, when you want to work with somebody else, it’s not ideal for them. I’m sometimes asked why C++ is so big and complicated. There are two reasons. One is history. I could not build the C++ I wanted back in the 80s for a variety of reasons—partly technology, partly computers, and partly because I didn’t know enough and I had to learn.

[24:55] But if we’re talking about more general-purpose languages, you’re then building something that, when you want to work with somebody else, it’s not ideal for them. I’m sometimes asked why C++ is so big and complicated. There are two reasons. One is history. I could not build the C++ I wanted back in the 80s for a variety of reasons—partly technology, partly computers, and partly because I didn’t know enough and I had to learn.

[25:37] So you do the standard engineering thing, you do the best you can, then you see what works and what doesn’t work and try to fix the problems. Then you repeat. That’s how C++ grew. There are some leftover things that just get in people’s way now. You have spans, you very rarely use pointers, and you should certainly not use pointers for resource handling. That was already built in with your classes in ‘79, but people didn’t get it, so there’s teaching to do.

[26:19] But anyway, once you figure out that you have a problem that requires a new language, then you start looking at what there is, and you have lots of help, lots of books about analysis and about code generation. You have frameworks like LLVM that most of the modern languages use to generate decent code. So most of the languages that compete with C++ do it by using a C infrastructure. It’s highly amusing.

[27:02] But anyway, focus on the problem and don’t think you’re the only user. If you think you’re the only user, you build a special-purpose programming language, and that’s fine. Domain-specific languages are great when you find the right solution to the right problem, but identify the problem first. In my case, the problem was I needed high- and low-level facilities in the same language. Otherwise, I had to use two languages, and I had to have them communicate properly.

[27:41] High-level languages at the time tended to use interfaces that took away performance. Quite often, they required garbage collection, which is not very good for device drivers, for instance, or for building garbage collectors. So yes, identify the problem and try to solve it.

Ryan:

[28:06] Back when you were creating C++ and you had identified the problem, you went off to build C++, and there were all these pieces, right? There’s the compiler, there’s a linker, and in the implementations of those, there’s a parser and all those things. When you were building the original thing, what was the most technically challenging part to implement?

Bjarne:

[28:33] I don’t think any part was particularly challenging. It was more those many parts, as you point out. One of the things I decided that caused trouble later was that I wasn’t going to touch the linker. It came simply because I asked around and realized people were using about 25 different linkers at Bell Labs, just in Bell Labs. If I wanted to serve my obvious initial uses, I would have to write interfaces or modifications to 25 linkers.

And nobody wants you to touch their linker because if you make a mistake, everything breaks. So I decided on a rule: don’t mess with a linker. Later, people have messed with the linkers and made them better for C++. But that was after C became a major issue. The other thing was that there were many different optimizers. Every computer from different sources had a different optimizer, and again, I couldn’t write a dozen optimizers.

[29:14] And nobody wants you to touch their linker because if you make a mistake, everything breaks. So I decided on a rule: don’t mess with a linker. Later, people have messed with the linkers and made them better for C++. But that was after C became a major issue. The other thing was that there were many different optimizers. Every computer from different sources had a different optimizer, and again, I couldn’t write a dozen optimizers.

[29:56] I mean, I’m going to write a language here, right? And I can’t write. If I wanted to be an optimizer specialist for deck computers, I could become that. I have the background, I have the training, but that wasn’t what I wanted to do. I wanted to build first a distributed system, and then when my friends and colleagues started using Simula classes, I wanted to help them. They were doing things like network simulations, hardware layout, positioning of satellites, all kinds of interesting stuff.

[30:39] So that was worth doing. I decided that actually there was a common interface to all these optimizers and code generators. It’s called C. So let’s use C as the assembler, and that worked nicely. C was very good at the low level, part of the reason I chose it. So let’s use it for the low level. I could have hidden C and recreated probably a better interface, but I decided that I’ll just use C and have C compatibility.

[31:21] At the time, what I said was that we can have Dennis Ritchie‘s mistakes, which we know, and we can have my mistakes, which we don’t know yet, so we’ll take Dennis’s. That’s much more manageable and understandable, and I don’t have to teach people how to write a for loop and things like that. So that’s how C compatibility came in, partly as an implementation technique and partly to get into the culture and tool support and such.

31:59 — Bootstrapping a language

Ryan:

[31:59] I saw somewhere in my research that C++ was used to write some part of the language toolchain, and immediately I had this thought of a chicken and egg problem. How do you use this language to build something that it is using itself? How does that work?

Bjarne:

[32:23] This is bootstrapping, and it was not an unusual thing. So I started with C++. I wrote a preprocessor that did some of the fundamental things in what became C++ classes, including fairly simple inheritance and overloading. Then, in that, I wrote a simple compiler for a subset of C++. Now I can use operator overloading and overloading in general classes, so I can build a scope class that handles lookup and naming and things like that.

[33:22] And then you work from there, just writing the next version in the previous version. You keep going, and after a couple of years, you have something that became known to the world as C++. I wrote a book about it, and a compiler came out in the world. But I didn’t invent this technique. This was known as bootstrapping. I think I was taught it as an undergrad that you could do things like that.

33:58 — C++ is not object-oriented

Ryan:

[33:58] Most people look at C++ and think that’s an object-oriented language. I’ve heard you say multiple times that that’s not the case or that’s not your immediate thought. Why is that?

Bjarne:

[34:10] Yeah, I never called it an object-oriented programming language. If you look at the C++ programming language, the first edition, the closest I come is to say some people call these techniques object-based. Actually, it’s more focused on classes; it’s type-oriented, class-oriented. It actually supports the techniques of object orientation very well. In particular, it follows Simula‘s model of defining types, defining classes, and defining class hierarchies to handle groups of related classes.

[35:00] But that was never all it was. For instance, I do not want object-oriented complex numbers. I don’t want to say 2 dot something to get to some parts of numbers. I really want to say two plus Z, and I want that to end up being roughly the same as Z plus 2—no dots, no arrows. Math has developed a notation over the last 300 years or so. Descartes was, I think, the first one to use this notation, and it is very good.

[35:54] I didn’t want everything to be object-oriented. Furthermore, I wanted things that did not require inheritance, that did not require runtime resolution, not to use it. So for arithmetic and for complex numbers and such, I wanted Fortran compatibility. I was rather keen on what’s called reuse in those days. But I saw it slightly different from a lot of researchers. A lot of researchers wanted to build a language, a system that allowed reuse.

[36:38] I wanted to reuse things that existed. Fortran was there with some great software, C was there with some great system software, and actually helped with compilers and such. There was a Simula that was used a fair bit too. So I wanted to reuse that, and I wanted to make sure that worked. And that meant I couldn’t go too far away from the hardware. I couldn’t build all of the things that were considered ideal, or even the things I would consider ideal.

[37:19] This is the real world; this is the real set of problems you are attacking. You have to respect the constraints that come with that view of what you’re doing.

37:32 — Discussing type systems

Ryan:

[37:32] At the time that you wrote C++, C was already there and it had a weaker type system than what C++ eventually had. Just give your thoughts on the trade-offs behind that. Why did you choose to make the typing system stronger in C++?

Bjarne:

[37:47] Because we needed it. The weakness in the type system is one of the most obvious sources of errors, and it’s certainly one of the sources of endless testing and debugging. I hate debugging. I would much rather do design. You can’t really have either design or debugging. Some people claim they can, but they can’t. So I want to move the arrow towards more design that helps the debugging and makes fewer mistakes at runtime.

[38:33] And the type system is one of them. And actually what you get and see today to a large extent is much stronger. Well, no, it is a much stronger type than it was in those days, and partly because of C++. Also, there are things you can’t express unless you have a strong type system. I mentioned overloading before. Overloading is essential for generic programming. And if you want to write, say, a vector of T where T is a parameter type, you have to have overloading because you can only operate on T’s, providing all the T’s have the same interface for what you need.

[39:17] So you need the type system to resolve those things, and that can be resolved at compile time. The compiler gets a bit more complicated, probably a bit slower, but you don’t need to do so much debugging. There was a large-scale experiment done at Bell Labs in Chicago where they had some groups using C switching to C++, and they wanted to know whether they were more or less productive. Some people claimed that the slower compilation slowed them down.

[40:04] Somebody simply measured how much compile time was used before and after switching to C++, and they found that the amount of compilation time on compute power was roughly identical. That is, C++ was slower by about a factor of two at that time. But the C++ people compiled twice as often. This is just one experiment. The factor of two is just one experiment, but I wanted to move towards using more compile time resolution.

[40:51] Still doing that.

Ryan:

[40:53] I mean, for every language there’s this dichotomy of having it being statically typed versus dynamically typed. And C++ is one of the most famous statically typed languages. Why did you choose a statically typed language?

Bjarne:

[41:09] Because of the problems I wanted to attack. What do you do when you get a runtime error in something like Smalltalk? You go into the debugger, and that makes a lot of sense. If there’s a programmer sitting at a screen getting the error, it doesn’t make any sense. If a telephone switch finds a runtime error, then you have to resolve it. Furthermore, you want performance, and you want small programs to fit into memories.

[41:48] This is true even today because I think 99% of all computers are embedded systems, and they tend to be memory constrained. Again, if you do runtime resolution, you need to have enough information, enough data to do the runtime resolution. I wanted to fit into small memories—small meaning 120k, 250k, 1 megabyte, things like that. I think it’s still relevant for many systems.

[42:32] You can build a camera like that; it can still have several megabytes of memory. But if you put in a lot of memory, it gets bigger, costs more, and the battery runs out quicker. So we don’t do that. Phones, cameras, and things like that are still memory constrained. Statically typed languages, languages optimized for memory consumption, are just better at that. That’s why we use it; we’re using it right now.

[43:10] I suspect that the microphones have chips in them too. There’s a lot of C++ in that world. The composition there is C and assembler.

Ryan:

[43:24] And you mentioned the research that was done on the compile time on

Bjarne:

[43:32] If you catch things earlier, you compile less often, but maybe it takes longer in this case. I could see a similar analogy where you catch errors way earlier if you have a statically typed language because the compiler is yelling at you before you put together that final thing. Whereas in a dynamically typed language, the errors may come later.

Ryan:

[43:32] You catch things earlier, you compile less often, but maybe it takes longer in this case. I could see a similar analogy where you catch errors way earlier if you have a statically typed language because the compiler is yelling at you before you put together that final thing. Whereas in a dynamically typed language, the errors may come later.

Bjarne:

[44:00] I don’t know any solid research on that, but you can look at it. JavaScript and Python are very popular, and they are runtime checked and they run much slower. I mean, raw Python runs something like 70 times slower than raw C++. The reason it’s viable is that a lot of key Python libraries are written in C or C++ to get the performance. So you get the performance by actually getting to the point that I was starting out with.

[44:43] You need high-level stuff, and you need the thing that can manipulate hardware. Here they are using two languages, but still the same fundamental needs. It’s easier to try out things in a dynamically typed language because you don’t have to know enough about the language; you don’t have to know about type systems. Your average web developer or astrophysicist is not a computer scientist and doesn’t want to become one.

[45:20] So there are advantages there. But the problem is that errors found by the type system in a statically typed language are discovered at runtime later. As systems grow, performance problems start. Furthermore, it becomes harder to write reliable software. You need much more unit testing, for instance, in a dynamic language because the compiler doesn’t do it for you. If you want things to guarantee to work—like the telephone switch mustn’t crash, your car mustn’t crash, your plane mustn’t crash—you want guarantees, and they’re harder to provide in a very flexible dynamic type system.

46:20 — Memory safety

Ryan:

[46:20] One thing that I think C++ is infamous for is memory safety issues or foot guns that exist there.

Bjarne:

[46:31] I’m so tired of that. I haven’t had those problems for years. Somebody did a study of the obvious problems with buffer overflows and people hacking in using that kind of stuff. Almost all of these cases involve people writing C-style code or in C. Herb Sutter has a talk with actual numbers, and they are quite significant. It’s sort of that kind of problem. More than 90% are from people who don’t write modern C++.

They use raw pointers to pass things around without the number of elements. No fat pointers, no spans. You have them in C++; you can use them, you can use vectors. We have hardened libraries; everybody has hardened libraries that do the runtime checking. Apple has it, Google has it, Microsoft has it. It’s just not standard until now. C26 has a hardened option that is standard. The work I’m doing on profiles will give you a way of guaranteeing that you don’t do the stupid things.

[47:24] They use raw pointers to pass things around without the number of elements. No fat pointers, no spans. You have them in C++; you can use them, you can use vectors. We have hardened libraries; everybody has hardened libraries that do the runtime checking. Apple has it, Google has it, Microsoft has it. It’s just not standard until now. C26 has a hardened option that is standard. The work I’m doing on profiles will give you a way of guaranteeing that you don’t do the stupid things.

[48:12] So anyway, fundamentally, theoretically, the problem was solved many years ago, and people just do what they’ve always done and get the problems they’ve always had. That makes me sad. It’s one of the things that makes me work on coding guidelines, enforced profiles, and education.

Ryan:

[48:40] I mean, education is one way to solve the problem. Is there a way to get the compiler to just prevent people from doing all those risky things? And is that enabled by default in modern C++ today?

Bjarne:

[48:53] No, but it should be. I’m proposing that for C++29, the simpler versions of that should have been in C++26. But there are still a lot of people, even in the C++ Standards Committee, that are very devoted to their old code and their old ways of doing things. There are people who say you should only standardize what is common in industry, but when the bugs are common in industry, you should do something else.

49:26 — Standards committee anecdotes

Ryan:

[49:26] The Standards Committee is a topic I want to talk about. It’s interesting. I mean, the language is now run by a democracy. One question I wanted to ask you is if it was a dictatorship, and you had full say over what language features would be included, would that make it harder to get by?

Bjarne:

[49:48] First of all, it never was a dictatorship. I never had full control. Once you have some users, in my opinion, you gain some responsibility for making sure that they’re helped and their stuff works. You can’t keep breaking the language. That’s what academic language development does. They break to improve all the time, and then they can’t maintain a user population. I didn’t actually choose to have a standards committee.

[50:23] I chose responsibility to the community. But one day, two guys came in representing IBM and HP, and I can’t remember if it was Sun or DEC that was the third thing they represented. But anyway, the biggest computer and software suppliers in the world at the time came into my office in SH9 and said, “Well, Bjarne, do you want to help us standardize C++ under ISO rules?” I said, “No, I can’t do that.”

[51:10] I’m still doing experiments. It’s still not complete. So they said, “No, Bjarne, you don’t get it. Our organizations cannot use a language that’s not standardized. They cannot use a language that’s owned by a corporation that we might compete with. And we do sometimes, okay, we trust you, of course, but not your employer. We compete with them sometimes. And you can get run over by a boss. No, no, no.”

[51:49] We need a standard, and we need a standards committee. So this goes on for about an hour. They twist my arm. Ow, ow, ow. In the end, they said, okay, I will standardize C++ under ANSI rules just like you suggest. The computer community needs that. By the way, what’s the ANSI rule for standardization? They told me, and we started a year later. But this was the way it came about.

[52:25] Some very important organizations wanted that standardization. C++ was on track to get standardized, and AT&T, being primarily a user of software, was also in favor of standardization. I found out, and so they supported it. The documentation I had written was based on it. Actually, I rewrote the documentation that became the ARM, the Annotated C++ Standards Manual, which provided the definition, the manual of the language, and for every feature, some rationale and some way it could be implemented or was implemented.

[53:16] And that became the foundation document for the standardization.

Ryan:

[53:21] When they were strong-arming you, what if you had just said no? What would have happened?

Bjarne:

[53:26] Well, I think C++ would have faded into becoming an academic cube language that was loved by some small community, and it would have disappeared out of the mainstream of computing. There are people who say this stronger than I do. They say that C is spread and its usage is that it has a standard; it’s not owned by a corporation. It is one of the things that sometimes blocks the wannabe C++ killers.

[54:17] I remember the ads for Java and people standing up saying, “We’ll kill, absolutely kill C++ in two years.” I thought that was rude. Anyway, we have 10, 12 times more C++ developers today than we had when they said it, so it didn’t work.

Ryan:

[54:43] How is it that C++—not exactly that it’s a war—but just if we looked at adoption, clearly C++ gained a lot more adoption than Java. Yet I know Java had the backing of a big company that was putting a lot of marketing dollars in, and C was kind of, I think you’ve said that it had almost zero, next to zero marketing done for it.

Bjarne:

[55:13] Next to zero was $5,000 to be used over three years. Sun used much more money on advertising and marketing Java than was ever used in C++ development. To this day, the standards committee has a problem. It has no funding, and that means it’s hard to do extra experiments; it’s hard to deploy things. Other language communities keep sort of stealing C++ compiler and tool developers because they’re good.

[55:59] But it makes it hard to predict how fast we can implement things today. Last I checked, the C++ standards committee had 527 members. We work on consensus because if you don’t have consensus, then you get dialects. We don’t want to have a feature that’s voted in say 60 to 40 or even worse, 52 to 48. No percent. We don’t do that. And that’s painful and tedious and good.

Ryan:

[56:45] When you say consensus, that 100% need to approve is not necessary. We don’t need unanimity. We need a massive majority. And basically, I would like to see 90%, and we often do 80%. I start to worry.

Bjarne:

[56:47] To approve 100% is not necessary. We don’t need unanimity. We need a massive majority. And basically, I would like to see 90%, and we often do 80%. I start to worry.

Ryan:

[57:08] What’s the lower bound that’s coded into the rules?

Bjarne:

[57:10] There’s no lower bound coded into the rules. The rule says that the convener of the ISO committee determines what is consensus. Pure numbers don’t say it. Could you imagine you had a vote, 95% versus 5%? But the implementers of C++ compilers and standard libraries from Google, Apple, Microsoft, and others were all in the 5%. Is that consensus? I can reassure you that no convener would call that consensus.

Ryan:

[58:02] And that makes sense intuitively. I kind of wonder, with democratic decisions, there need to be objective rules. So what if the convener made the wrong decision?

Bjarne:

[58:13] It happens. But you can’t just have numeric rules. Not everybody cares for the whole language. Not everybody understands what’s going on. You can vote at your third meeting. So you might have somebody with a vote that has eight months of experience with the standardization and doesn’t understand standardization and knows only what they know from their development organization that they have been part of, which might be a small one.

[58:55] You need some judgment, and you hope that the convener has that judgment. The convener always asks the national representatives. I mean, the other way of getting a consensus is that you have a massive consensus, but you have ten countries where the representatives didn’t agree. That’s not consensus. And even when there looks, if there is a look, if everybody is for it, if it’s massive and all of that, there’s not a problem.

[59:35] But if there’s a problem, the convener asks the national body heads, he asks the implementers, sort of key people that are necessary for getting the voted change into real use. Sometimes educators also before they make that decision. Not everybody weighs equally once there’s a disagreement.

Ryan:

[1:01:45] I saw in some of your writing you said one of the most negatively received ideas you’d ever presented was Auto. I know Auto eventually made its way in there, but what’s the story behind why it was so negatively received?

Bjarne:

[1:02:02] At that time, it was just unusual. People thought it was weakening the type system. It also opened the door to fairly general generic programming that is not heavily syntax-based. Auto is the beginning of concepts, which is the ability to put constraints on generic code. Auto is just the simplest constraint; it must be a type as opposed to seven. Maybe I didn’t explain this well enough, and there’s a variety of backgrounds in the committee. Maybe they didn’t know languages of the generic types, such as ML and Haskell.

[1:03:04] So it was a horrible ache. Anyway, we still got it because we needed something like that, but it wasn’t enough. I have looked at industrial software and problems with the overuse of Auto. You should only use Auto when you have an idea about what is needed there. It’s good in generic code where you go and eventually check that the type is correct, that Auto has resolved to something that supports the operations that you’re going to do on it.

[1:03:50] And that’s what concepts formalize. But it was always checked at the end, and I noticed a group of people that was overusing auto, actually in a framework for networking. They were saying that they were being slowed down, not so much with bugs, but they had to look up the functions being called to see what that auto could possibly bind to. And then they had to put in comments that said what the auto was meant.

[1:04:31] So you have auto, and the comment says must be an input channel. Now you simply define input channel, and then instead of saying auto, you say input channel auto. Fine. That’s what the system is. My design of that simply said that auto is the simplest concept, and you should simply only set input channel C++ equals blah blah blah. But anyway, the committee wanted an indicator that this was going on.

Ryan:

[1:05:10] Oh well, I saw another anecdote about the standards committee being heated at times. You mentioned there’s this thing about shuttle diplomacy between two corners of the room. I think it was IBM and Intel. They both needed different support. What’s the story behind that?

Bjarne:

[1:05:30] I was actually talking to Brian McKnight, who was the IBM representative at the time. We were discussing some of the things that were happening then, and it’s still remembered. So basically, IBM was doing the... What’s that architecture called?

Ryan:

[1:05:54] PowerPC. And Intel was doing well. They had different models of the underlying hardware, especially coordination with the caches and things like that. The guy representing Intel wasn’t actually an Intel guy. Anyway, also a good guy. I used some of his slides in my presentations. I knew these guys, but they were totally deadlocked.

Bjarne:

[1:05:55] PowerPC and the x86. The Intel was doing well. They have different models of the underlying hardware, especially coordination with the caches and things like that. The guy representing Intel wasn’t actually an Intel guy. Anyway, also a good guy. I used some of his slides in my presentations. I knew these guys, but they were totally deadlocked.

[1:06:36] I mean, these are massive organizations with massive amounts of code out there. Basically, some things could be done. But the IBM guy, Brian, said that they had a lot of software, mostly in the lowest level, even down in the microcode, that was relying on the way they had done it. The people who had done it had left the company a long time ago, and they just couldn’t rewrite all of that code and get it right, even if the Intel guys were right.

[1:07:38] Okay. And the Intel guys had similar arguments and similar points. This is better. We are using it. I was shuffling; they were in different corners of a large room. So I go up to the Intel guy, the guy representing Intel here, and what’s the problem? Tell me about it. Explain it. I go down, explain. He’s saying this. They say, well, there’s this, this, this. I go back again and I spent a couple of hours literally doing shuttle diplomacy, walking from one corner of the room to the other.

[1:08:18] And we reached an agreement. That’s in C++. A couple of years later, they both agreed that they were now using a combination of what they had before and what the other guys brought in. So actually, the result was improvement, cross-pollination.

Ryan:

[1:08:45] That’s funny. Why did you have to do shuttle diplomacy? Why not just a group conversation?

Bjarne:

[1:08:50] Because they’ve been trying that for days and probably for meetings before that, and it didn’t work. So I guess I was just translating and asking questions. I mean, those were experts. I don’t consider myself an expert at that level. I’ve done hardware, I’ve done microcode, so I’m not an amateur, but these guys are really good. So the IBM guy is now the guy doing most of the synchronization on the Linux.

[1:09:30] We’re still using his stuff today. He has a C++ version of it, so he can get it into the bottom of the Linux kernel.

1:09:40 — Adding automatic garbage collection to C++

Ryan:

[1:09:40] When I was reading your writing in these papers, there’s this part where it seems like in 1995 you had this idea to introduce some form of automatic garbage collection into C++. That kind of surprised me because when I think about C++, one of my immediate thoughts is, no garbage collector. We’re going to manage the memory ourselves. How would that even work?

Bjarne:

[1:10:10] There’s two things there. One, I wanted to automate resource management in general, not just garbage, not just memory. For that, you have constructors, destructors, and the techniques later known as RAII—Resource Acquisition Is Initialization—which is probably my worst naming ever. But I was busy at the time. So in the standards committee, those people insisted that we needed to be able to do garbage collection.

[1:10:50] And there were garbage collectors out there. Oh, there was Hans Boehm here. He was the one representing the Intel model of stuff. He has a conservative garbage collector still used today. We thought we needed an interface so that it could be standardized how you used such a garbage collector. Basically, I was listening to the users, expert users, and they thought it was necessary. I thought that support for memory management and resource management was important.

[1:11:32] I’ve thought that from the beginning. I didn’t think garbage collection was appropriate for a lot of what I was doing. But certainly, automating the management was ideal. After a long set of discussions, we found an interface that people agreed on, and we put it into C++. What we found was that over the next 10 years, the amount of usage of garbage collection decreased as RAII, the resource management that had been there all the time, got more and better understood and was used more.

[1:12:24] Furthermore, the people who still used garbage collectors didn’t use the standard interface because they had figured out ways of doing it better. Today, there are still a few people doing garbage collection, but it’s not part of the standard.

Ryan:

[1:12:43] How does that work? Is it kind of like a wrapper around the memory allocation methods?

Bjarne:

[1:12:48] Yeah, okay. You have a different implementation of new or malloc or operate a new or whatever it is you’re using at your lowest level, and then delete becomes something slightly different too.

Ryan:

[1:13:11] There’s this cautionary tale in the C++ community about this ship, Vasa, and I was kind of curious why that’s popular.

Bjarne:

[1:13:21] Oh, we had a meeting in Stockholm at some point, and they have a wonderful ship that, if you ever get to Stockholm, you should see: the Vasa. It’s a battleship from the 1600s. There’s a story to that, and that’s the one I tell people. The King was ordering a battleship that should be the best and most beautiful battleship around. It was going to be a good fighting battleship and used for diplomatic visits. So it should be beautiful. They laid down the keel and started building it. Then they heard that a likely opponent was building battleships with two gun decks. This was an old-fashioned battleship with only one gun deck. If you put a one gun deck battleship next to a two gun deck battleship, the highly predictable result is a lot of holes in the one gun deck battleship, and it’s gone.

[1:14:05] So it should be beautiful. And they laid down the keel and they started building it. Then they heard that a likely opponent was building battleships with two gun decks. This was an old-fashioned battleship with only one gun deck. If you put a one gun deck battleship next to a two gun deck battleship, the highly predictable result is a lot of holes in the one gun deck battleship, and it’s gone.

[1:14:42] So the King orders that this ship should now have two gun decks. And they’ve already started building it. So they add another gun deck; they add cannons up there. And the King also watched. Now the ship is bigger, they want more statues and beautiful things. So it becomes a bit top-heavy. And rumor has it—I’ve never checked this rumor, so it might be wrong—that the ship designer committed suicide out of horror.

[1:15:23] Also, a thing that I don’t believe is just a rumor was when it was built, they tested it for stability. The way you test a ship like that for stability is you take the whole crew and you run them from one side to the other back and forth, creating harmonic sway. If you can do that 14 times, then it will stand up to the Baltic and the North Sea. Rumor has it—actually, as I said, I think it’s a fact—that they did it seven times and then they stopped because it looked dangerous.

[1:16:08] So this was 1624, I think the ship gets finished. It’s sailing out, the most wonderful ship you’ve ever seen. It’s sailing out in Stockholm harbor, trumpets blaring, flags flying, families of crew on board, the whole thing. It gets halfway across the harbor, a gust of wind comes, it kills a roar, and it’s gone. And it ends down in some place where there’s not much oxygen. So it was well preserved, and they fished it up again, and you can see it.

[1:16:56] And so I tell this story to the standards committee and I point out there’s something they did wrong. They built more features on top without improving the foundation. Always improve the foundation to make sure that it’s not just a random set of features that you have added, because that’s complexity. Furthermore, do not compromise your testing. That’s really dangerous. And furthermore, you’ve all noticed when your high bosses say something should be done, and the high bosses don’t always know what’s right. Sometimes the professional thing is to say, no, we are not doing this.

[1:17:43] We have to take it easy. If they had said, okay, we’ll build a one-gun deck battleship, we’ll just not call it the Vasa, call it something neutral. Then next year, you can have a battleship that’s been designed from the bottom up to be a two-gun battleship. You wouldn’t have any problems with that. But the high management, meaning the king who was in Poland at the time, so he couldn’t even see it, says, nope, must be delivered on time.

[1:18:17] And so they delivered something on time that just couldn’t do the job. But go see the ship. It’s great.

1:18:25 — Template instantiation is Turing complete

Ryan:

[1:18:25] At some point, someone demonstrated that the C++ template instantiation mechanism was Turing complete. So what the compiler is going to do to kind of preprocess that C++ program can actually be used for computation. I was just trying to understand how that is possible. It was mentioned something about calculating prime numbers at compile time. How does that work?

Bjarne:

[1:18:57] Well, the prime number thing was just a curiosity. It used the error messages to report the result. But when you build something, you can get Turing completeness. You need some form of iteration or recursion, and you need a comparison. That’s about it. Then you can achieve Turing completeness. At least some of the theoreticians say, “Hey, we can’t do that. It’ll run forever.” The guy who came up with the first example of this, Erwin Unruh, actually thought I should prohibit it somehow.

[1:19:50] I should ban it. My reaction was, “This looks useful. Great.” I think I was right. Furthermore, nothing runs forever. If you have a Turing machine, you have the tape, and the tape has to be infinite. So if you imagine building a real Turing machine the way Turing designed it, you have to have a bunch of navigators building track all the time when it gets out there. Of course, we don’t do that. The point is that the compiler will run out of resources long before we get into real problems.

[1:20:30] Machines are finite, so the problem doesn’t become real unless there are bugs and the bugs get caught. Guaranteed. So, it’s not a problem. What happened, though, was that people were misusing templates to do simple calculations like prime numbers or trustworthiness. Sieve or calculating factorials and such. It’s so awful, and it’s so expensive, and it uses up so much memory that it becomes a problem. So that was why I and Gabbitas Reyes built constexpr, which basically says you can calculate perfectly ordinary code at compile time. It is much simpler, much more what we’re used to, and much faster to compile, usually resulting in much faster code.

[1:21:37] And you have that today, and you have Constexpr if you want to guarantee that this is done. That takes care of the obvious misuse of the idea of templates being true and complete. It turns them into ordinary functions.

1:21:57 — Abstraction and performance

Ryan:

[1:21:57] Generally with programming languages there’s this high level intuition that the closer to the machine you are, the higher the performance is. And I tend to see C as closer to the machine than C++, for instance.

Bjarne:

[1:22:18] No, it’s not the case, it’s not the case. It’s not as good as compile time calculation that C++ is. And anyway, we have exactly the same machine model because C borrowed the C11 machine model. So if you write the same code in both languages, you get the same result. Except the C++ compilers can do more at compile time and so C++ runs as faster, faster than C. In most cases there’s more information. If you give Optimizer more information, it can do a better job.

Ryan:

[1:23:00] Okay. Yeah, because that was what I was going to ask you. You had said somewhere that C++ can be more performant than C, but I tend to think that more abstraction costs you something.

Bjarne:

[1:23:12] It’s compiled away. This is why I talk about zero overhead abstraction, and people are beginning to take me to task for that because that’s understating the ability of the C++ compiler. We can do negative overhead abstraction.

Ryan:

[1:23:34] What if I was really good at writing assembly and I had all the time in the world to write it? How does that compare?

Bjarne:

[1:23:42] If you are very smart and you have infinite time, you can do better. By and large, we are not as smart as the optimizers anymore, and we don’t have infinite time. So if we are smart enough, we can only do a small piece of code. And now the question is, did we get enough time to use our smarts? This is even starting to affect clever code. I gave a talk to Slack last year, which is a group of very performance-oriented people from the finance industry.

[1:24:33] My title was “Don’t Be Clever.” Actually, the written title was “Don’t Be (Too) Clever,” but I can’t pronounce parentheses. I got out alive. My main point was that C++ is good enough for more than 98% of your code. So if you want time to be clever, you use these techniques, and I showed modern C++. That way, you get time so you can do all the clever optimizations. The problem is, clever optimizations these days tend to be machine dependent.

[1:25:15] That is, if you get a new computer or if you get a new version of the compiler, you might actually have pessimized your code. I’ve seen this repeatedly ever since the 80s. There are people who do nothing but use different optimizations on the next generation hardware. My standard techniques for improving things actually are to first throw away the clever stuff and then see if you run faster or slower.

[1:25:58] Usually, you run faster because clever stuff, at least 1990s store style clever stuff, which there’s a lot of it still today, because the techniques carry on in people’s heads and some of the code remains, tends to use a rats nest of pointers, and that gives the compilers and optimizers problems. They also sometimes use more allocations, which is not good. You want to minimize memory access, you want to maximize your cache performance and things like that.

[1:26:41] And compilers are getting very good at that. I have seen this kind of thinking. I wrote a paper about it together with a friend of mine in Spain doing fluid dynamics, and we threw away the clever stuff—actually a performance test suite example. So it was not a toy, and we got only a 20% improvement by reducing the code to about 80% of what it was before. Some people didn’t think that was significant.

[1:27:26] I thought it was a significant proof that the technique was appropriate. You apply optimizations only when you need them. Knuth says don’t do premature optimization, but he also pointed out that 2 to 3% is where you should optimize, which is exactly the number I’m using. First, build the stuff using high-level facilities, see if it’s good enough, and if it isn’t, and you have to time it, you don’t guess your time. Then you figure out where the time is spent and then you optimize that.

[1:28:10] But a lot of the time, you don’t need to go to that stage; it’s fast enough.

Ryan:

[1:28:15] I see. So when you say cleverness here, it’s like human-level manual management to eke out performance.

Bjarne:

[1:28:23] Yes.

Ryan:

[1:28:24] And you’re saying that actually if you don’t do that, you’re giving the compiler more to optimize, and it can do a good job. It’s much, much better than it used to be. Code that was cleverly and correctly optimized in the 1990s is often pessimized today because machine architectures have changed and the compilers have improved.

Bjarne:

[1:28:31] A good job, and it’s much, much better than it used to be. Code that was cleverly and correctly optimized in the 1990s is often pessimized today because machine architectures have changed and the compilers have improved.

1:28:51 — AI writing code

Ryan:

[1:28:51] When I look at the industry today, more and more code is being written by machines than by humans. I feel like a lot of programming language design is thinking about how to make it amenable to humans solving problems and writing the code. I’m curious if you have any thoughts on whether you think programming language design will change if more and more of the code is written by models and machines.

Bjarne:

[1:29:23] I think that in the field I’m mostly interested in, code will still be written by humans, and they will use abstraction. The examples I’ve seen of attempts for AI to generate code in this domain have not been successful. They generate more bugs, more security holes. They have bloated code, which pessimize again because you use more memory, and it’s hard to validate. The senior developers that would be needed to validate it have started to retire because they don’t want to deal with the validation of something that changes every time you make a change in your code or your prompts. Furthermore, a lot of the things I think about involve regulatory bodies for this validation. You have to be able to validate what you changed. When you make a change and the AIs, the tools change, even if you make a slight difference in the prompt, a lot of the code will change, and you have to check it again.

[1:30:08] I’ve seen some of them starting to retire because they don’t want to deal with the validation of something that changes every time you make a change in your code, in your prompts. Furthermore, a lot of the things I think about involve regulatory bodies for this validation. You have to be able to validate what you changed. When you make a change and the AIs, the tools change, even if you make a slight difference in the prompt, a lot of the code will change, and you have to check it again.

[1:30:47] All of the code that was generated knows more code generated than if it was written by humans. When a human makes a change, it will make a change that’s localized. You can look for the effects of that localized change. If an AI writes it, you don’t actually know where it’s changed. You have to try and figure that out. So if you’re doing something that has been done many times before, like writing a standard web app, what you say is correct.

[1:31:22] Also, AI is not useless. That’s not what I’m saying. It can be used to write documentation. Again, it has to be humanly validated, but it helps write things. It’s good at text. It’s not, at least now, good at safety-critical, performance-critical code. Now, let’s say that 70 or 80% of the world’s code doesn’t fit that pattern. But it’s that 10 or 20% of the code that I’m interested in. And there, it’s not there.

[1:32:09] And I don’t see it coming with the LLM model. Furthermore, when fed with training data, it has to be trained with old code. My job, as I see it, is to make sure people write new things and use new techniques that are improvements over the old code. I find that LLM-based code is imitating old code and getting old performance and old bugs again. Maybe you can improve that. I hear rumors of Bjarne apps being written that fit my writings, but even that is problematic because I’m not saying exactly the same as I did 20 years ago.

[1:33:06] But anyway, we’ll see. Also, even Dijkstra was looking into the possibility, and he claimed that the idea of having natural languages as the programming language was idiotic. He’s less polite than I am. I think that a language like English is very flexible, and what we say is often very ambiguous. We need a programming language that’s precise, that’s engineering, that’s math; it’s not English.

Ryan:

[1:33:48] And for that code that is performance or safety critical, I imagine there will be some group of people using LLMs for that. Based on what you’re saying, the intuition is that you would foresee more breakages and bugs because it’s not valid.

Bjarne:

[1:34:10] People that are really good at that kind of stuff tend to not want to spend all their time validating. Another problem is that they want to eliminate junior programmers because there are lots of them. But if you do that, where do you get the senior programmers from? We’ll see. I mean, you can ask me the same question again in ten years, and there will be more knowledge. Undoubtedly, some of what I said will not be correct. And my guess is some of what I say will be correct in ten years. I’m always told by AI proponents that either the problem has already been solved or it’ll be solved in the next release. But I hear Anthropic 4.7 is having more problems than 4.6 for reasons I don’t understand. The idea that the next version will solve the problem is always a dangerous assumption. Furthermore, it’s getting more and more expensive.

[1:34:42] And my guess is some of what I say will be correct in 10 years. I’m always told by AI proponents that either the problem has already been solved or it’ll be solved in the next release. But I hear Anthropic 4.7 is having more problems than 4.6 for reasons I don’t understand. The idea that the next version will solve the problem is always a dangerous assumption. Furthermore, it’s getting more and more expensive.

[1:35:23] If you have to build a $100 million center and use and run the electricity for it, how many junior developers does it take to be cheaper? They’re starting to need money. It’s not unproblematic. And I’m in a subfield where it is probably more problematic than most.

1:35:54 — His motivation

Ryan:

[1:35:54] One thing I saw in a profile that you did is they asked you what keeps you going on C++ or what motivates you, and you said one is the fun of kind of building the future. The second thing was the obligation to make sure C++ moves forward. When you started C++, I can’t imagine you knew that you were embarking on a journey for decades.

Bjarne:

[1:36:21] And so not decades, but I knew it was a longer journey because I knew I couldn’t build the language I wanted. I could build a subset of it. There were two reasons for that. One was, well, I was the team that did it. Secondly, there was a lack of resources and a lack of time. I didn’t have the input needed to make sure that what I designed was right. So we have the engineering issue: build what you can, see what works, improve it.

[1:36:59] And so I knew I was getting into something like that. I knew I was building a language meant to evolve. And meant to evolve means that you make certain decisions in knowing that that is different. For instance, that’s one reason C++ wasn’t just an object-oriented programming language. Because I could see in the world that there were things that didn’t seem to fit that paradigm. And so I knew we would evolve.

[1:37:35] The other half of that answer to that question, what keeps me going is applications. It’s really nice to see interesting uses and such. I was at JPL and I talked to the people who are doing the Mars rovers. That’s cool stuff. I’ve been to CERN. I’m going to CERN this summer to see how you do high-energy physics. I don’t know anything about high-energy physics. Well, probably more than the average, but nowhere near being a physicist.

[1:38:10] And so you can go there and see they do interesting things, and there are things that surprise you. I was talking to a guy at CERN some years ago, and his job was to open and close doors. These doors weigh a couple of tons and are made of lead; they move across to close off an area to protect against radiation or something like that. I don’t know the details, but the point is he has to start up this door, which is not too hard. You have engines. But then you have to make sure you stop it. Because when you have a couple of tons this wide going into a wall, it will not stop. Normally, you have to write. And the code for that was interesting. I learned something. I still travel around and talk to people and see what C++ is being used for, what it can be used for, and what it can’t be used for. Just learning and learning is fun.

[1:38:45] You have engines, but then you have to make sure you stop it. Because when you have a couple of tons this wide going into a wall, it will not stop. Normally, you have to write. The code for that was interesting. I learned something. I still travel around and talk to people and see what C++ is being used for, what it can be used for, and what it can’t be used for. Just learning is fun.

1:39:18 — Famous quotes

Ryan:

[1:39:18] There’s a few quotes that you have which I thought would be interesting. If you could just give some context behind them. One of the quotes is, “C++ makes it easy to shoot yourself in the foot. C makes it harder, but when you do, it blows your whole leg off.”

Bjarne:

[1:39:38] Somebody asked a question at a talk I was given in Boston back in the 80s, and I shot that one back not thinking, but it’s a good quote and it’s correct. Arno Penzias got a Nobel Prize in Physics, so he’s not a nobody. I was trying to explain to a large group of Bell Labs managers about C++, and he said, “You can’t have a power tool without knowing how to use it.” So if you have a saw, you saw like this.

[1:40:22] If you have a power saw and you try to do that, it’ll bounce, and you will have to be very lucky not to get hurt. Notice it’s roughly the same story. What is behind that is if you get a power tool and you misuse it, you will encounter more problems. Get a car that accelerates faster, and it can wrap you around a tree in a way an old-fashioned, slow-accelerating car can’t. It’s fundamental to having power tools.

Ryan:

[1:41:03] You also have this other great quote: nobody should call themselves a professional if they only know one language. Obviously, you’d recommend people learn C++, but if they had to know a second or a third language for the sake of being a better engineer or programmer, what would you recommend?

Bjarne:

[1:41:22] Yeah, and if you’ve heard, if you’ve seen that interview, you’ll know I waffle on that deliberately. It is not so much which other languages you know, but that you get a set of ideas that are embedded in those languages. So what you should do is learn languages that are different from yours. I’m not too fuzzy about which languages they are. I think I said learn a scripting language today that would be Python or JavaScript.

[1:42:07] Then I guess it was Unix shell or something like that. Have a look at a functional language. ML or Haskell would be obvious solutions. Or just pick something different. The point is that you mustn’t get stuck with just what’s in your language. It’s not good for you to be monoglot. I mean, you know what you call somebody who knows three languages? Trilingual. Who knows two languages? Bilingual. One language? American is a very popular joke, at least outside America. And it’s the same idea with programming languages. But it’s more important with programming languages, I think, because you’re building things, and you should broaden your mind with ideas and techniques.

[1:42:43] American is a very popular joke, at least outside America. It’s the same idea with programming languages. But it’s more important with programming languages, I think, because you’re building things, and you shouldn’t. You should broaden your mind with ideas and techniques.

Ryan:

[1:43:08] Another quote is, “People who think they know everything really annoy those of us who know we don’t.” I was curious about the context behind that or your thoughts on it.

Bjarne:

[1:43:19] Well, that’s. I mean, that’s very simple. There’s so many people who think they’re simple solutions to just about everything in the world. In this context, they’ll come and tell me how much simpler C++ could be. And this is true. If you only want to do one thing, usually the one they have in mind, you can make a much simpler language. But this is like if we threw away this part of C++, it will be much simpler and nicer.

[1:43:57] Usually, they want to throw away things like C++, but then you annoy a few million people, and you don’t actually succeed because they’ll stick to the old stuff. So it’s a way of expressing my frustration with people who oversimplify. People think they can program without learning to program. They think they can be engineers without learning engineering. They think they can be politicians without knowing how to run a company or a country.

[1:44:36] Oversimplification annoys me, and I probably shouldn’t express annoyance. I very rarely do, but in this particular case, my frustration showed.

Ryan:

[1:44:50] Yeah, I saw somewhere in response to C++ being difficult for some people, that programming should be approachable and anyone can learn programming. I think you expressed the opinion that C++ is not necessarily for everyone; it’s for serious programmers.

Bjarne:

[1:45:11] Yeah, I mean, the first line of the C programming language, version one, first edition, was, “C is designed to make life more pleasant for the serious programmer.” I took away the first version, which was professional, because I saw amateurs, and it was really, really good. A serious programmer is probably programming for somebody else. If you program for yourself, it doesn’t matter; it’s you, and that’s your problem.

[1:45:56] If you do it for your friends, you can lose friends. If you build something for a million people, you can do harm in the world. And so that’s what it’s for. I mean, Guido van Rossum built Python with the explicit aim of allowing many people, or even everybody, to program. And he succeeded. I designed C++ to be a really good tool for serious programmers, engineers, mathematicians, and such. And I succeeded too.

[1:46:38] It’s just not the same problem. Remember where we started? I said the problem, look at the problem, and then learn from what worked and what doesn’t.

1:46:48 — Reflecting on building C++

Ryan:

[1:46:48] Looking back on C++ and the whole journey, is there any part where you think, “Oh, that was a mistake,” or something that you learned from in the design?

Bjarne:

[1:47:00] Many, many times I learned something. I think most of the things never made it into C++. That is, this is what you have experiments for, and that’s what you have initial uses for. I think I got the major part of the language right, and I think I could improve every single detail. But stability and compatibility are essential. If you make an insignificant change, it will annoy a few people, and it wouldn’t matter. If you make a significant change, you will annoy a lot of people, and it will not work because, say, a million people will stick to the old way. So I try to grow the language without breaking it. I have this thing that happens again and again. I explain it. People come up and say to me, “C++ is too complicated. You must simplify it.” And I need these two features. I need them yesterday. You must, when you’re doing this, give me these two features.

[1:47:48] If you make a significant change, you will annoy a lot of people, and it will not work because, say, a million people will stick to the old way. So I try to grow the language without breaking it. I have this thing that happens again and again. I explain it. People come up and say to me, C++ is too complicated. Yeah, you must simplify it. And I need these two features. I need them yesterday. You must, when you’re doing this, give me these two features.

[1:48:34] Yes. And whatever you do, don’t break my code. I have a million lines of it that doesn’t work. That’s impossible. And so that is why I’m working on coding guidelines and on profiles, which are enforced guidelines. That way, you can design a profile that ensures you can use the libraries that you need and ensures that you don’t misuse the features that are unnecessary and dangerous in your field.

1:49:12 — Top C++ book recommendation

Ryan:

[1:49:12] For anyone who wants to learn C++, what is the top technical book recommendation you have?

Bjarne:

[1:49:22] There’s a book I wrote when I was teaching undergrads. This is accidental; I didn’t mean it to be there. Anyway, this is the second edition of Programming Principles and Practice Using C++. This is a big fat book written for undergrads. The third edition is not as thick because the language has improved, and I can actually get the ideas across better with less text.

[1:49:59] But use the latest C++. Learn the modern way first. Don’t start learning all the bad ways of writing C as the starter. A lot of courses still say you learn C first, so you learn to misuse malloc and pointers. Then later, you can learn how to use a vector and a string and not have the problems. But yeah, profiles are there to enable compiler and static analyzer support for that kind of thinking.

[1:50:40] And educators are asking for something like that too. A lot of people think profiles are simply to deal with memory safety and performance. No, it has to give people a better tool both for learning and for doing specific kinds of work.

1:50:59 — Advice for his younger self

Ryan:

[1:50:59] And then the last question for you is, if you could go back to the beginning of your career and give yourself some advice, what would you say?

Bjarne:

[1:51:06] Oh dear. Yeah, that’s the time machine question. I sometimes set that for my students. You have a time machine. Go back and give Dennis some advice. And once you’ve done that, step 10 years forward and give me some advice. It’s a good exercise. I usually get some really good stories out of it. Some suggestions. I think a lot of—I tried to avoid the two-way conversions of the building types in C++. I should have fought harder for that.

[1:51:58] I tried but was stopped by the people in Bell Labs, and these were more experienced people than me. I should have gone further there. Furthermore, I should have delayed the release of C++ until I could have something template-like so I could do a better standard library. It wouldn’t have been good enough, but it would have gotten people into the habit of using a standard. Everybody was building standard libraries, and we got saved by Alex Stepanov with the STL.

[1:52:41] But that was real luck because I made a mistake in not delaying until I could have built a good vector and class hierarchy. And then finally, if I’d known what I know now about standards committees and bloated bureaucracies—and we have more subgroups now than we had members to start out with—I would have tried very hard to set up some kind of steering group so that people could make suggestions.

[1:53:27] But we wouldn’t have a vote with, say, 500 people. We would have suggestions from a community of 500 people, and we would have maybe a group of five or six people with vast experience who cared for the whole language and made the decisions based on what was proposed. Something like that. But I did not have the experience or the knowledge to make such a suggestion. Notice that I did not mention tools.

[1:54:03] C++ has a weakness in tools, and that was because it grew up early in a time with limited tools, limited compute power, and limited memory. One constraint on the exercise I give to the students for time machines is to try and make sure that it would be possible to follow your advice. If I just said I want this, then a lot of the things couldn’t be done until 20 years later and therefore would never have happened.

[1:54:49] There’s a lot of languages designed to be perfect for the future computers and the future programmers. Most of them die because by the time ten years later they get the language, the world has changed.

Ryan:

[1:55:07] On that first one, I imagine that would have been really tough to do because the Bell Labs people were so senior.

Bjarne:

[1:55:15] I failed. I tried, but it’s obvious that you don’t want narrowing conversions. I even wrote a paper about how to get rid of them last year in a library. But it is a fundamental flaw in the type system of C++. It came because they needed, being people like Dennis Ritchie and the Unix team, to handle both integers and floating point. They didn’t think of explicit type conversion and thought explicit type conversion was too clunky.

[1:55:59] They didn’t actually get casts until about five years after they got floating point integers. And of course, you have to be able to turn a floating point into an integer, right? So that you get implicit conversions whenever you can. Also, things like integers into characters are problematic. But since you are writing fundamental software, you didn’t want to write something complicated, and you didn’t want to have runtime checking.

[1:56:33] You couldn’t afford that. It was established long before I came, and my attempts to deal with that failed.

Ryan:

[1:56:47] It sounds like it wasn’t for no reason. It saves resources, maybe, or is very limited.

Bjarne:

[1:56:53] They didn’t have the resources to deal with it. They built lint, the static checker, to deal with some of it and small machines. Another thing was that that group of programmers was significantly smarter and significantly more experienced than the average developer. Today we have the law of large numbers. I think the latest estimate I’ve seen on the number of software developers in the world is 47 million.

[1:57:28] And at that time with Unix, the number of programmers was probably a few dozen. The ones that didn’t have a PhD from a good university were geniuses. It’s easier to get a PhD than to be a genius. They offered a different set of problems with a different set of machines and a different set of people. Boy, it was a pain and still is.

Ryan:

[1:58:02] Awesome. Well, thank you so much for your time, Bjarne Stroustrup. I really appreciate it.

Bjarne:

[1:58:05] Okay, thank you.

Also, since he’s on the ground floor of CS education I wanted to know how AI was impacting students outside of the rumors I see on social media.

Below are my top three takeaways to save you time.

You can find the full conversation on YouTube, Spotify, or Apple Podcasts. The transcript is on Substack if you prefer to skim.

Brought to you by:

Cursor3: a unified workspace for building software with agents. I used it to build tools for the podcast and shared the process in this episode’s ad read.

WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code

Takeaways from the conversation:

1) How he lectures well - I was most curious about this since most professors don’t teach in a way that gets students excited like David does. He attributes his success to two main factors.

First, he makes sure to design a “memorable moment” in each lecture. This is usually some fun, visual exercise that gives students a unique moment to anchor their memory to. His classic example is ripping a phone book in half to demonstrate binary search. People often say to him years later that they still remember those moments and the concept he was teaching.

Second, he brings as much energy as possible often motivated by insecurity. He said that one of his worst fears is being in front of a bored audience. He also feels he owes it to the students to represent the subject in a way that makes them want to learn.

2) AI’s downstream impact on enrollments and cheating - Less students are enrolling in computer science. Part of it is fear, but a stronger motive is that companies just don’t come to hire from Harvard like they used to. David said they just don’t seem to be hiring as many junior engineers as before.

The more interesting part to me though was how AI is impacting cheating. He said although cheating has stayed consistent at 5-10% of the student body each semester, what has changed is how hard it is to prosecute cheaters. When people cheat now using AI, they get answers that are difficult to attribute to a source. That makes it harder to hand a smoking gun to the adminstrative board.

3) Why learn C in 2026 and the most challenging concept to learn - C is such an old language at this point so some people critique why you’d need to learn that today. His argument was that C is low-level so you can learn how the computer works without being too low-level like assembly code.

Also, since it is such a small language it forces students to reimplement basic data structures which is a great learning exercise.

He taught so many years of the same introductory courses so I asked him what concept gave people the most trouble. He answered immediately that it was pointers. If you understand pointers well you should feel proud!

We talked about a bunch of other topics too but these are the three I figured would be most interesting.

As I experiment with ad reads, two things I’m sure of are:

1. Only place an ad after value is delivered, say ~30 minutes of content before an ad is shown

2. Keep ads brief, ideally <1min

One thing I’m not sure of though is if people prefer ads to be placed all clumped together or spaced out. People who listen to podcasts, please vote below so I can get a sense of what you prefer:

As always, thanks for reading!
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

1:09 - Getting into computer science

3:27 - Becoming the professor of CS50

11:19 - How to lecture well

14:25 - Depth vs engagement in education

18:11 - Why don’t we consolidate educational resources

23:20 - Why start with C

31:51 - The ideal use of AI in education

34:54 - Cheating and AI

38:21 - Should we really learn CS still?

45:24 - College vs online education

47:06 - The most difficult concept to learn

51:00 - Growth vs fixed mindset

52:35 - The future of CS50

55:56 - Biggest career regret

1:00:29 - Top book recommendations

1:02:36 - Advice for his younger self

Transcript

1:09 — Getting into computer science

David:

[1:09] Yeah, it’s a really good question. So I was that problem set. My first hello World program was in September of 1996 when I first stepped foot in a class called CS50. I was a sophomore at the time and hadn’t honestly gotten up the nerve until sophomore year to even consider taking the class. Like when I came into college first year. And this is a story I tell at the very start of CS50. Most years I really gravitated toward fields that were more familiar, more comfortable for me and I share with students that I really liked Constitutional law, like that was my favorite class in high school.

[1:41] And I figured that meant when you get to college I should probably major or concentrate as we say at Harvard, in government. And so for the first year, year and a half, I was heads down focused on studying government. But when I finally got up the nerve to follow some friends, I think into the CS50 classroom that day when this very well reputed computer scientist Brian Kernahan was at the helm for just that one year, I was totally hooked.

[2:05] And it was the first time in like 19 years that homework was fun. So much so that when the week’s problem sets or assignments would come out, I would look forward to going back to my dorm on Friday evenings to work through the night on my CS50 problem sets. So I felt like that was a sign that I’m supposed to study computer science.

Ryan:

[2:22] And I, I saw on that assignment it says minus two and this little there’s a little note there that I actually couldn’t quite quite make out. Looking back now as the teacher, why did you get a minus two there?

David:

[2:35] It was something stupid and very easily redressible. But I did not start off computer science or CS50 on the best foot. I got -2 out of probably 2 points on hello World.

Ryan:

[2:46] I transcribed it and I tried to compile your program and it didn’t compile interest.

David:

[2:52] Okay, so thanks, 30 years later I learn it should’ve been minus 3.

Ryan:

[2:55] Actually I was able to compile. It’s kind of interesting, but you have to kind of force the compiler to do your bidding. It’s like void main and you’re supposed to be like int main typically.

David:

[3:09] Typically. Now it has since been standardized in C99, which was in my defense a version of C that did not exist in 1996. And nowadays we use C11 or newer in CS50, so you probably had default compiler settings on that did not like that old fashioned main signature. In my defense, that part was correct.

3:27 — Becoming the professor of CS50

Ryan:

[3:27] In 1996, what was the the story behind you going from a student to the teacher?

David:

[3:34] Eventually I had an opportunity my senior year of college to teach an introductory course in computer science at Harvard’s Extension School, which is the continuing ed program largely for adults. The course was literally called Introduction to Computers and the Internet, because in 1999 we were still in the habit of introducing people to computers and the Internet as opposed to taking it for granted more so nowadays.

[3:54] And it was me. In a room full of hundred or so adults, I was easily the youngest person in the room at like 21 years old. The story I usually share is that that was the first and last time I wore suspenders because I was so determined to project being older even though I was still just a senior. But that really whet my appetite in turn not only for more computer science, even though that was very introductory and more technology focused for teaching.

[4:19] And over time I real over the next few years I realized that doors would be closed to me if I wanted to teach at a place like Harvard College. And I eventually returned to Harvard to do my PhD for five years in computer science. And then in 2007 when I was finishing up, the former professor, my own advisor, was moving on to be dean of the school. And so there was this vacancy at the helm of CS50 and I somehow talked my way into the job.

[4:43] They were interested in having a senior faculty member take over the course. I guess they weren’t able to find someone that first year, so I was meant to fill in for just one year until they found someone else.

Ryan:

[4:57] when you took over CS50. It feels like the course completely changed when you got into it. Was your first thought, I want to take this online and kind of make all these major changes, or was that a process over time?

David:

[5:14] Very much a process. And so what people see now is really a evolution of the class, as opposed to some overnight revolution, so to speak. The online aspect really started with Harvard Extension School, since I was fortunate right place, right time to be teaching that extension school course my senior year, when distance education, so to speak, was becoming a thing. And my mentor and boss, Henry Leitner at Harvard, offered me the opportunity not only to teach this class, but also to put it online, I think in the year 2000, give or take.

[5:43] The very first year I taught it, spring of 1999, we were still filming the class, but on VHS. And fun fact, at the time, it was the course’s teaching fellows who were, in fact, my friends, who were also undergraduates at the time, who, who were doing the filming and operating a tripod with a camera. And I swear to God, we still have footage, probably on some VHS tape of one of the cameras going when one of my dear friends and teaching fellows fell asleep during the middle of class.

[6:08] So we’ve aspired to higher production value since, but it was really, it happened very organically. So once we started participating in the extension school’s online education program for several years, Apple released the iPod around 2003 or 4, give or take. And so at the time, we were distributing the course’s video via real video and or real audio, which are older technologies now, but were state of the art for streaming at the time.

[6:35] We figured out a way how to sort of download all of those bits for redistribution as MP3s initially and put them on itunes, which back in the day you could upload, and maybe still today, your own RSS file, which is a flavor of xml, which is kind of similar to HTML, but more custom, that just describes what and where your audio files are. And we made this available primarily for our own Harvard Extension School students, because at the time, even though it was very progressive to have streaming, it was a little annoying that you needed a constant Internet connection.

[7:03] Buffering hadn’t really been invented yet. Certainly downloading those videos was, by design not possible. But with podcasting, of course, you could download the assets in advance. And for our own students, the vision was that they could listen to the class only on audio at the gym or during their commute and just generally make the educational Content a little more accessible. A year or two later, Apple released the video ipod.

[7:24] And so we very much steered into that and downloaded those same assets, but in video form, to make them available on the video ipod and people’s Macs and later PCs. And we were fortunate around that time, early 2000s, where Wired magazine took notice and noted for us that we were among the most popular podcasts in some category. And it was kind of news to us that people cared. We did know that we were starting to see bandwidth problems at the very early days.

[7:50] In the very early days, we were using Dreamhost to host the MP3 files. And Dreamhost had, I think, free accounts, but only up to some number of megabytes per month. And we kept hitting the threshold. And so what we would do is sign up for another DreamHost account, another DreamHost account, another Dreamhost account. And we would use, like, DNS trickery to just reroute the URLs in the RSS file to the new web host to keep up.

[8:14] And so that’s when all of our attention sort of was drawn to the possibilities and that we can make available this content not just for our own students, but for anyone else who either geographically, socioeconomically, couldn’t access that same kind of class.

Ryan:

[8:29] I see. So the idea was it was a supplementary material for people who had paid for the Harvard Extension School and then. But other people just open on the Internet could listen if they wanted.

David:

[8:41] Yeah. And that alone was a touchy subject on campus at first, because here we were giving away the very thing that other people were paying for. But at the end of the day, what people are paying for is the additional support structure you get in a more traditional environment, the more traditional credentialing, the transcript and the credit and so forth. But the knowledge itself, it does society no good to withhold that.

Ryan:

[9:01] I think so. I think this is a brutal reality of making content on the Internet. There’s so much competition for attention that all the content has to kind of hook people. Let’s say you’re scrolling on. On Instagram. That first second’s really got to capture someone, otherwise you’re going to go to the next one. And I’m curious, because you are making content for the Internet with these lectures that you post online.

[9:27] Do you think about, how can I, you know, hook the audience? Does that ever factor into your. Your lecture design lectures?

David:

[9:35] No. Some of our social content, yes. Insofar as we are producing, in addition to the courses traditional in classroom content, things that are designed clearly for social media, whether it’s interviews, for instance, whether it is excerpts from CS50‘s Fair, this exhibition of final projects, whether it’s steering in the directions of some popular meme just to sort of catch the attention of our own students or new students.

[9:58] But we have been very deliberate over the years from the beginning and especially since the age of the MOOC in 2012 when Coursera and EDX and other platforms were coming around to not steer into this bite sized 3 minutes, 5 minute chunks that I know there is data to confirm that it drives engagement but not necessarily educational outcomes. We have fairly unusually in recent years stuck to the more traditional model of like CS50.

[10:27] Lectures are like three hours long. And there are downsides to that. Absolutely. But ironically, online you avoid some of the in person downsides because you can pause and rewind and fast forward, you can search for something, open another tab, go down this intellectual rabbit hole or rewind to figure out what it some detail you missed. I mean I, I can’t even count how many times in college, high school, more generally that I’m sure I zoned out.

[10:51] And it’s too often the case that if you miss some detail you’re now you might as well leave the class at that point. And like online solves that for you. And I haven’t seen a need pedagogically for us to break up the content into such small pieces and to sort of take that ownership away from the students. I think we’d prefer that they meet us halfway. And if you need to take more than three hours to watch the lecture, that’s totally fine.

[11:15] But it’s not designed for scrolling type consumption I would say.

11:19 — How to lecture well

Ryan:

[11:19] Right. What about in the, in the idea of like a long video, there’s still this concept that you kind of want to keep people engaged. And so how do you keep people engaged for a three hour lecture?

David:

[11:30] Yes. So that we are very conscious of and the way we’ve generally described that is trying to create memorable moments. And this is not for social media. This is as much for the in person audience as it is for the online audience. And by memorable moments I mean in like CS50‘s first lecture we do this bit where I’ll tear a phone book in half to demonstrate the performance of something called binary search and algorithm for finding information more quickly than linearly searching the phone book one page at a time.

[11:55] And that’s really just to be dramatic. It has less and less impact these days because most students of school age are no longer familiar with the Technology known as a phone book. But it’s literally analogous to the digital form in iOS or in Android, of course. And so it doesn’t take a huge leap for someone to realize, okay, this is the physical incarnation of that, and it’s just kind of fun, even though it’s getting harder and harder to find the actual phone books.

[12:19] When we get to searching algorithms and sorting algorithms, we very often have a student come up on stage, for instance, and for sorting or for searching, we have a whole bunch of like gym lockers, small versions thereof, that we put numbers in, either on printed paper or little plastic numbers. And the goal is to have a student like, find the number 50 or some other number that we’ve hidden behind the doors.

[12:38] And it’s an opportunity just to talk about either searching the doors linearly, left to right, right to left, maybe randomly, maybe more methodically as via binary search. If the numbers are sorted, that then leads us to opportunities to talk about bubble sort and selection, sort and insertion sorts. Algorithms that you probably wouldn’t bother implementing or using in the wild because there’s more performant ones certainly.

[12:59] But we use them as these pedagogical tools to explore the. The problem to be solved, which is, all right, I can only use binary search if the data is sorted, so how do I sort it? And how expensive is that going to be worthwhile, the whole process in the get go? And so we’ll have eight students come up on stage and sort themselves physically and really perform in certain ways. And there’s so many examples of that through CS50, we try to do at least one such thing every class.

[13:25] I mean, not unlike the way John Oliver, for instance, ends last week. Tonight we’re trying to do some big song and dance, which ironically is always at the end as the hook. We do try to put it maybe toward the beginning of lecture or in the middle, but that’s not just for engagement and not to make, to keep folks attention, but really to latch onto their memories so that even if you’re in the weeds of this introductory computer science course, there’s so much new information, it’s the proverbial fire hose hitting you in the face you can cling to.

[13:52] Okay, all right. I remember like my. My roommate was literally the one on stage acting out bubble sort. And you can kind of picture in your mind’s eye what was happening. Or that phone book example is a perfect one that years after leaving Harvard, we have alumni coming up to us and saying, oh, I still REM the phone book demo. And those memorable Moments, ideally theatrical in nature, I think are what brings material, whether it’s CS or something else, to life.

[14:16] But that is very much to engage any learner, present or remote, as opposed to being optimized for some technology.

14:25 — Depth vs engagement in education

Ryan:

[14:25] One thing that I, I wonder because computer science is such a dense topic where I guess there could be a trade off with, imagine another style of lecture is, you know, there’s a chalkboard and you’re up there kind of blandly writing out all of the information. I, I could imagine you could go through much more depth in that case. Do you feel like there’s any trade off there?

David:

[14:52] Absolutely. I mean, this is a criticism we get from some students, particularly those with prior background or among those more comfortable where do we really need 10 minutes on linear search or you know, another 15 minutes on binary search? And the answer is quite clearly no for those students. And even for most students, I would say we do spend a disproportionate amount of time on certain cherry picked topics that we do think lend themselves to this theatricality for really getting students interested in fundamentally and excited by the material to help motivate them the rest of the week when they’re going to be spending 5, 10, 15, 20 plus hours on some weeks assignment and to help them see sort of the forest for the trees, like what is actually important and meaningful and fun about this.

[15:34] Because it’s probably not the keystrokes and the debugging and the lower level implementation details. So we talk about this actually in a paper, in a talk we gave at a CS education conference a few years back, trying to find that balance between the theatricality and the density of material. But to your question about dryness, I mean, I would like to think that dryness really has no place in most forms of education because then it could have been a book or it could have been an email if it’s just someone reciting words, worse, writing things down in ways that aren’t at all interactive, like do we really need to be all in this room together?

Ryan:

[16:13] design to me in my research. And looking at the reception of CS50 and people’s perspective on how you are as an educator, they absolutely love your energy and like your delivery. Do you rehearse that or is that something that you always had as an educator?

David:

[16:31] Yeah, I think it’s just sort of the expression you got to bring your A game when you get up onto stage. And I think a lot of it honestly comes from a place of insecurity. Like I really don’t want to be the one on stage in front of a bored audience, like no one wants to be in that situation. And so I think a lot of the energy that I try to bring to bear and the excitement I show is really meant to one, make sure people want to be and stay there for the duration of the class, whether it’s short or long.

[16:56] And also help them see the excitement in and the possibilities of, and the applicability of a field that I myself have fell in love with so many years ago. Because then they can decide for themselves if they feel that same emotion, if they too look forward to going home to their dorm or their home on Friday nights and working on programming assignments, of all things. But I feel like we certainly owe them, and I think teachers owe students more generally educationally, the opportunity to make that decision for themselves and to put forward our best foot, or to present the field in the most exciting, inspiring way that we can, if indeed we think it’s important that other people after us study and research and apply these fields.

Ryan:

[17:40] I noticed this dichotomy in education when I was going to ucla, which was there were roughly two types of professors that I had. There was one type of professor that was beloved by the students and really passionate at educating and kind of involved. And then there’s this other type of professor which was really they might have been really incredible and impressive as researchers, but their classes were really poorly rated by all the students.

18:11 — Why don’t we consolidate educational resources

[18:11] And with the online education that we can do today, I mean, like CS50, all the resources are available. I could imagine that objectively we could take the people who are most passionate about teaching and give it to everyone, rather than having some teachers that are not excited kind of consolidate. Imagine taking the world’s best instructor and giving that to everyone. Why doesn’t consolidation happen if we have all the technology for spreading literally the world’s best resources to every situation?

David:

[18:41] Yeah, that’s a good question, since I think our own goal is certainly not to put out of business other educators, other classes. However, as a computer scientist, I would be among the first to notice that there’s a huge inefficiency in the way education broadly works, not only in this country, but presumably internationally as well, whereby you have hundreds, thousands of teachers essentially all doing the same thing in parallel, and not necessarily the meaningful part of interacting with and uplifting and mentoring students, but like preparing their presentations or their lecture notes, or writing the assignments, or grading that work.

[19:17] There’s so much meta work involved in education that probably doesn’t need to be borne by so many people that the computer scientist in me invariably wants to factor that out somehow. And so I absolutely think that we, educationally, as universities, as schools, should be leaning on each other much more. I’d like to think that’s a lot of what drives us in terms of CS50‘s own mission and the work that we do.

[19:39] But we too have encountered frictions along the way. It was very unusual, for instance, for 10 years that we were collaborating with our friends down the road in New Haven at Yale University where we were offering CS50 on both campuses in parallel. And more recently we’ve been doing this with some of our friends at Oxford and lifelong learning group there. This is very much still the exception to the rule.

[19:59] Unfortunately, even after 10 plus years of MOOCs massive open online courses, I dare say a lot of institutions, a lot of faculty are very set in their ways. In fact, one of my regrets of the COVID era was that we had this unprecedented opportunity now and almost mandate to move everything online. And we therefore had this opportunity to say, hey to Harvard students, why don’t you take this Stanford course or this UCLA course or this Yale course, or this MIT course that you couldn’t necessarily take in person for lack of transport or for lack of safety at the time.

[20:32] And I could get no one on campus to, to get on board with this idea of maybe offering one computer science course on this campus and let the other students take it. Then you offer, as we did at Yale, like a digital humanities class on some other campus and let the Harvard students take it. And there’s just not much of an appetite, I dare say, in higher education, if not education writ large for that resource sharing.

[20:54] And I think there should be, I don’t think there should be one computer science course, introductory course. I think some healthy competition is a good thing. I don’t think there need to be thousands, probably not hundreds, maybe dozens, from some of the best teachers. The best schools would probably benefit us all if those same teachers then were not put out of work, but then could lean on each other, use some of the materials we’ve created, adopt or as we say, adapt some of our own resources and treat education as a buffet of educational materials that you can then make your own without having to do so much of the same legwork and reinventing wheels across school and state lines.

Ryan:

[21:32] The mindset that you said, the computer scientist mindset of getting rid of the redundancies and kind of consolidating resources, I mean, it’s immediately logically obvious to me. But what would like, let’s say there’s three top courses in America. There’s like MIT‘s, there’s Stanfords and there’s Harvards and they want to give that to other institutions, like, I don’t know, other colleges. What would stop those colleges from taking an objectively better set of resources and giving that to their students?

David:

[22:10] I sense, but I would defer to others who hold these views that there is a concern that maybe we are putting ourselves out of business or there’s more of a school pride. And I definitely saw some of that on Harvard’s own campus where like, no, no, no, no. We should be offering courses to our students that we have created and that we are teaching and not lean, for instance, on our friends down the road at MIT who have and have for decades had a larger, richer course catalog than us, simply by nature of being a bigger place, at least for computer science.

[22:40] And it seems silly to me not to lean on each other. Maybe each of us can specialize a bit more. Maybe each of us can offer slightly different modes of instruction for students. Maybe that’s a little more intimate on this campus versus that so as to really leverage shared resources. I mean, this is done endlessly in research, having cross campus collaborations and there’s something very personal about or very fundamentally threatening.

[23:07] I think about leaning on each other educationally. And I wish people would move away from this mindset because it doesn’t mean some failure of the institution to offer these courses. It means we can do a better job offering what resources we do have.

23:20 — Why start with C

Ryan:

[23:20] I would like to think in designing the curriculum for CS50, how did you pick C for instance? Because I think a lot of people would look at that decision and think, I’m not going to use C in my day to day full stack job, so why do I need to learn this?

David:

[23:36] Sure. So it was not picked by me per se, but I chose to keep C in the course certainly since as far back as 1996 when Brian was teaching it, Professor Margo Seltzer before that and many other faculty prior to me and them. It is a wonderful foundation on which to build your understanding of how a computer works and how software is built. It’s about as close as you can get to the hardware before things devolve, at least aesthetically into assembly code, which is much scarier looking code.

[24:07] I think for most people, almost everyone perhaps, and certainly beyond that is zeros and ones, which is not going to be fun for anyone. So C kind of strikes, I think pedagogically this really nice balance of having English like syntax and abstractions on top of lower level primitives that allow you to explore procedural programming in particular with some fundamental constructs that are now fundamental to those kinds of languages.

[24:34] Loops and conditions and functions and variables and return values and so forth. It’s sort of got, it’s got everything, but it’s also a pretty small language and unless you download third party stuff, there’s not a very large standard library in fact. So most anything you want, you need to build yourself. And so we leverage this significantly in CS50. So much so that by mid semester in week five of the class, students are building their own hash tables.

[24:56] And we’re talking about how you could construct singly linked, doubly linked lists, hash tables and tries trees, abstract data types like stacks and queues and so many more. And what I think is especially meaningful about C is that you can’t just instantiate one of those data structures if you want one, like you can in Java and C with STL and certain other libraries. Like if you want it, you’re going to have to build it yourself.

[25:18] And I think that alone is a good educational exercise. Not because you’re going to need to build that thing again, but because there’s value, I think, in really understanding from the bottom up what is going on inside of that device. So that one, you can make more informed decisions as to how you want to engineer and design your own data structures down the line. Two, you can diagnose problems by reasoning through first principles what could possibly go wrong because you understand how the data is being stored and what the algorithms are that are performing on that data.

[25:50] And three, it’s a wonderful scaffold to higher level languages because one of my favorite things from week 5 to 6 in CS50 is students go from having written in week 5 like their own hash tables implementation for adding data, retrieving data and so forth, which is like this many lines of code for some font size and it gets whittled down in week six to one line where you just instantiate a Python dictionary and you can be productive with a dictionary.

[26:17] And many courses do teach programming by way of Python alone. And we too have done it for some of our audiences. But you really never get around to understanding what’s going on underneath the hood. And our goal in CS50among them is not to output programmers, but engineers and educated citizens and folks who really understand from first principles how technology works. And so C for instance, strikes I think, just that right balance.

[26:42] And for those students who want to go even deeper in a systems class, they can go learn about assembly and compilers and so forth. But those students who want to go on to web programming or data science stuff or AI nowadays can just build on top of the C and then in turn Python layers that we use in the class.

Ryan:

[27:50] This is not my perspective, but I was doing research and I saw a YouTube video. It said, Whatever you do, don’t take CS50, clickbait, whatever. Yeah. So I watched the video and the person. So it worked. Yeah, it got me. It got me. And the perspective of the author of this video was that CS50 teaches you all this stuff that you don’t really need to know if you were like a full Stack engineer or something.

[28:15] If you, if I was just going into the industry and I’m just making web apps that a lot of this underlying stuff you might not need and maybe it’s not a good use of time. And I’m curious what you would say to someone that has that mindset that you don’t need to actually know how the computer works.

David:

[28:35] I don’t want to get into a whole Internet fight here, but I think that is absolutely the wrong mindset, certainly for a full stack engineer. I mean, by definition a full stack, you should be understanding everything that’s happening among those layers. So I think the better formulation isn’t that you don’t need to know these things, but rather you won’t need to use these things. Use in a literal sense.

[28:56] Like I don’t. My C is a very popular language, even according to some rankings each year it’s, you know, the number one, number two language in terms of its omnipresence still to this day because it’s very highly performant, albeit more challenging to write than some languages. I only use C for five weeks during CS50 itself, but that doesn’t mean that it hasn’t helped me understand higher level languages.

[29:19] What is going on inside of a system, how you can improve the performance of or the design of some system by understanding again those first principles. I don’t use Scratch except for one week out of the year. I do use Python more frequently and I use JavaScript and some HTML and CSS. But I think if you’re going to call yourself an engineer, you should absolutely have mastery of and knowledge of those underlying building blocks if you want to not just output something that frankly AI could output nowadays, but you can understand and you can create the next thing or the solution to some other problem that we haven’t even yet solved.

[29:57] I think that’s the better mindset to appreciate. Yes, I’m not going to need to use Scratch or C or maybe some of the other things we touch on in CS50, but the knowledge and the principles that we extract from those implementation details are incredibly valuable if you want to be an engineer and not just say a coder, which is a distinction that some folks might make.

Ryan:

[30:18] I was looking at the syllabus and there’s C obviously, and there’s all the amazing, you know, basic data structures, those types of things, bread and butter. And then in the end of the course there’s a week on artificial intelligence, which I was surprised to see in an intro course because it’s kind of like there’s no way that you could teach AI in a week. So I’m assuming it’s a more high level introduction.

David:

[30:44] It is. It’s only for an hour and it technically temporarily, it’s offered in the middle of the semester for the on campus students. It coincides by design with family weekend when first years and juniors students, parents come to town very frequently. And so we do it as a very broad introduction to what everyone is talking about nowadays in the AI space. There are full fledged classes of course at Harvard and other institutions that students can take and it’s really meant to whet their appetite but also give them some context for the very tools we’re using in CS50.

[31:14] We have this virtual rubber duck that’s built on top of OpenAI‘s APIs and Microsoft Azure’s web API service that they’re using every day or every week certainly throughout CS50. So among the goals pedagogically is to help them Understand how do the tools work that you yourself have been using. Two, to help them understand what is it that the world is talking about nowadays. And three, prepare them to use these tools more effectively by the end of the semester.

[31:38] Because for instance, for CS50‘s final project, students are encouraged and welcome to use Claude or ChatGPT or Gemini or any number of off the shelf AI tools, though we don’t allow those through policy for the course’s assignments.

31:51 — The ideal use of AI in education

Ryan:

[31:51] Yeah, I had a conversation with another professor and specifically about, you know, is AI affecting how the kids are learning? And his perspective was it was kind of maybe he didn’t do it properly and the students were over relying on it. And I think that is a concern that a lot of people have, is that students these days can be more brain dead. Like you could just go to ChatGPT and say, honestly just solve the problem for me.

[32:21] For you as an educator in computer science, what is the ideal relationship with AI for your students?

David:

[32:28] Yeah, so this virtual Rubber Duck at CS50AI, which anyone with a free GitHub account can use, is really meant by design to be a less helpful version of ChatGPT, one that is also more tuned to CS50‘s own material and syllabus and so forth. And that’s because all of these tools off the shelf can pretty much do your homework for you. And this has been true for several years, even before the fall of 2022 when ChatGPT came out.

[32:52] I mean we were looking closely at GitHub Copilot for some time because if you created an empty text file in VS code called Mario C which is the file name we use for one of CS50‘s problem sets. And if you so much as type I think hash and then I for include, you pretty much get a suggestion to autocomplete the entirety of that particular problem set. And that’s just because, I mean we, for better for worse are part of these models insofar as the OpenCourseWare has presumably been slurped up, as with the rest of the Internet, as part of the training data, so to speak, for these AI models.

[33:23] So that has both good and bad. And that’s why we set out to make our own sort of duck themed version of these tools that puts downward pressure on that willingness of the tools to be too helpful. And we’ve tried to attune the duck to be more akin to a good teacher or tutor that leads you to the solution, but certainly doesn’t auto complete your whole way through it. I see.

Ryan:

[33:42] So there’s like a system prompt or some scaffolding that says, don’t answer the question but help me figure it out.

David:

[33:50] Pretty much. And it’s much easier to do this now than it was in 2022 and 2023 when we first rolled this out. Which is to say there’s a lot of tools, even commercial tools, that faculty can use to do the same kind of thing for their own course. For us, it was important to draw a clean line in the sand to students because you could approximate this duck by just telling students, hey, everyone, go copy paste this system prompt as you describe into ChatGPT before you ask your homework question.

[34:16] And no one’s going to do that. And like it’s going to drift out of date and it just feels too clunky. Students are just going end up typing into the prompt. Whereas I think it’s a lot cleaner, if not simpler to know. I cannot, through policy, go to ChatGPT, Gemini, Copilot, any of these things, but I can go to and use as much as I want CS50AI or the plugin we have in VS code of the same. And that to me is a very healthy line because, you know, if you’re crossing that line, if you’re pulling up chatgpt.

[33:50] com or gemini. google. com and the like. And at that point, it’s a conscious choice to be academically dishonest, as we would describe it in CS50‘s syllab.

34:54 — Cheating and AI

Ryan:

[34:54] When I was going through my CS education, cheating was already pretty rampant. I mean, people put their code on GitHub and you kind of paraphrase someone else’s code. Well, not me, but other people. And I imagine with the new technology that actually, I mean, with anything with cheating, it’s adversarial. There’s people who are cheating and there’s people who are trying to catch the cheaters. Right.

[35:19] I could imagine that the cheating tools are advancing faster than the ability to catch them because if you just generate the code, it’s not easy to say, oh, that was, you know, AI generated or whatever. So I’m curious if you see more cheating on your end.

David:

[35:41] of dishonesty, statistically, we are not catching more. I would like to think that the, the misbehavior has not markedly increased, if only because we, like a lot of intro courses around the world, have a tradition of looking for academic dishonesty, plagiarism, code that was copied and pasted off the Internet or YouTube video. And we within CS50, like a lot of peer institutions are very good at catching that.

[36:08] And like, we have tools that cross compare all of students submissions against each other against GitHub repositories of past submissions that we have of YouTube videos that have been just transcribed. So we have historically administratively disciplined, so to speak, between 5 and 10% of CS50 student body every semester. And that’s kind of a norm among across peer institutions, you know, upper bound of roughly 10%.

[36:31] There’s certainly probably some percentage of students who have been cheating in some form all these years and never have been detected. But just based on the rigor with which we go through this, the messaging throughout the course, like I’d like to think that a good 90% of students are behaving in the way that they should, at least when the boundaries are set through not only policy, but course culture and through the support structure that we provide to students.

[36:56] That said, we’ve not seen an uptick in detections of academic dishonesty. What has gotten harder is the so called prosecution of those cases in the sense that it’s harder for us now to hand to the administrative board of Harvard or the Honor Council, so to speak, be a smoking gun. Like, here is the URL from which this code was copied, because it’s not really coming from a URL or a YouTube video, it’s coming from the composition of all of the URLs out there and all of the YouTube videos about CS50‘s problem sets.

[37:26] Because the AI, somewhat pseudo randomly is spitting out an amalgam of these various training inputs. That said, when you’ve been doing this long enough, like you can tell that this is not the work of this particular student either in comparison to past work that they submitted or, or the sophistication with which they’re solving a problem or the incorrectness with which they’re solving the problem.

[37:49] And there’s often telltale signs like this is not this year’s problem set, this is last year’s. And so clearly this did not come from you alone. So we certainly are able to detect the easier to detect behaviors, right?

Ryan:

[38:02] 5 to 10%. That feels spiritually like a lot like 1 out of 10 students doing something. But I mean, you have the.

David:

[38:12] Yeah, I mean, 20 years ago I probably would have agreed after seeing it year after year after year. I’m just glad it’s not higher than that.

38:21 — Should we really learn CS still?

Ryan:

[38:21] On the topic of AI, a lot of students, they see where these, these agentic tools are going. They’re generating code that is superhuman in many cases. And I think it gives people this sense of dread. Like imagine you’re spending all this time getting better at programming and you think that maybe programming will be useless in the future. You know, if an anxious student comes to you and says, I don’t know if I should be learning computer science, what would you say?

David:

[38:50] I wouldn’t worry about getting better at programming. The whole point of so many of these courses has been about getting better at problem solving. And like that is a life skill, whether you’re going to stay in CS or tech more generally, or leave it for some other field in which there’s still going to be problems, just different types of problems. And there too is why we, we constantly tell students the course is not about Scratch and C and Python and JavaScript.

[39:14] Like these are all implementation details. And while they definitely have and consciously have pedagogical value, that’s why they’re in the syllabus. The goal is not to teach students Scratch and C and Python and JavaScript, but how to solve problems and to give them representative tools with which they can solve those problems and also an understanding of those same first principles from which they can reason about some symptom they’re seeing or some problem they are ultimately trying to solve.

[39:39] AI is pretty darn good already at generating code. It’s only going to get better. But that’s wonderful insofar as it’s going to save us humans from the stuff that was never very fun to begin with. Even I am guilty of never really writing as many like unit tests for my code to test the correctness automatically. That stuff can be very easily and robustly generated nowadays. What I don’t want the computer to take from me is sort of the fun part of that.

[40:03] The fun part that’s always been there, which is the designing of the system itself. Maybe thinking about what the UX should be so that you can optimize for the user experience, or what the database design might be or what data you want to collect in pursuit of some business problem. Because you know you’re going to need to analyze or mine it for some valuable insight later on. Like that stuff’s probably not going away anytime soon.

[40:25] And the other reassurance I would give students nowadays is like the world is only getting more technological. And we’ve kind of seen this pattern before. Where I took CS50, for instance, at really the peak of the so called. com era era, so 1996. And then sort of the world started trending down when all the startups were going out of Business and students were exiting computer science. But like then, the pendulum swung back and I don’t think we have saturated the world in terms of technology.

[40:50] Every business, small and large is probably going to keep using it more and more. We in our personal lives are only going to keep needing it more and more. And so, like, those are problems to be solved. And yes, I don’t think we’ll need as many humans to write HTML and CSS anymore, which was, at the risk of picking another fight, never that much fun anyway. And maybe we won’t really need to use C or Python anymore, but we can start programming like a pm, a product manager, using our voice, using our words in English or some other human language, but that alone would be wonderful.

[41:20] Case in point, I was playing with Claude this morning to just help me prototype something, and it was wonderful for me to be able to just think about the problem and sort of the engineering side of it and say, all right, Claude, much like an intern, please go implement this for me. Because I really don’t want to go reading the documentation, which was never that intellectually interesting. I could if I wanted to, and I understand how it’s structured, but I don’t need to keep repeating that kind of exercise again and again.

[41:44] But at least in the near term, case in point, Claude came back very confidently with a solution that was like 90% correct. And I had to convince Claude, so to speak, by chatting back with it that, like, no, no, no, no, like this is not possible per the API docs at this URL. So of course you get the apologetic response from the LLM and this kind of error rate or hallucination will decrease with frequency.

[42:08] But, like, this is why the human should stay in charge for some time.

Ryan:

[42:12] In today’s environment, we can relatively confidently say, these tools empower us and they don’t replace fully yet. But I think a lot of people are worried about, okay, but what if, you know, it gets. What if it gets even better? What if that 90% becomes 99% or maybe it becomes 100%? Then I think people get scared, but it’s hard to say exactly in the future.

David:

[42:39] But insofar as our goal within even a class like CS50 has always been to output a more educated citizenry, to teach students how to teach themselves, to teach them how to be more methodical, more algorithmic and problem solving, like all of that is valuable, whether or not we have robots doing work for us in our future.

Ryan:

[42:59] I think you talked about the enrollment rates and I have this little screenshot because you had written this, I think, this paper from long time ago about redesigning CS50. And you kind of, I’ll put this overlay this in the video, but you know, there’s this trend of kind of enrollment rates going up and then they tank with the top convobole and then they went up as well because of some adjustments that you’ve made.

[43:25] Right now we’re kind of in this. You know, AI is really shifting things. How if you were to continue this graph forward, is enrollment and interest in computer science declining because of the AI?

David:

[43:39] Oh, lately that absolutely seems to be the case. And that was preceded, I think over the past few years really by the downturn in tech offerings. I mean, we were seeing this in the recruiting pipeline on campus where there were just fewer opportunities and there was less of an appetite among some of the big tech companies to even bother coming to campus if they just didn’t have many entry level roles for students.

[43:58] And that was before ChatGPT and similar tooling was sort of omnipresent. So AI has only exacerbated, I think, that concern. But I think what we’re going to see if we sort of extrapolate out is that there’s going to be these ebbs and flows over the years, much like we’ve seen, or the pendulum swinging each different way, whichever metaphor you prefer. Just as humans have this tendency, I do think, especially in tech, to overreact to things both positively and negatively.

[44:24] And while AI is absolutely here to stay, I do think it’s going to become equivalent in our minds to just what technology is. AI is a nice sort of buzzword and way to distinguish it from some of the more deterministic or predictable outcomes that like technology to date has maybe had. But. But I think back Even over my 19 years with CS50 and AR and VR was to be all the rage. And then we had blockchain and bitcoin and Google Glass even excited the imagination like no other technology at the time.

[44:56] And all of these things are still with us in some form or some future incarnation. And especially with blockchain and bitcoin. There’s some resiliency there. And AR and VR still very omnipresent in games. But it’s just been fascinating to watch the world, the tech world, especially the VC world, sort of react and overreact habitually to these kinds of things. Until we sort of find a healthier medium where people appreciate the real value and not just the opportunistic value of it all.

45:24 — College vs online education

Ryan:

[45:24] You’re in a unique position because you teach the in person Harvard class and then you also do the massively online version as well. And I think it’s an interesting case study because some people have the opinion that, that the important part of college is actually not the education. And it feels like this in person version versus the online one is kind of proof of that, which is that you could not go to Harvard but get a world class education, yet the outcomes are very different.

[46:01] So I’m curious your thoughts on going to college versus getting the world class education for free?

David:

[46:09] Yeah, I think it depends on your goals. I think there’s absolutely value of places like Harvard really because of that credentialing. And also it’s a filter from the get go. I mean these admissions departments are already identifying strengths among people and sort of the audience you then end up with is very similar along certain axes, certainly along academic abilities and whatnot. But the value of places like Harvard I do think has long been the networking opportunities, the connections you make, the life experiences you might have and academics sometimes to a fault like is often secondary.

[46:39] But when it comes to knowledge and learning outcomes and practical skills, I mean I’d like to think that you’re getting as good, if not better an experience taking a class like CS50 online, if only for the simple technological reasons that you can pause and fast forward and rewind and open multiple tabs, go down this intellectual rabbit holes in ways that a traditional campus is just not designed to support.

[47:02] Unless those same students are are taking the class in their dorm rooms or at home.

47:06 — The most difficult concept to learn

Ryan:

[47:06] Like during COVID You’ve taught so many years of this same material. What is the most difficult concept that you see? People consistently having trouble pointers in C.

David:

[47:19] And that was true for me 30 years ago. So much so that I’m always very weirdly excited to tell students in class nowadays that I still remember where I was on campus the day I finally got pointers and and I’ll share here. I was in one of the dining halls on campus with my teaching fellow or TF who himself was a student probably one or two years older than me. We were sitting at the back of the dining hall on these wooden tables in this beautiful old school sort of Harvard esque room.

[47:47] And somehow or other something he said that day, after weeks of struggling, that light bulb went off for me and I’m like oh, it’s just an address which you know, never quite clicked. Even though I’m sure many people tried to hammer that into me previous to that. But. But it’s just hard like memory Stuff is hard, especially if computers to students from the get go is very much this black box, this abstraction itself.

[48:09] To really go down to that level takes some, some new thinking. And it took me a while.

Ryan:

[48:15] Let’s say someone is trying to help someone else grasp a difficult concept. Like, what’s your best tip on how to really teach someone something that they’re really struggling with?

David:

[48:26] My instinct is to say metaphors, or what we more broadly call memorable moments, where it’s a phone where you’re trying to teach binary search, or even a Screenshot of the iOS context app where you’re trying to introduce the same algorithm. It’s during COVID for instance, we worked with a local theater and we had seven physical doors on a stage that were from previous plays that they had put on. And we put fuzzy little numbers behind that the prop shop had kindly developed for us so that we could play that same game of using not lockers, but physical doors.

[48:56] And creating those binary doors for students, I think helps them latch onto a visual that makes a very esoteric topic that, that frankly, who really cares about binary search day to day just kind of come alive and you realize, at least back in the day of actual phone books, like, oh, I’ve been using this the whole time. I just didn’t know what to call it. And I think that’s very empowering to use metaphors to create memorable moments that help students realize that even the seemingly most arcane topics are still very much within their grasp and perhaps not so dissimilar.

[49:28] Even when we talk about binary for students, we like to show them that, hey, here’s a number, like 123. Well, why is it 123? Because there’s the ones place, the tens place, the hundreds place, and so forth. Hey, guess what? Binary is the exact same methodology, but you just have the ones place, the twos place, the fours place, and so forth. And helping people realize, like, oh, like that’s all it is, I think is very empowering because suddenly they feel like they’re part of the conversation and they actually know what they’re talking about all the more, even when they thought that was a domain for those more comfortable only.

51:00 — Growth vs fixed mindset

Ryan:

[51:00] I’m curious if you think that there’s anyone that’s just born that can’t learn computer science, and that’s not the world that I want to live in, but let’s just say you have someone and you explain a race to them a hundred times, and they kind of don’t get the concepts. Do you feel everyone can grow and learn computer science?

David:

[51:25] I think to an extent. I mean, if you, if it’s still not sinking in after 100 times, like, I mean, frankly, I think good educational advice would be like, maybe this isn’t the field for you, and this is how you’ve been empirically figuring that out. If it’s just not clicking in a way that’s exciting you, that’s making you happy, then, okay, stop beating yourself up. Like, find something else that’s of interest to you.

[51:43] But if you’re struggling and you’re thinking of being up after 2 times, 10 times, 49 times, I’m not quite sure where the cutoff is. Like, you should probably continue to try to power through, not necessarily by taking that same approach, but find some other class, some other professor, some other teaching assistant, some other book to try a different perspective or angle on it. Because maybe you’re just not jiving with, like, the form of instruction or the style of the teacher, myself included.

[52:10] So I wouldn’t give up thinking it’s you alone. And I do think, too, that’s really the secret ingredient. Is time. Like, fine, if you don’t. If you can’t quite finish the course in 12 weeks, take 24, take 52. Like, that’s perfectly fine. Especially if you’re juggling other things. This isn’t necessarily your primary focus. I think time is a perfectly reasonable knob to turn. And I think that puts it well within reach of so many other people.

52:35 — The future of CS50

Ryan:

[52:35] CS50 is beloved. People have. The reception online is so overwhelmingly positive, and you’ve been doing it for so long. What is next with CS50? Like, do you see yourself kind of of continuing the same curriculum for the next decade, or will things change?

David:

[52:54] Yeah, I think CS50 itself will continue to evolve as it. As it long has. I do think it’ll continue to change both in terms of tooling and concepts, as AI sort of permeates our lives and technology all the more. That said, I think it’s always meant to be this introductory course in problem solving. So I think there will be this backbone that remains and has been there even since my day. Even though the problem sets are very different nowadays, the style of the lectures is very different.

[53:18] If you really compare the syllabus from fall of 1996 versus now and what will be fall 2026 like, they’re not that fundamentally different. They’re just packaged very differently and stylized, I think very differently. But there’s a lot of knowledge and conceptual framework and practical skills that are very invariant over the years. What we have been doing in recent years, though, is steering in the direction of developing all the more of a complete curriculum.

[53:43] So courses that students can take for free online through OpenCourseWare after CS50. Earlier courses, gentler courses that students can take before CS50 itself, or even instead of CS50 itself, depending on the goals that they have and the background that they’re bringing to bear. The plan in the near term is to branch out beyond computer science, working with a colleague of ours, Tom Crawford, at the University of Oxford, on courses in maths, or maths, as he says, in all sorts of fields, so as to help uplift students who are taking, say, computer science courses, but don’t have the requisite math background that might be ideal to have, so that you don’t struggle with certain things that, that we might otherwise just take for granted.

[54:22] Beyond that, STEM more broadly is of interest. And while we’re not experts ourselves, working with colleagues of ours in the arts and humanities and trying to bring this form of theatricality and these memorable moments to a stage, but with far more people and subjects than me and mine.

Ryan:

[54:38] When you mention math and these other subjects, it kind of reminds me, I mean, I haven’t been in the schooling system for a long time, but that Khan Academy was like a very popular resource. I, I imagine that’s still around. And so that would be, I guess, a competitor.

David:

[54:53] I do think some healthy competition is good, as if it drives different courses, different teachers to sort of try new things and teach differently. But I think the more the merrier. The world only benefits from more educational resources.

Ryan:

[55:05] And so if those are, let’s say I’m okay, let’s say I’m a parent and I’m deciding, should I, I put my high schooler in the Khan Academy learning track of math or what you plan to offer?

David:

[55:18] From what I’ve seen, certainly of Khan Academy’s work is that they’re like ours, I think is more of a cast of characters, at least on video form, at least. I know the earliest of Saul Khan’s work was sort of the handwritten style for his niece or nephew, which was wonderfully compelling and engaging. I think we stylistically are just very different. And I think we try across CS50‘s courses traditionally to have a shared aesthetic, a shared pedagogical style, a shared lecturing style.

[55:47] So that when you take a CS50 course, so to speak, you sort of know what you’re getting because it’s very similar in spirit to the others in the ecosystem.

55:56 — Biggest career regret

Ryan:

[55:56] Coming to the end, when you look back on your career, what is your biggest regret?

David:

[56:01] I would say I wish someone had told me, or I wish I had known to just calm down. Like, at least when I was in university, I was one of these students who again came in thinking I was going to major in something else, government, in my case. I was one of those students who went to my first RA meeting, proctor meeting, advisor meeting with probably on a sheet of paper, all 32 courses I thought I was going to take at Harvard, which did not come to pass.

[56:25] But it really led me astray in some way the first year and a half of school, and that I was really focused on crossing requirements off the list, taking the courses I thought I should. And it wasn’t till sophomore year that I took CS50, which was eye opening and allowed me to start to pivot. But even then I hadn’t broken out of that mold probably until senior fall, senior spring, when I finally took an Introduction to Latin course, which was just for fun and was just so interesting and eye opening.

[56:51] And I finally understood aspects of English and Spanish retroactively that I never really had mastered when taking those same classes is I took a course on Dramatic Arts 101, which was a survey class on theater and writing and set design and acting. And pretty much every elemental aspect of theater was introduced in this class. And I wish I had thought to take a class like that. And even in graduate school I was Only gradually learning this lesson finally audited essentially a course in archeology which was so much fun, so much so I had this existential crisis, thinking maybe I should have gone to a PhD in archeology instead.

[57:26] But I stayed with CS and I just wish someone had told me to calm down early on and sort of explore more, particularly areas in which I thought I’d be less comfortable. And that’s at least a message I try to give to students nowadays. As for professional regrets, thankfully relatively few come to mind, if anything. You know, I used to joke, even with some of our own TAs, that insofar as I’ve largely stayed in academia, you know, I’ve never had a like a real job, but as recently as a recent sabbatical, I spent a whole semester or essentially interning with GitHub as a professor in residence and learning the ropes of a real world software engineering company and having access to the internal issues and bug reports and feature requests and sitting in as a fly on the wall to some of the team meetings and design chats.

[58:14] And it was so much fun to actually live the life that we’ve been teaching about for so long. And there too, I wish I had done that earlier or at least in parallel had done maybe more consulting opportunities or done a little bit more in the real world before coming back to school and then staying in school.

Ryan:

[58:32] Did you ship a project or like what was that time at GitHub? Like, what was the.

David:

[58:36] It wasn’t. I don’t think I contributed code. I definitely contributed issues and bug reports and feature requests. My sort of role was to be this educational voice at the table, especially in the design of certain products and to at least speak up on behalf of of educational use cases. Give teachers a voice at the table when it comes to certain features that would really make the student experience better, particularly in tools like VS code, albeit on the Microsoft side, or some of the internal GitHub tooling.

[59:05] So I don’t think you’ll find any code that I wrote, but hopefully one of my issues underlies some code that’s now in the code base.

Ryan:

[59:13] What about looking back on your time doing CS50? Like if you were to just reflect on what went well and what didn’t go well, just the highlights of, you know, your time running the course. What, what would you say?

David:

[59:28] I think it’s been very meaningful. So much so that it’s become our mission organically to have this now tradition of OpenCourseWare that wasn’t certainly our own idea. It was really inspired by our friends down the road at MIT who really championed this decades ago, but very happy with how that has evolved because it’s made the work all the more meaningful in ways that go beyond just education, but helping, as we’ve heard anecdotally from some of our students out there, very professional impacts, life impacts, being able to better care for your family or to pivot professionally into a field that you didn’t think was for you like that has just been very meaningful and rather makes it all worthwhile.

[1:00:09] And I hope too, in so far as we’ve been a bit unusual both with our production value and openness, trying to sort of offer an example that others could follow that, hey, maybe more people should be doing, doing this, more people should be sharing, leaning on us, us leaning on them. Because I do think that’s still a missed opportunity educationally.

1:00:29 — Top book recommendations

Ryan:

[1:00:29] What’s your best technical book recommendation? If someone wants to learn programming or if there’s maybe some other topic that you recommend a book for, I’d love to hear it.

David:

[1:00:38] My favorite book is Hitchhiker’s Guide to the Galaxy and I think everyone should read that or play the terminal based video game that existed when I was growing up of it. It funny enough, within CS50‘s courses it’s been a few years since we officially recommended any books. This is partly consistent with our commitment to openness and for everything being free so that there isn’t some financial barrier to someone out there to needing a textbook for the course.

[1:01:04] That said, over the years and to this day, some of my favorite tech books, even though they’re increasingly outdated, are this brilliantly colorful, picturesque, easy to understand and accessible book called How Computers Work, which we used in that course I first taught in 1999 and how the Internet Works. And I think they eventually stopped updating them some years ago. But they were such wonderful educational tools.

[1:01:31] I think every subject should have a book that looks quite like that. More arcanely, there’s a book that we still recommend to students unofficially or if they really want to go pursue something, something more arcane in C called the Hacker’s Delight, which is this wonderful hardcover book that just has really neat bit trickery and low level hacks, low level techniques you can use to make your C code especially all the more performant.

[1:01:56] So there’s a lot of value in a book like that. But I think back years ago to when I was learning some stuff in technology and like the four Dummies books was great. I think for some time we might have recommended one of the Complete Idiots Guides or something like that, which doesn’t have the best semantics. But the reality is they are by design written for a very broad, accessible audience, and I think that has a lot of value as opposed to the more traditional textbook where you sort of have to have taken the class in order to understand the book.

Ryan:

[1:02:24] Is there an official textbook for the Harvard CS50? Not the open one. Okay, so that one also has the.

David:

[1:02:31] Other than having access to a computer and the Internet, there’s no. No financial requirements.

1:02:36 — Advice for his younger self

Ryan:

[1:02:36] That’s great. I love that idea. Last question for you is if you could go back to the beginning of your career and kind of give yourself some advice, what would you say?

David:

[1:02:46] Explore more. I do think I wish I had more jobs under my belt early on, which I think was the result of my being pulled back into choosing to go back into academia. I think it would have been healthy if I had allowed myself maybe another year or two in between undergrad and graduate school. Or it probably would have worked out well if CS50 hadn’t opened up for me right as I was finishing my PhD, because then I probably would have gone off to industry for some amount of time.

[1:03:16] That said, not sure I would have gotten the job if I had wandered off campus at that point. So it probably worked out for the best. But I had such fun when I was interning, interning, so to speak, at GitHub, that it would be fun to live more of the lives that some of my friends in tech have lived.

Ryan:

[1:03:31] Awesome. Well, thank you so much for your time. I really appreciate it. David.

David:

[1:03:34] Yeah, thanks so much.

He ran promotions for AI infra and was refreshingly honest about promo culture, how big tech has changed, and org dynamics. We also went over the parts of his career that led him to Director at MSL.

Below are my main takeaways to save you time.

You can find the full conversation on YouTube, Spotify, or Apple Podcasts. The transcript is on Substack if you prefer to skim

Brought to you by:

Cursor3: a unified workspace for building software with agents. I used it to build tools for the podcast and shared the process in this episode’s ad read.

Takeaways from the conversation:

1) Promo culture in big tech and its downsides - John mentioned that promo culture became so strong that every single engineer he’d meet had the primary goal of promotion. This caused people to ship bad software just because it was their “promo bar”.

For instance, in some of his stories engineers were incentivized to ship software they would delete weeks after it shipped. In another story, people made their projects detrimentally complex just to show their work was sufficient scope. Also, orgs would cannibalize each other by stealing engineers through promising faster promotion paths.

2) How employee power has weakened – Pre-COVID, big tech cared a lot about keeping employees even when it was at employer expense. Engineers had lots of perks and optionality. Now the sense is there are too many software engineers, and maybe way too many if you count AI costs and productivity gains.

This has a lot of downstream effects. For instance, in reorgs, employee retention is a major factor when management is making decisions. If it doesn’t matter if employees leave, then management can be much more bold in making decisions that don’t favor the employees.

3) How to avoid playing the game - The main power you have is in where you choose to work. Even at a big company where promotion is a natural incentive that impacts everyone, there are still many teams that focus more on engineering excellence. Your career may grow more slowly but you’ll enjoy the path more.

He shared an example of how the PyTorch org focused on engineering first. Because of that engineers were almost always under leveled. But the benefit of that was the engineers in that org were more skilled and had better career mobility because people respected their levels more.

We talked about other topics too but these were the three I figured would be most interesting.

I’m going to start sharing quick summaries of the most interesting takeaways. If you don’t have time for the full conversation but want the key points, these should help. Let me know if you have feedback.

Also, a couple of operational updates:

1. Cursor is sponsoring the podcast! Huge milestone towards this podcast supporting itself so I’m not spending my savings to chase this dream.

2. I rebranded the newsletter to “The Peterman Post” so I can focus more on what’s useful and interesting to software engineers instead of just how to get promoted.

If you’ve been here since “The Developing Dev,” thanks for sticking with me. I’m keeping the domain the same for now, since I haven’t found a better one yet. Maybe someday if the podcast earns more I can grab a nice one.

Thanks for reading,
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

0:54 - Is he bullish on MSL

5:23 - Running promotions at Meta

15:15 - Growing at Meta

22:22 - Julia core language contributor

29:24 - Academics failing into industry

31:48 - Stats book recommendations

38:02 - Biggest career regret

41:05 - Advice for his younger self

Transcript

0:54 — Is he bullish on MSL

John:

[0:54] Oh, this is I guess like a more radical question. But I guess I will be honest. I am very bullish on Meta as a company that I am now a stockholder of, but not an employee. I am very bearish on Meta if you’re an employee who’s not a stockholder which just turns out to not be really anyone. But I think you can think of yourself as mostly employee if you’re mostly getting cash and mostly not equity. The more senior folks are the ones who are more mostly stockholder than employee.

[1:27] And I think that’s the tension. I do think, like my sense is that it’s enforced with. I don’t get the perception it’s unique to msl. My perception is that I think just Meta as a place to be an employee is less enjoyable than it used to be. But it actually is being run very effectively if what you care about is the bottom line of the business. And so I continue to invest in Meta and I suspect I will continue to invest because I do think it’s actually from a business perspective run quite well.

[1:55] But I do think pretty uniformly, I think it’s not unique to msl. I do think it’s a much more stressful time to be an employee there than before.

Ryan:

[2:03] What’s the part that makes it where you’re saying employees might not enjoy working there as much as they used to?

John:

[2:12] I think that, I mean, I think this is true of all of Silicon Valley. I think in general there was a lot of sense that it was incredibly easy to lose your glute employees and so you had to do everything impossible to sacrifice to retain them. And you are constantly constrained by an undersupply of employees. And I think basically everyone in Silicon Valley’s view, as far as I can tell, is that that’s mostly not true outside of a small set of AI researchers in Frontier Labs, where I think people do still behave this way.

[2:39] In fact, maybe behave this way more than ever before. But I think for everyone else, the general perception is the supply of engineers is way oversupplied. And especially I think, actually people’s concern is with AI, maybe they’re really oversupplied. I think what happens is a new market dynamic, which is like, I think as an employer, you’re thinking, I don’t actually have to make so many sacrifices to acquire and retain talents, and so therefore I’m going to make fewer of them.

[3:06] And this can play out in a million ways, but it can be like compensation, but it can also be things of like, do we do things that upset the employees or do we not? And how hesitant are we? How much do we give them a voice, and how much do we not give them a voice? I think that stuff has changed a lot in the years I was at Meta, but my impression is Meta is not in any way unique here. This is just a general property of all of Silicon Valley.

[3:30] But I do think as an employee, I think the labor supply and demand situation is super different from when I started. And I think it is a thing that is going to make generally being an employee rougher, period.

Ryan:

[3:44] It feels like it can be subtle as well. Like what you described of, I guess, employees having less leverage. You could see that affecting maybe even reorg decisions or things like that, where it’s like, if people leave, that is part of the calculus of the reorg and that is less of a downside now and maybe in the future. So I can see there being a lot of downstream effects where things just don’t go as well for employees.

John:

[4:12] Oh, yeah. I mean, there’s just so much stuff that in the previous versions of Meta, was done that arguably was a bad decision for the business, but made sense from the context of retaining talent. And I think that the company is just doing less of that. But on the flip side, I guess, to be clear, I didn’t say as explicitly before, I do think for most people, fundamentally, just the possibility might get laid off is the number one emotional thing that causes people stress.

[4:37] And living in a world where, you know, that has both happened recently and may happen again in the future, I think sort of probably beats out all the other questions of sort of lifts and food and stuff. I think prior to the layoffs, when those Things got taken away, people freaked out and were like, this is unforgivable. But then once they’re like, oh wait, actually you guys might just fire me. So now I’m much more tolerant of you taking away my food away.

[5:04] But I do think that fundamentally it probably drives for most people, the vast majority of it. And actually this probably is one of the things that makes say MSL more stressful than the other Frontier Labs is the sense that it might have layoffs in the sense that I think so far there have been fewer rounds or at least perceived to be fewer rounds.

5:23 — Running promotions at Meta

Ryan:

[5:23] and I think another thing that is oftentimes used for retention is the promise of growth or I guess career growth for an employee. And I know you ran promotions in AI Infra for a long time. I was curious if you saw things change over the years.

John:

[5:41] I observed a lot of divisions of Meta would really stop talking about, well, your comp is why you’re here and we pay you good money and you enjoy the work and that’s why you stay. And a lot of people are like, the only reason you do anything is because there’s a clear story about how it’s going to get you a promotion and you would like especially. This was really striking when I moved from Data Infra, whereas one of the places I was in earlier in Infra, moving to AI infra and I was in AI infra for a while before I came into sort of Pytorch, part of AI Infra and then later MSL.

[6:13] One of the things that blew my mind is there was not a single person I would meet on any team in AI infra whose first and foremost goal wasn’t promotion. This was a thing that came up in Data Infra but was not 90% of people’s attention. And when I started at Meta, it was just not ever a thing. Actually I used to manage one of your previous guests, Adrien, and Adrien and I were talking about career growth back in the day and one of the things that was really amazing was we started in this org growth that actually had never had anyone above IC7 ever in history.

[6:48] On the SWE side, to be clear, there were some other rules that had it, but at least this is what we were told. I actually don’t know for a fact it was objectively true, but we were told this by several people. I think in that world you weren’t focused on promotion, so it wasn’t a big thing. But then it became a thing where it was like this is the thing like you will one get more money, more promoted, but also you’ll have a title that you can use for your next job.

[7:13] And everything is driven by promotion dynamics. Actually, I would say the org I sound that’s by far most strong in was monetization, which is like when monetization would hire people out of AI infra, it would always literally be like, here is a very detailed plan of work you will do in order to get promoted. And. And that was a thing that worked very effectively. But it also my experience wrecked tons of teams and I managed a bunch of those teams now to fix them.

[7:42] But they were teams where basically everyone agreed the thing we were working on was bad. No one thought it would succeed, but people were like, oh, but I have to ship it because that’s my promo bar. And you get into the state where it’s sort of just impossible to even make good decisions about software anymore because the promotions were so important. And then I think what happened as the market became less pro employee is that people were like, oh, no, no, no, you should be afraid of being laid off.

[8:08] You should not be worried about the positive chance of a promotion. But I don’t think that was a good cultural fix. But I do think it’s been a bit of an attempt to undo some of the damage from before. But I do think that promotion mindset was incredibly intense everywhere. And ironically, it led to this thing which I found really troubling, which was the thing that would actually let you develop real skills that would do better in your career going forward.

[8:34] Increasingly got decoupled from the promotions. And I’ve seen a lot of people on Twitter over the years say this, and I find it very compelling, which is people are like, the main thing that’s wrong with the engineering cultures at the big tech companies is the promote culture. And I really agree. I actually think meta, ironically, my perception, seems to be doing better at this than many of the other companies that have even more formal processes and more anonymous parties involved.

[8:59] But I do think though, like, oh, what really matters is that you ran a rollout that affected 10 other systems. This causes people to not build clean systems. It causes them to build systems that maximally are coupled to the other systems in order to be able to hit the promo bar and that I think actually I met so many people who didn’t even like the work they were doing because they’re like, well, this is what will get me promoted.

[9:23] And so do you think that stuff was really unfortunate? And especially I think actually in the AI world has been especially unfortunate because it means unless someone is clear how using the AI tooling is going to get them promoted, they may not do it. But it is without doubt in my mind the only thing that’s going to matter for actual professional development and growth and actually being able to do more interesting work in the future.

Ryan:

[9:45] Yeah, I definitely saw some unusual behaviors, but we’re all trying to play the game because it helps people get promoted and you retain people and all that.

John:

[9:56] So, yeah, I mean, unless your VP is ready to fix that and totally shut it down, like, you have to play the game, it’s totally irrational not to play the game. And I think, you know, you either. If you don’t want to be in that, you have to leave the org. If you’re there, you’ve got to play the game. You cannot be the one who sort of unilaterally disarms. But it is, I mean, it was mind blowing to me to see this culture, I think was unique to both monetization, AI infra, but it was completely dominant in AI infra and it was really, it was not great for any parties.

[10:24] And funny thing is, it was like, it wasn’t even good for the people who were in it. Like they themselves had gotten to this rat race that seemed to be demoralizing to them.

Ryan:

[10:32] You saw this, I saw this. What could you do, though? When I was in it too, the people who were talking about it, they were not necessarily saying, yeah, this is good. They were saying, yep, I’m doing this that I don’t believe in, but we both know that I need to do this for this reason, so I’m doing it.

John:

[10:50] I think there are sort of two high sources of uncertainty that compete, and I don’t quite know how they balance out, but I think one is, I think people have a ton of agency in choosing the culture of the team they want to be on. And if they’re in a situation like that and they don’t love it. Meta has a lot of teams that don’t have that culture. There are a lot of teams like Pytorch was one of them, where people were just genuinely in it for the love of the craft.

[11:14] And people loved engineering as engineering in Pytorch in a way that I think was not present in a bunch of other parts of Meta. I think people wanted to come to Pytorch for that reason, and I think people came and were happy for that reason. So I do think people just have agency in that sense, which is like, yeah, within the context of that machine, you got to follow the rules, but it’s not like you’re forced to be in that machine.

[11:34] There were other rules. Not everyone would get them and Pytorch was very choosy. But I think many people could and it was worth doing. I think the alternative though was also even within a team, managers can differ a lot in how much they push on this. And I have personally at least been a person who I think mostly benefited from actually trying to bet on be on the stable team that will gradually succeed over time and will have a healthy culture and not collapse and not do things like over level ourselves.

[12:06] But it does mean that you get promoted slower. And this is where I think I do struggle with this a bit, which is, I think if you think of the day, what you really want to do is do something like compute your total sum of earnings over your entire lifetime. It may be better to be in the orgs that get promoted really fast and get fired really fast. But there are a lot of those orgs at Meta where people are like, well, they’re the stars.

[12:27] Oh, actually no, it turns out they’re terrible and lied to us. So we got rid of all of them. This happened a bunch of times in the years. It might be that that is actually economically rational, I’m not sure, but certainly I think from myself about emotionally rational and feeling like I enjoy the craft of stuff being on the teams. And Pytorch was like this. I mean, one of the challenges actually Pytorch had with hiring was actually the perception that Pytorch held a higher bar.

[12:53] People would be like, oh, I’ve heard you guys are actually like much tougher about promotions. And I’d be like, yeah, honestly, we are like, honestly, like our eights are like as good as you’re going to get anywhere in this whole world. And if you want to be the best engineer you’re ever going to be in your entire life, you should work with them. But like R8s probably are better than the 10s in a couple of other teams.

[13:05] If you want to be a 10, you maybe should be in that team instead. And I think for some people that would really turn them off. But I think part of this was again, the sort of agency and selection mechanism is I think Pytorch selected people who actually just loved engineering, as engineering. And then there were people who were willing to tolerate slightly fewer promotions. And ironically, one of the things I think was interesting though was that it became a bit of a magic thing that turned out well, which is because people perceive Pytorch to hold such a high bar, it was much easier for us to convince people outside of Pytorch that actually our people were ready for eight or nine or ten.

[13:51] Whereas other teams, when they tried to make this argument, they’re like, oh, you guys are not well known for holding a high bar. Maybe you guys are just overselling these people. But in Pytorch, people were like, oh, yeah, you guys hired our guy and thought he was pretty bad. So actually we think you probably are credible. And I think that again, it’s like, in the short term, doesn’t actually accumulate as fast, but I think in the long term does have a ton of benefits.

[14:15] Whether it is the absolute compensation maximizing algorithm, I’m not sure. And that’s where I have a little bit torn. But for me, I was also like, I was willing to get 20% less compensation to be in a place that I was more proud of, actually.

Ryan:

[14:31] That’s funny, because I remember someone from Pytorch would join one of our collaborations, for instance. You know, someone would join and go, oh, he’s a five, but really he’s like a seven. Like, he’s, he’s better than all of our engineers, but we don’t know why he’s a five, but he’s a five. That was not uncommon working with that. Or.

John:

[14:49] Yeah, I mean, I mean, again, this is like, I mean, it would come out like, flat out in like opportunity chats where we tried to hire someone and they’re like, hey, you know, Pyke is cool, but like, aren’t you guys, like, really under leveled? That would be the first question people would ask and have to be like, maybe, or maybe we’re holding the right levels. You gotta decide where you want to take a chance on us.

[15:08] But I think the flip side is we really did train people. A lot of people who were in Pytorch really learned to be remarkably good.

15:15 — Growing at Meta

Ryan:

[15:15] I know before Pytorch, you worked on some of the data and experimentation tools at Meta. How was that work and how was your experience growing in early Facebook before all this promo craziness?

John:

[15:28] Well, I came into a wild ride, which was actually, honestly an amazing experience and some of my happiest years of my life from my first two years at Meta. But also some of the most fearsome and scary years were also there. But I joined the team that I signed up for, and when I signed up, it was supposed to be called Data Science before I arrived because I asked for a six month leave to Work on Julia, the programming language I was one of the core contributors to.

[15:48] I asked for a six month leave between finishing grad school and going to Facebook at that time. During that six month period, the guy who hired me quit and then the team that was called Data Science that I was hired into got split into two teams, one called Core Data Science and one called Data Science Infrastructure. Then that itself became really tricky because I wound up joining Core Data Science but really loving collaborating with the Data Science Infrastructure.

[16:19] People who were the ones who owned the experimentation tools, working in the experimentation tools was honestly, I think to this day most people who know me from Meta are like, oh yeah, John was really helpful for that stuff. I actually think almost nothing I worked on as an IC ever went anywhere close to being as valuable to the business as the experimentation stuff. And I don’t think I was ever as good at any of the other stuff.

[16:43] I really loved being in that space and I think it was really influential to the business. That said I was on this Core Data Science team that was like an insane ball of stress. I joined, it was radically reorged, it had new managers, actually wound up really liking the new managers, but that was still a source of churn. But then a few months into it, someone who actually was on the Data Science Infrastructure team, which is particularly what’s amusing, but attributed his team to being Core Data Science because he perceived that to be sort of the team he really was on when he wrote this paper, published this paper in pnas, the Proceedings of National Academy of Science called something like emotional contagion in social networks that wound up just becoming the absolute singular worst piece of PR for Meta as a business that year.

[17:32] Just absolute disaster. I was really, really trying to prevent this from happening, but I didn’t have the authority to prevent it. But at least people perceive I was on the side who was not happy with this decision. But it meant that I was on this team where I was doing the Data Science Infrastructure work that was sort of very inward facing and very safe. But I was affiliated with a more researchy division that was publishing these papers that went from becoming sort of a PR win to a PR nightmare very rapidly.

[17:59] And that team, I think was one of the most formative experiences at Meta because it really was like, well, what happens if this team just gets fully disbanded? And this was in a world where there weren’t layoffs. It was like, well, Meta doesn’t do layoffs, but this team maybe has got itself to a state where it’s going to get laid off. And the guy who wrote this paper wound up having to do a company wide Q and A where effectively he sort of just apologized to the entire business as his Q and A.

[18:24] It was really just a mind blowing experience. And especially it was one where it’s like I came as an IC4 and all of a sudden we were in these meetings, we’re the head of legal being like, well why did you guys do this? And having to have these discussions with them on a regular basis and really trying to figure through what was the future of our team. But it was great. That blew over for what is worth.

[18:47] And then I spent several years just working on our experimentation tools. I was one of the main developers of Deltoid 3, which is I think now just called Deltoid because I think it’s been Deltoid for so long they just referred to it as Deltoid. But at one point it was like the third iteration and I worked a ton on that. And then especially it was a great example of how career growth can actually happen, which is I was on the team and then all of a sudden almost all the senior engineers left the team in the span of six months.

[19:13] And then I went from being like. And at the point maybe I was already in IC5 when they all left, I’m not sure. But I went from being one of the people on the team building experimentation tools to the only one who remembered how anything worked left. And suddenly I went from being sort of random IC5 to de facto TL for a bunch of stuff, which was actually an amazing opportunity for growth. And I think people understate how often these things can happen in tech.

[19:37] But it meant that I wound up really being able to drive a bunch of the vision for the A B testing tools for years, which were hugely successful Meta and it really was like some of the most fulfilling work I ever did on Meta.

Ryan:

[19:51] And just to give people context, Deltoid is the AB testing framework or the thing that you opt in one code path to A, one code path to B and you measure all the downstream benefits of ideally your test. Right?

John:

[20:05] Yeah. I think one of the things that’s actually kind of mind blowing to me is actually this is one of these decisions where maybe I did make bad career decisions, which is as far as I can tell, every time I’ve ever looked at it. The statsig product, which I now don’t know what its state is after they got put OpenAI is just like is Deltoid. This is one of the things where stuff in Meta when you’ve been in Meta, you work on these things.

[20:26] Like I also managed data swarm, but when people are like, what’s data swarm? It is literally airflow. And it’s not metaphorically airflow. The guy who wrote airflow built data swarm, quit, and a week later open sourced airflow. And the deltoid is not quite the same because it’s not the same people left and built statsig. But my perception is that for most people who will be watching this if they’ve ever seen statsig, my perception is that like almost the entire UI is the same, almost all the functionality is the same.

[20:54] It’s one of these things where I like, you know, probably should have built a deltoid startup many years earlier and I did not have the wisdom to do that.

Ryan:

[21:01] I think there was another one called Optimizely as well. Yeah, so I’ve seen many companies. It seems like a very repeatable playbook where you just take something that people take for granted that’s state of the art from a big tech company and you just give it to everyone in the industry and it actually creates like a billion dollar companies pretty. I mean, it’s hard, but at least the product market fit and idea part are relatively solved since it’s creating so much value for these big companies.

22:22 — Julia core language contributor

[22:22] You know, I saw before you worked at Meta, you were working on the Julia programming language and I actually wasn’t familiar about it. So I read into it a little bit and looks like it was part of these data science language wars, basically where there was R versus Julia versus Python. What is Julia and what is the context on that war there?

John:

[22:45] Yeah, well, certainly I think AI made it more of a part of the war. I don’t think it had to necessarily be part of it, although I do think also the simple fact of the reality is programming languages are products and products exist in an ecosystem where they’re in zero sum competition and claiming that they’re non zero sums. Competition is a very cute thing that people say is appropriate, but it’s clearly false and I think just makes everyone worse by misleading them.

[23:10] Julia for me and essentially sort of why did Julia so appealing? I mean, for me, what Julia’s pitch was, we should be able to write code in a high level language that looks like Python or like matlab, which is really the language it was originally designed to destroy. It was really designed to get rid of matlab. It was made by MIT math people who wanted to get rid of matlab and it really targeted that market much more than data science had started.

[23:34] And I think I was involved in pushing it towards data science. But to me, the thing I always do when I give talks about Julia is be like, listen, let’s look at the R function for distance. Like compute a distance matrix between a bunch of vectors, pairs of vectors, and you get all the distance matrix. If you look at that function and then you actually try to figure out how it’s implemented in R, what you find is C code that is very reasonable.

[24:01] C code that is just a bunch of for loops, loop through all the rows and all the columns and then compute the distance at that row and you’re done. If you basically take that code verbatim and just translate it naively into R, you’re going to take some type information away, you’re going to get rid of some ints and float signatures, but otherwise you’re going to basically write for loops that look exactly the same.

[24:23] VR code is going to be somewhere between 1,000 to 10,000 times slower than C. And this to me was the thing that just drove me insane where I’m just like, wait, what? These two programs are 80% the same. Why is one not as fast? And Julia really was all about this notion that that was unacceptable and that’s what made it so appealing. When the first post by the original founders went out, I was like, oh, you guys are doing the thing I wanted people to do, which is not claim that it is impossible to make high level languages fast, which is so much of actually how the Python R community sometimes behave is to be like, oh well, we can’t be fast, but also fast isn’t important.

[25:04] And to me that double hit of well, we can’t be it and is not important really didn’t work for me. So Julia really resonated. I think I probably is the guilty party of trying to make it more part of the data science wars, because I was myself a heavy user of R and was just so disappointed in R, just so incredibly disappointed in how often I would try to do a project. And R just like fought me at every step of the way.

[25:32] But Python is also like this. I mean, if you look at all the really great libraries like Pytorch, like, you know, deep down, at the end of the day you’re going to look at C code, or you’re maybe even looking at like handwritten assembly or handwritten like kernels for GPUs, or at least you’re looking at something written in a much lower level language. And so Julia was really about trying to solve that.

[25:51] And I don’t think it totally won, which I think is probably why you didn’t know about. I think it was very hip at one point and has become less hip, but it’s actually doing okay. I think it’s in the top 25 programming languages by users in the world. So I think it’s a real language that’s really out there. But for me, the thing that really matters, even though I don’t know that it’s killing it, Julia is like the only people still actually fighting that fight.

Ryan:

[26:15] Well, what’s the intuition behind why R is 10,000 times slower when the code is. The symbols are relatively similar to the

John:

[26:22] C. Fundamentally, any code that’s slow is slow because it’s doing stuff it doesn’t need to do. That’s just the most basic fact about slow code, is that the reason you’re slowed is because you could have done something else and you did something slower instead. And something like R is doing this. Pretty easy to see this in Python. It’s not quite as dire, but it’s still there is you wind up paying an enormous amount of overhead cost for the possibility that someone might do something more dynamic and because they might do it.

[26:50] And to give you an example, which is really astonishing about R is in R, for instance, the brace that you use to define a block is an operator that can be overridden and the user can redefine so they can make braces mean something else. So that means when you see a brace in code, you can’t just be like, I know what this is, I can move on. You have to be like, no, I need to look up and check, did the user redefine This I think it’s brace and not parenthesis.

[27:18] But it’s been a while, so I haven’t dug in. But it may also be parenthesis. Or it’s possible I flipped them. We can check offline and see whether my memory is good. But you just wind up with so much stuff like this that is so maybe changed and you don’t know whether it changed. So you need to go check whether it changed. And the checks are very expensive, especially if you’re doing something like adding two 64 bit integers.

[27:41] That’s like one machine cycle. Like just one machine cycle. But to check does addition still mean what I think it is? Could be hundreds to thousands of machine cycles. And so you wind up swapping in things that are very inefficient places that you don’t need. And this is particularly R is amazing. There’s this amazing paper by a couple of students and a senior professor named Jan Vitek. But it’s about the design called something like evaluating design of the R programming language.

[28:12] And one of the things they look at is especially R has an especially tricky thing, which is unlike Python, R is also a lazily evaluated lay language where the arguments to functions are not evaluated before you start the function body. They wait until the function kicks off and they just are passed as promise objects. And what they look at is they look at, well, how often are these promises could have been eagerly evaluated and how often is the overhead of these promises worth it?

[28:38] And their conclusion is like 70% or maybe more, maybe it’s 90%. I forget the numbers. You basically have no reason you needed to do this almost never do you need this. But you actually pay an enormous overhead cost for having agreed to do this. And a good example saying Python also has in Python you can manipulate the symbol table using functions in the inspect module. And so what that means is you can never be sure of what something’s bound to.

[29:06] You always have to be afraid and check and just sort of general the lack of invariance, that’s what makes a language fast. It’s like you have lots of invariants. What makes your language slow is you have lots of stuff you might have to go confirm at runtime and ours just incredibly pervasively like this.

29:24 — Academics failing into industry

Ryan:

[29:24] I looked at some of your popular past tweets and I thought maybe we could discuss some of them. So one of them, this is the most popular tweet that I think you ever wrote and you said that you’re continually, continually disappointed by how many grad students and postdocs get the impression that industry is a safe position of last resort and they can always fall back on if things sour in their academic careers.

[29:52] And I thought that was interesting because I thought the opposite was also very commonly true where people might want to avoid industry so they go and get higher education. So I’m curious your thought on this and what made you think this.

John:

[30:07] Really, what drove me nuts was there were just a ton of people who fundamentally wanted to be professors or postdocs and were in a PhD program. And they were like, well, if I fail out, I’ll go into industry. And this one is that you would interact with people during interviews who clearly didn’t want to be there, just so unambiguously did not want to be there and clearly viewed this as a failure that they were interviewing.

[30:35] And you’re like, well, that’s not really a positive sign that we want to hire someone who doesn’t seem like they’re going to enjoy the job. But in addition, a bunch of people. And this is what drove me so insane, because I think it’s like all parties involved in the academic system hurt students doing this, is that so many people just assumed that when they finally decided to get an industry position, it was going to be trivial, and then they didn’t find it trivial.

[31:00] I think a lot of academic people are like, well, smart people are in academia and the dumb people are in industry. So if I need to go compete with the dumb people, it will be easy. And I think there was a lot of that. But, you know, there was a person, I’ll give you an example. There was a person who was like, effectively a CS professor who I interviewed. And this person could not figure out how to pass values between the various functions that they were calling in the interview.

[31:23] Like, literally, they were like, what I would do is I would call this function, it would print out in the repl, then it would read it as a human, and then I would go type it into this other piece of code. And I was like, oh, you are better at programming than this, right? Because you’re professor of computer science. And they’re like, no, no, this is how I work. And I was like, oh, this is not going to set you up for success if we actually have to get you writing code and prod here.

31:48 — Stats book recommendations

Ryan:

[31:48] I saw a few other popular tweets that you had. They were about, like, favorite statistics papers and favorite recommendations of statistical books that you’re saying, oh, everyone’s got to read these. How come you have such strong recommendations on statistical literature? And then also, what Are those recommendations?

John:

[32:07] I love statistics. I think I’ll never not love it but I think it’s the craziest field. And what I mean by crazy is it’s a field that fundamentally sells people the idea that they can use statistical methods in real life but in reality what they do is do pure mathematics and study how statistical methods work in an idealized theoretical world. And in pure math. As an undergrad I did pure math and I loved things like number theory and pure math.

[32:34] You just period. It’s pure, you prove it. It’s internally coherent. There’s no attempt to reconcile with reality. Reality doesn’t even matter. You’re just like, there’s rules, we follow the rules. We’re in this internally consistent system. And then in super applied fields like software engineering you’re just like well the thing runs, the code runs. I don’t know what to tell you, I can’t prove this code runs.

[32:55] But we ran it in prod and it had eight nines of reliability which makes it better than most of the software everything for humans we’re good. Statistics is the super crazy field where you reason in about mathematics but then make all these claims about how it’s going to be useful to people in practice. And this I think is where opinions come in so strongly is that I think some people just are very, very honest and hold themselves to a super high bar.

[33:21] And some people I think are super cavalier about stuff and are roughly just like well I said it was true. And then you’re like well is it true? And they’re like and you’re like let’s really dig into the proof. And they’re like fine. One of their four people I love, love, love, love more than anybody is Larry Wasserman. I’ve never met the guy so I don’t know what he’s like as a human but his books are to me the embodiment of hyper intense honesty.

[33:48] He just seems like a person. Just like I cannot tell a lie and just like therefore everything you get from Wasserman is exactly true. He tells you exactly what he’s assuming. He tells you exactly what to imply. And he’s also extremely clear about being like I actually don’t claim these other things that you might want me to claim because they’re not true. And I think a ton of statistics books are not like that.

[34:11] A ton of statistics are like use our methods, they’re great. And so I think Wasserman is really at the top. Another book that someone recommended to me sort of halfway through my career a meta that I loved. I think it’s called something like Introduction to Agnostic Statistics is a book by a guy named Peter Aronau. And he’s, I think, another person likely sort of just like incredibly concerned with whether the things he says are true or false.

[34:33] And he’s hyper rigorous and hyper careful. And again, I think a lot of people in statistics are not hyper rigorous and hyper careful. So I love the book. So the recommendations I gave Wasserman, like, literally anything you can get by Larry Wasserman. Buy and read if you want to learn statistics. Like, I don’t think any book has ever been better than the books he’s written. In my entire life, I’ve never seen anything come close to being as good.

[34:54] The Peter Arnault book, I think, is probably as good as a thing you could ever read if you’re a social scientist or someone working more practically. It’s a little less math heavy than the Wasserman books.

Ryan:

[35:07] And I think there is a lot of value in big technology and understanding some statistics or doing it rigorously because I’ve been in so many A B test review meetings. And I think a lot of people who kind of just enter the industry and they see the UI and they go, I got a green bar here. Please give me the approval to ship my code path. But actually if you kind of dig in, you ask some whys and you’re like, wait, it was red yesterday.

[35:37] Why is it? What’s going on here? The understanding is very superficial and people are just trying to. And just trying to move forward whether or not it’s actually statistically beneficial across the user base.

John:

[35:53] Ironically, it’s a great example of everything we talked about today summed up as a story I have when we were trying to get Deltoid 3 to ship out. At that point, Deltoid 1 was still the default. And there was a person who came to us and this person was actually otherwise great. And I loved interacting with them, but this interaction was really. I was like, this reflects a lot of cultural pathology at our company, where they’re like, hey, I can’t let you ship Delta 3.

[36:15] And I’m like, why can’t we ship it? And they’re like, our holdout goes from being statistically significant win for the company to being not statistically significant until I think it’s a regression. And I was like, all right, it might be a regression, but maybe it’s also the truth. They’re like, I actually don’t know which it is, but you guys are going to wait until the half is over and we’ve decided we hit our goal and then you can ship Delta 3.

[36:40] And I was like, oh, but wait. I was like, your win is so close to the border between you did nothing and won that literally mild tweaks in our code have turned it off. I feel like that’s not a win. People be so concerned about, and especially this notion that you’re almost significant, so therefore you failed or you’re just barely not significant, win or fail. That, I think is this super dangerous culture.

[37:10] I mean, one of the worst things I ever saw was a team that I managed with Sweze, and one of the things they told me was they’re like, hey, you know, we have to do this thing because we made a goal. And I was like, okay, but what are we going to do next half? They’re like, oh, definitely on July 2nd, we’re going to delete this code. And I was like, wait, then why are you shipping it? Well, because we can’t miss our goals.

[37:29] And I was like, I feel like our goals should not be written in a way where shipping a thing we intend to delete literally a day later is a success. And they’re like, yeah, that’s fine. Fair. And I was like, what do you mean, fair? Come on, man, don’t do this. And eventually I convinced them not to do it. I was like, I’m the manager. I can decide the ratings. We don’t have to just do this. But people really firmly believe this.

[37:50] And I think the statistics thing, the problem is a lack of understanding of statistics bleeds into other weird pathologies of how people are evaluated and the two together become extra dangerous.

38:02 — Biggest career regret

Ryan:

[38:02] Coming to the end. Just a few questions, kind of reflecting on your career so far. Do you have a regret that maybe other people could learn from?

John:

[38:11] Oh, yeah. People ask me this so much over the years, especially as I became a more senior manager and then a director, that I have a keen answer because I’ve been asked this a million times. I, for my first several years, had Javi as my skip. For people who don’t know him, he’s currently, I think, the COO of Manta. But he was first ahead of growth and then the head of growth and ads. And then I think now he runs roughly 50% of meta.

[38:36] But Javi was my skip. And every time my actual manager, this guy named Danny, would be like, javi wants people to come to his office hours. No one shows up. And he feels like it’s a waste of his time. Someone should Go. And I basically just was like, well, I don’t have anything actually that valuable to say to Javi, so I’m just going to waste his time and I don’t want to be the person who wastes his time.

[38:56] So I never went. And I look back and I’m like, because especially as I started running office hours, as I had a large at work where I couldn’t do one on ones with everyone and had to do office hours and people wouldn’t go. And I was like, man, I wish people would come to my office hours. And Javi was also feeling this way. And he’s like, man, I wish I would give anything for one of these people to show up.

[39:17] But I just never showed up. And I look back and I’m like, first of all, this was an amazing opportunity that I wasted. But in addition, it’s not just that I wasted it for myself. I also probably just made his life worse off by not actually getting him to be able to take advantage of this thing he was offering us. And so I was like, this is a decision that basically harmed both parties that I did out of fear.

[39:40] I’m sure there’s probably a million other examples of me doing this that are not as clear in my head, but this is one where Danny probably came to our team once a week and told us to sign up and none of us ever did. And I think of this every time where I’m like, listen, if you know something important about the future of this org or this team, or even if you just honestly have any questions at all, if you have a leader who’s showing up and saying, I want to hear from you folks, go.

[40:07] I think especially if you’re a more junior IC watching this, I think it is hard to understand how painful it is at director and above to actually know what it’s like on the ground. You just look so removed from being an IC3 and IC4 anymore. Just so removed. And not just from their place in life, but also what’s happening to them, what’s true in the code base, what the team dynamics are. There were points where I had a manager.

[40:33] Managing managers, managing managers. There’s just so much indirection. I think people way, way should. More often, if a leader says, please show up and talk to me, do it. The flip side, obviously you come and say something weird that can be bad. Don’t show up and be like, I want you to tell me how to get promoted. It’s probably not your greatest first conversation. But if you’re like, hey, I Think this part of our org could be better?

[40:56] Could you help me? Leaders love that that’s what they want and like so many people are afraid to do it. And I think it basically makes all parties to ourselves.

41:05 — Advice for his younger self

Ryan:

[41:05] And then last question for you. With all the experience that you have now, if you could go back to the beginning of your career and give yourself some advice, what would you say?

John:

[41:14] It’s tricky because I think I am a person with very strong opinions, but I also think I can be more self conscious than I should. I do think that like have more confidence that you can do bigger stuff and that as long as you hold yourself to a high level discipline, the bigger stuff is really possible. I think when I was younger, basically at every step of the way from a high school student to a college student all the way, I think I tended to way too often cast doubt about what I could do.

[41:44] And I think that is the single biggest set of mistakes I’ve made is just this repeated pattern of not being ambitious enough or not believing something was tractable. And I think I’ve gotten a lot better. Why? I don’t like the defeatism and other programming language communities, but I think it for me is this thing that I think really could have been better. And I think it’s true for a lot of people.

[42:04] I think a lot of people, they over convince themselves of the greatness of other people and under convince themselves of how much they can achieve. And that combination means that they just way less try to do risky things than they could have. And I think they especially until you’ve interacted with enough people who have succeeded, I think it’s hard to realize how often they’re not actually doing that much better than you.

[42:29] They just tried and you didn’t try. So I think that is probably the biggest thing that if I could go back and tell 20 year old me probably is that.

Ryan:

[42:40] Thank you so much for your time, I really appreciate it John.

John:

[42:43] It’s been a real pleasure and thanks for having me.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

1:00 - Getting rejected from Princeton

2:53 - The software crisis

9:03 - The drawbacks of Python

10:17 - Getting into distributed computing

13:09 - Paxos vs Viewstamped replication

21:44 - The significance of Dijkstras letter

25:04 - Why she stayed in academia

30:39 - Why her award was questioned

Transcript

1:00 — Getting rejected from Princeton

Barbara:

[1:00] Okay, so this was when I got my bachelor’s degree and I applied to several graduate programs in math, which is what I majored in as an undergraduate. And I got this little card back. It was a postcard from Princeton saying, we do not admit women. Which was surprising. I knew they didn’t have women in their undergraduate program, but I hadn’t realized it extended to the graduate program. But it was how it was.

[1:33] So what happened was I did get into Berkeley, which is where I did my undergraduate work. But I decided I really wasn’t ready to do a PhD in math and that I should get a job instead and just sort of see, you know, see how things were. And the best job offer I got was as a programmer. So that’s how I got into computer science. By a happy accident.

Ryan:

[2:00] You were consistently, at least at Berkeley, you were consistently a top student. So it is odd that you wouldn’t even get an opportunity to play in some cases.

Barbara:

[2:11] But that’s how it was back then. It was just. I hadn’t realized it was more than an undergraduate thing. And Berkeley was co ed. So, you know, there. What I found was there weren’t very many women in my classes. There were lots of women students, but very few of them majoring in math. And there were only maybe a couple of women in my classes. But the idea that a door was shut and women couldn’t do things, that was not what went on at Berkeley.

[2:42] So it was a different environment. Of course, nowadays in the top schools, the women are about 50% in computer science.

2:53 — The software crisis

Ryan:

[2:53] I want to talk about some of the core problems that you were solving in your career. And so I understand that there’s the software crisis in the 1970s, and I was wondering if you could give the context behind what the problem was at that time.

Barbara:

[3:09] Well, it was a huge problem at the time because people did not know how to build big programs that worked. And so you would often pick up the newspaper and see an article about some company that had spent millions of dollars and hundreds of man years developing some software system for their company. And then in the end they’d have to throw it away because it simply didn’t work. The problem was that to get a big program that works, you need modularity, and you need to break your program up into small pieces.

[3:52] Each piece provides an interface with a hopefully complete description of what service it provides for you. And then inside is an implementation that’s hidden and nobody pays any attention to it on the outside. And if you have a system organized like that, you can actually reason about its correctness one module at a time. But in those days, people didn’t know how. They couldn’t figure out how to design systems that were modular.

[4:25] And the only kind of modularity mechanism present in programming languages was a procedure. And procedures didn’t match the kinds of modules you needed. Because if you think about a file system or a database or, you know, Amazon or whatever, you know, they. They aren’t a procedure where you put something in and get something out. They’re a much more complicated thing. So there was no notion of a module that sort of matched the kind of things that people were looking for.

Ryan:

[4:58] So when you saw that problem, what were the early solutions like?

Barbara:

[5:02] Well, so people were proposing modularity and they were talking about how important modularity was. They didn’t exactly know what a module was. And so it wasn’t like they could give you a rule, this is a module. It was just a chunk of code. In fact, there were papers written that talked about how big a module should be and stuff like that. Just a chunk of code that’s not going to work because you need to to design these systems.

[5:28] So you need a way of thinking of a design that sort of fits into this notion of modularity. They also didn’t really know what the rules should be for modularity. And it turned out that in some of the early work I did after grad school, when I was working at mitre, I had already invented a notion of modularity that was sort of more complete than what people had been talking about. Just because I had a small team of programmers, we were building a complicated system and I wanted to keep us out of trouble.

[6:02] And so I had that sort of sitting there when I started to work on this topic. And it enabled me to see this idea of data abstraction. All of a sudden I sort of saw that this thing, I had this kind of modularity mechanism which consisted of basically what I just described, a bunch of code providing you with an access through a number of what I called operations, so you could call it in various ways.

[6:31] And then inside was all hidden, and whatever data it was using was not accessible to the outside. And then at some point I saw I could see this as a data abstraction. It could be a set, it could be a sequence, it could, you know, and so forth. And that meant we had a new type of module. Now, when I look back at the papers from the time I see that idea, it’s almost there, except people hadn’t managed to pick it out.

[6:58] And so it was probably. I think this happens in science a lot. There’s sort of a time when an idea is ready and I happened to see it.

Ryan:

[7:07] You pieced these together and you put it into the CLU programming language that you’re working on. How did you see it influencing the industry?

Barbara:

[7:17] The first thing that happened was I wrote a paper with a man who was a graduate student at MIT, Steve Zilles, about this idea of data abstraction. And it sketched a notion of what abstract data types would be and how a programming language could support them. And this was very impactful paper. And so there was a big impact on the research community. And then CLU came along, and that involved a lot of additional research in the programming language area.

[7:52] The next thing that happened, so people were watching this in the research community, but of course, people who are in companies that want to write programs, they need a programming language. And I decided I wasn’t going to try and turn CLU into a product because that would have required working in a company in those days. You didn’t just put software out on the Internet and people used it. There wasn’t an Internet yet, for one thing.

[8:22] And I was much more interested in doing research than in working in a company. So I put CLU on the side. It had a user base. But for companies to use a programming language in those days, they wanted a company behind that language. The next thing that happened was the government put out a call for a programming language they could use. This led to the Ada programming language. So that was already a big impact.

[8:52] If you think about it, there was a language explicitly being designed to have data abstraction in it. And then finally in the 90s, Java came along.

9:03 — The drawbacks of Python

Ryan:

[9:03] I thought it’d be interesting because you worked on CLU, you designed that programming language to ask you about other programming languages. You said somewhere that there’s something wrong with Python. I was curious to hear your thoughts of why.

Barbara:

[9:15] Python has modules. But it doesn’t have encapsulation, so it allows code on the outside to muck around with what’s going on on the inside of a module. And that’s all I was talking about. Encapsulation is a crucial part of making modularity work. And when you’re building big programs so you have many programmers working on them, your team is really only as strong as your weakest programmer. So it’s nice if the compiler can enforce things and make certain kinds of bad behavior not possible.

[9:53] I mean, Python is, you know, has another intended use. It’s helping naive programmers learn quickly how to write programs and stuff like that. And people in the programming language world do think about issues like this, like, you know, how to make languages safer and so forth. But since I stopped working in that area, I’m no longer an expert in programming languages.

10:17 — Getting into distributed computing

Ryan:

[10:17] What got you into distributed computing?

Barbara:

[10:20] I read a paper by Bob Kahn, who was with Vint Cerf, considered to be, you know, the founders of the Internet. And Bob talked about his dream of distributed computing, where you would have a program composed of pieces on different computers connected by a network, and nobody knew how to build those programs. And so I just thought, great problem, and I was looking for a new problem. So I jumped into distributed computing and the first project was actually a programming language to write distributed programs in.

[10:57] And then I started looking at other problems in the distributed systems area.

Ryan:

[11:03] What was that first programming language?

Barbara:

[11:05] It was called Argus. It was very strongly influenced by CLU. It was an object oriented language, had a special kind of object called a guardian, which was a module sitting at a single computer. And then guardians could compute, could communicate through remote procedure calls. One of the things you run into in distributed computing is if you have a computation that starts at one guardian and then makes use of other guardians at other nodes in the network, in the end you want that computation to either complete entirely or have no effect at all.

[11:44] And how do you do that? Well, I borrowed the notion of transactions coming out of the database field, and that was part of how Argus worked. It ran computations as atomic transactions.

Ryan:

[11:57] Oh, interesting. Like distributed transactions across those nodes.

Barbara:

[12:02] Yeah, And I think it led right into the work I did on Viewstamped Replication, which was. That was the beginning of cloud storage. I was thinking in terms of a file system, but it doesn’t really matter what it is. You know, how do you have data out on the Internet stored at multiple sites with correct behavior and always accessible as long as enough nodes are up and running and the network is working.

Ryan:

[12:32] I see. And what is a viewstamp in this context?

Barbara:

[12:35] Oh, that had to do with some details of how the system worked. And it was a way of noticing when some nodes failed and other nodes had to take over. You know, could figure out which ones had the most recent state in them. So you could pick up and not lose anything that had happened in the past.

Ryan:

[12:54] What if the clocks are out of sync?

Barbara:

[12:56] It had nothing to do with clocks because it was just numbers. In other words, we were in view 25, the next view was 26, so.

Ryan:

[13:05] Oh, okay. Just incrementing some numbers passed around.

Barbara:

[13:08] Yeah.

13:09 — Paxos vs Viewstamped replication

Ryan:

[13:09] This reminds me a lot of Leslie Lamport’s work, actually.

Barbara:

[13:13] Yes.

Ryan:

[13:14] Did you ever work with him or.

Barbara:

[13:17] No. Leslie and I developed what is essentially the same idea independently and he had the system he called Paxos, and I had this thing called Viewstamped Replication.

Ryan:

[13:28] Oh, interesting. And they’re essentially the same thing.

Barbara:

[13:32] They are, yeah.

Ryan:

[13:34] Are there pros and cons of the two approaches?

Barbara:

[13:37] Yeah, I mean, when you do this kind of system, there’s lots of little details you can play around with, so you could, you know, decide to do it. Gets technical, but, you know, there’s tiny differences. But no, they’re basically the same. In fact, what happened was the first time that I’m aware of that this was actually used in a real system was when the Google file system came along and one of my former students, who was, I think he was a consultant at Google, looked at what was going on in there and he said, oh, he says that’s Viewstamped Replication.

[14:11] So. So the people at Google seem to think it was Paxos, but in fact they are the same system.

Ryan:

[14:17] I also noticed, like when you were, when you came up with abstract data types, maybe it’s just because there wasn’t the Internet wasn’t, as you know, wasn’t there that there was also the object oriented stuff going on on the West Coast?

Barbara:

[14:33] Yes. Alan Kay was developing SmallTalk at the same time that I was working on CLU. And then at Carnegie Mellon, Bill Wolfe and Mary Shaw were working on alphard, which was another data abstraction language. And you’re right, there was no Internet, although there was the arpanet, you know, so we did, but it was like there were two independent streams of research going on and we weren’t talking to each other and so I wasn’t paying much attention to small talk.

[15:04] And on the West Coast, I don’t think they were paying much attention to data abstraction, and this led to the Liskov substitution principle, which, I mean, the story is a sort of a cute one, because in 1986, I think it was, I was asked to give a keynote at uppsla. OOPSLA is the Object Oriented Programming Conference. And this was the second year it was happening. And so I decided, I think I’ll read all those papers about SmallTalk, and there were other languages being developed that were based on SmallTalk and see what’s going on with them.

[15:41] And I discovered. So SmallTalk has this idea of inheritance in it, where you can have a class and a subclass which borrows from the way the class is implemented and changes it a bit. And CLU doesn’t have that CLU, just has what we call clusters, and they’re all independent. So I saw that they were talking about this notion of classes and subclasses, and then I saw that they were also talking about something they called type hierarchy, where they wanted the type implemented by a class to be related in some way that they didn’t understand to the type that was implemented by a subclass.

[16:24] I really thought about modules in terms of their specifications. This partly had to do with a class I developed at MIT that I developed jointly with my colleague John Guttag. And in that class we taught the students how to do design about modularity, data types and so forth, but also how to write specifications, how to reason about correctness. And so it had a big focus on think about the meaning of things first.

[16:52] And the implementation is something that’s kind of hidden inside and you don’t worry about it very much. And so that was a different way of thinking about things than what was going on on the west coast, where they were really thinking in terms of classes and subclasses. And I even saw papers where they would describe the behavior of a class by explaining how its implementation was different from the implementation of the superclass.

[17:18] So they were kind of focused on implementations in a way that we weren’t. And so when I read these papers about type hierarchy and I saw they just couldn’t figure out what it was supposed to mean, I was thinking about it from the terms of meaning. And so I was able to say it has to do with the behavior. And this subclass better behave like the superclass if you use it in an environment where the superclass is expected.

[17:47] And that became what ended up being called the Liskov substitution principle.

Ryan:

[17:59] it get that name? Because you didn’t go off on stage and say, hey, everyone, here’s the Liskov. No, no.

Barbara:

[18:04] But no. As I said, one day in the 90s after the Internet had arrived, I got an email saying, can you say if this is the correct interpretation of the Liskov substitution principle? That’s when I discovered that there was such a name. And I don’t think I had really been aware of how important it was until that point, because I wasn’t thinking about this stuff. I was working on other stuff.

Ryan:

[18:32] In the academic community, it seems very reasonable to have multiple people have similar ideas. But it seems like some ideas stick more than others. For instance, like the Viewstamped Replication versus Paxos. They’re the same thing. I’m wondering, like in that case, for instance, why would Paxos be more known than Viewstamped Replication?

Barbara:

[18:57] Yeah. So what Leslie says is, I went around giving all the talks and she implemented it. Really something to that.

Ryan:

[19:11] Then he got notoriety for speaking about it.

Barbara:

[19:16] Yeah, he did give lots of talks about it and wrote several papers and so forth. And I didn’t realize it was the same system, but finally it became clear that they were the same system.

Ryan:

[19:28] I also wonder how much the name matters, because Paxos is kind of catchy.

Barbara:

[19:33] It is a cute name.

Ryan:

[19:34] It’s a cute name.

Barbara:

[19:35] Yeah, Right.

Ryan:

[19:36] So I could see maybe it has better marketing around the idea, I guess.

Barbara:

[19:41] I also think that this approach to solving this problem, which both Leslie and I picked up independently in my case, came from the work I’d been doing with transactions. Because in transactions there’s a leader that says, now we’re going to commit and ask everybody, can we commit? And if they all say okay. But unlike in transactions, in transactions, if the leader fails, there’s what’s called. This can be a real problem here.

[20:12] We had to have a way of moving to a new leader if the old leader failed. And that’s the big step forward. That happened in both the Viewstamped Replication and Paxos. Now there was also Byzantine fault tolerance, which came along later. That was developed. So view stamp publication. I developed with my student Brian Oki. It was his PhD thesis. And then Paxos I developed with my student Miguel Castro was his PhD student thesis.

[20:43] And Miguel got interested in this problem because there was a DARPA request for proposals and there was one about this problem on the Internet. By then there were Byzantine attacks and malicious attacks and nodes that would purport to be working correctly, but actually they had been compromised and so forth. And so Viewstamped Replication and Paxos only handled crashes. And if there were messages that had been played with, you could tell when they arrived that they were bad.

[21:18] And that was about the extent of what we dealt with. But these malicious attacks with nodes that purported to be working when they weren’t, that was a big step forward. And Miguel saw this request for proposals, and he said to me, why don’t we see whether we can come up with a protocol that works in the presence of Byzantine attacks? And I think, by the way, Leslie is the one that invented the word Byzantine too.

Ryan:

[21:43] I think he did, yeah.

Barbara:

[21:44] Right.

21:44 — The significance of Dijkstras letter

Ryan:

[21:44] I saw in the software crisis stuff, there’s the paper with Dijkstra that you had mentioned was one of the papers you wrote that said need modularity. And that paper was the go to statements considered harmful.

Barbara:

[21:59] Right.

Ryan:

[22:00] And I’m curious because Dijkstra, I mean, when I was studying computer science, we all know his name. Did you ever meet him or work with him?

Barbara:

[22:08] Oh, yeah, many. I never worked with him, but I did meet him multiple times. And, you know, he had very interesting ideas. And that paper, go to statement considered harmful. It’s not actually a paper. It was a letter to the editor of the communications of the ACM. But it was very impactful. And what was really important about that paper was that Dijkstra was talking about how difficult it is to reason about the correctness of code.

[22:42] And this was at a time when, in the sciences, computer science was kind of dismissed as nothing much and anybody can write code. And, you know, Dijkstra was trying to make a point, which I think was actually important, that it’s not as trivial as you think it is. It’s not trivial at all. And he was also pointing out that Go Tos can be misused.

Ryan:

[23:08] And it was controversial at the time.

Barbara:

[23:11] It was, yeah.

Ryan:

[23:12] But today I’ve written code for over a decade.

Barbara:

[23:17] People don’t use Go Tos.

Ryan:

[23:18] I haven’t even seen a Go To. So why was it controversial at the time?

Barbara:

[23:24] The programming languages were different then. First of all, there were people writing programs in assembler. In assembler, you have to use Go To. Programming languages didn’t have some of the constructs in them that we think of today. And so some people were using Go Tos because they had to. And also compilers didn’t do all the kinds of optimizations that they do today. So there was a concern if you didn’t have Go Tos, maybe your program wouldn’t be efficient enough.

[23:57] And then there were people who used gotos and wrote really good code and they were offended that Dijkstra was saying, your code is bad. So there was a whole bunch. And Dijkstra was not the most diplomatic person, so he didn’t write it in a nice. You can imagine writing that paper more nicely where you said, but there were many reasons why it was controversial. People were offended. But then there were also concerns about programming languages don’t have these features.

[24:33] I need, what am I supposed to do? And then there were concerns about what the compiler was doing. And so the world is very different now. But clearly Dijkstra won the day here because no, go to.

Ryan:

[24:48] And is Dykstra in person also like his writing?

Barbara:

[24:55] He was not always as tactful as he might be. Yeah, but, you know, he was a very distinguished researcher.

25:04 — Why she stayed in academia

Ryan:

[25:04] Computer science has had such a huge impact on the industry. Why did you choose to stay in academia instead of going into industry?

Barbara:

[25:14] So I like doing research and I enjoy working with students. And teaching was never my favorite thing. But I always felt teaching and research were very closely connected. But also it was a different time. So this business about how professors are all forming companies and so forth, which goes on today, that wasn’t happening 20 years ago. Or maybe it was happening 20 years ago, but 30 years ago it wasn’t.

[25:48] And so when I was young, it wasn’t the thing that you did. It was sort of a either or sort of thing. So I wasn’t even thinking about doing stuff like that. I did work in a startup briefly at the end of the 90s. I didn’t like it. I much prefer doing research. And as a professor, you have this. It’s a gift and a curse. The gift is you can do whatever you want. The curse is you have to figure out what it is that you’re doing.

[26:19] But I like that freedom and the fact that I just could go off in any. I mean, my career is full of these interesting zigzags where I would switch to something else. I had the freedom to do that.

Ryan:

[26:32] What stops you from going off in a very useless direction?

Barbara:

[26:36] You won’t get tenure.

Ryan:

[26:40] Who’s the judge?

Barbara:

[26:41] The community. Yeah. It’s not, you know, the way that you’re viewed at your university has mostly to do with one very important facet of it is how are you viewed in your research community? So. Because that’s one of the important things they want from their faculty. If you’re in a research university.

Ryan:

[27:07] You mentioned earlier that research and teaching were heavily related. In your opinion. Why is that?

Barbara:

[27:15] Because when you teach, you have to teach from first principles. And if you’re doing good research, you need to understand deeply what’s working, what’s not working, what assumptions you’re making. It’s really the same thought process.

Ryan:

[27:32] It sounds like in research, I mean, this is true in every walk of life. There’s the, I guess, the direction that you take and then the work that you do towards that direction. And it sounds like for research, the direction is maybe the most important thing because you could have a phenomenal researcher doing an idea that has no, you know, even if you did it at 100%, it’s not going to lead to anything.

Barbara:

[28:00] That’s right. So you, you know, you tell students, graduate students don’t do incremental work. Don’t, you know, you’ve got, you. You can’t just keep working on the same thing over and over, just making little teeny improvements. You need to find. You’re right, you have to find a good problem, but you also have to find a problem that’s amenable to a solution and one that matches your skill set. And you have to recognize when you’re going in a good direction versus not going in a good direction.

Ryan:

[28:32] Looking back on your career, you say that it was luck and that it seems like things just fit together.

Barbara:

[28:41] I feel there was luck involved. There was a lot of hard work involved. There was not allowing something negative to cause you great difficulty. So, for example, when I finished my PhD, I would have liked a faculty position, but I didn’t have any good offers. So I went back to mitre, the company I had worked for initially as a researcher. In other words, I just kept on marching along. And it turned out to be a very good choice because I was switching from AI to systems.

[29:26] And this gave me. I worked there for four years. Gave me four years to make that transition without having to worry about students and teaching and all the other stuff. So it worked out well. But it also meant that I didn’t feel really sorry for myself that I didn’t get a job. It meant I just kept on going. And then Title IX was about to be passed, and finally academia was opening up to women and I was ready.

[29:58] But I think when I talk to other people about their careers, they talk about doors opening and doors closing. So I didn’t get the academic job I wanted, but I had this other opportunity. I majored in math, and fortunately I was offered a programming job. I mean, doors opened and then you have to decide, am I going to step through? And this is a pattern that people see in their careers. It’s not just me.

[30:27] Many people have. Have those kinds of choices that you make along the way. Opportunities and things that aren’t so great, you know, and you’re sort of working your way through.

30:39 — Why her award was questioned

Ryan:

[30:39] I’ve interviewed a few people who have won Turing Awards, and I’ve done research and watched a lot of these speeches that people give after they receive the award. And I noticed something different in your speech or some of the stuff that I read about your work, which is that in yours, it seems like when you got the Turing Award, there were people saying, why did she get the Turing Award? It was kind of negative commentary on that.

[31:10] And I didn’t see that in the other cases. Why do you think that they said that about your work?

Barbara:

[31:16] So I’m not so sure. This doesn’t happen to other people, by the way. It was just that my husband was out on the Internet, by the way. There was an Internet by then.

Ryan:

[31:24] Okay, okay.

Barbara:

[31:25] As you know, the Internet is not necessarily nice.

Ryan:

[31:29] I know that.

Barbara:

[31:30] Right. And so I can imagine that other people might have had a similar experience had they bothered to look. But I also think that the world had changed so much and the work that I had done was so basic to what was going on that people really didn’t understand that there was a before. I mean, my graduate students really didn’t understand that there was a before because I discovered that at the time I got the award, it really hadn’t struck them that data abstraction didn’t always exist.

[32:13] So I think that. I mean, I viewed it as, in a way, a huge compliment, not really just to me, but to the group of researchers who all together got us to the point where we had these modular programming languages and we understood about abstract data types and we understood about how to reason about correctness. All that stuff was so fundamental that people thought there was no time when it hadn’t already existed.

Ryan:

[32:44] So they took it for granted.

Barbara:

[32:45] They took it for granted. I mean, the Byzantine fault tolerance, you look at that there’s a protocol, and you can understand that there was a protocol. Somebody invented that protocol, you know, so you get the credit for that. This was much more fundamental. This was the whole mindset you had about how to develop programs, and that’s what people were talking about. And things had changed so much that all these inventions of mine and others were just there in the.

[33:17] Would work. And people didn’t realize that there was a time before.

Ryan:

[33:21] That’s validating then. Yeah, it’s. It’s so impactful, and everyone uses it all the time. That they don’t even know what it was like before what you did.

Barbara:

[33:30] So I thought, you know, it’s a huge. The person who said it didn’t mean it nicely. But I thought really, it’s a huge, A huge compliment and a statement about where we are now as opposed to where we were then. So.

Ryan:

[33:44] Okay, well, that’s all the questions I have for you. Thank you so much for, for your time. Really appreciate it.

Barbara:

[33:49] You’re welcome.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

1:03 - How he got into databases

6:43 - Competing with Oracle

9:07 - What made Postgres special

15:55 - One size fits none

21:37 - Why he disagreed with Google

29:14 - Why he chose academia over big tech

30:58 - Replacing state in an OS with a DB

42:02 - Future problems in databases

51:36 - Technical book recommendations to learn databases

52:20 - Advice for younger self

Transcript

1:03 — How he got into databases

Mike:

[1:03] When I graduated I had the good fortune of being hired at Berkeley and I, it was clear I had to, you know, continuing what I did for my PhD was not going to go anywhere then as well as today. You’re way ahead if you get adopted by a mentor who knows the ropes. So Gene Wong, who, who is still alive and still, still kicking, took me under his wing and said well, let’s do something together. And this was 1971, which was the year after Ted Cod wrote his pioneering paper in CACM.

[1:50] Gene Wong said well let’s, let’s take a look at database stuff. And at the time the competitors were a thing called the codicil proposal which you’re probably too young to have ever heard of. And so it was a low level spaghetti network proposal where you executed queries by following pointers. And then the, the alternative was the IBM proposal which was a higher, a thing called ims, which is still available.

[2:25] And it’s hierarchical data, it’s a tree. You organized your data as trees. And even at the time IBM realized that trees were not general enough to, to solve many people’s problems. So they hacked on a, a way to make it a limited network structure. So it was clear that was a horrible hack. The codicil proposal had all kinds of bad properties besides being low level and, and really hard to debug. It also had the property that if anything changed in your, what’s now called your schema, you basically had to throw away everything and do it all again because it was absolutely rooted at the physical level, whereas Ted Cod stuff made perfect sense.

[3:22] And so Gene said well let’s build one of these puppies, that’s clearly the next thing to try. So we started building ingress in 1972 while I was an assistant professor at Berkeley. As you know, if you’re an assistant professor, you have to, you have about. You get five years to prove that you’re a big shit, and they fire you or they give you tenure. So Ingress was my ticket to getting tenure, which happened in 1976.

[3:55] That was where it started. And then again, you know, happenstance at the time. You know, a lot of people would build prototypes which were sort of studenty, like code, which means you could get it to run, but if you gave it to anybody else, they couldn’t. So we put in the first 90% to get something we could run. And then for whatever reason, we put in the next 90% to get it to where it really worked.

[4:29] So the University of California version of Ingress was really worked. And so over the next couple years, about 100 universities started running it because UNIX became the big thing. And so this was a database, a free database system that ran on Unix. And so it was quite popular in the academic world. And so we got started getting, you know, lots of visitors at Berkeley who would say, gee, this is really nifty looking stuff.

[5:06] What’s the biggest app, biggest anchor application you have? And we’d be forced to say, not very big. And so this was brought home in spades when Arizona State University considered running Ingress on their student records data, all 40,000 students worth. And they could get over that. They had to get an unsupported operating system from Bell Labs. They could also get over. They had to run an unsupported, unsupported database system from these guys at Berkeley.

[5:49] But the project went down in flames when they realized there was no COBOL available for Unix. And they were a COBOL shop. So, unsupported operating system, unsupported database system. No, COBOL doomed us to, you know, irrelevance. And it was clear the only way out of that was to start a company. And so in 1980, we got Venture capital as it existed then and started Ingres Corporation to move ingres to dex VMs, you know, a real operating system.

[6:36] And we had a real company that would support Ingress. And that was the start of the commercial journey.

6:43 — Competing with Oracle

Ryan:

[6:43] I saw that Ingres was competing with Larry Ellison’s offering at Oracle. Yes, I saw that Ingress was, was certainly better than what they were offering, but they were still competing somehow. How? How did they compete?

Mike:

[7:02] Larry Ellison is a fabulous salesman and he, at the time he, he was, he made present tense and future tense indistinguishable. And so he basically lied to customers. He would ship stuff that didn’t work and have his initial customers help him debug it. So I think he, he engaged in what I consider very shady business practices. But lying to customers, I think is, is, you know, unconscionable. So for instance, there was a thing called referential integrity, which is if you, if you fire an employee and he’s the last person in a given department, do you want to delete the department or do you want to have it be a department, a ghost department?

[8:01] It’s, it’s all that kind of stuff. And so Ingres Corporation implemented referential integrity. Oracle Corporation wrote two manual pages that said, here’s the definition of referential integrity, which everybody agreed to. And then, then down at the bottom it said not yet implemented.

Ryan:

[8:25] Interesting. Yeah. I had interviewed someone who worked at Sun Microsystems and they had a similar opinion that they, Larry Ellison was a little bit shady. So it seems to be a commonality. I also saw somewhere else in something that you had said was that when Oracle acquired MySQL that everyone kind of got afraid of that and moved to Postgres.

Mike:

[8:55] That was the genesis of Postgres replacing MySQL as the preferred open source relational database system.

9:07 — What made Postgres special

Ryan:

[9:07] So you created Ingress, and there was a lot of technical innovations in it so that it was better than the incumbents. But ultimately it went away and you developed postgres. What was the thing that Ingress didn’t do that Postgres would do?

Mike:

[9:24] The big thing that guided us at the very beginning was the original reasoning for the academic version of Ingress was we were going to support a geographic information system that the neighboring professor Praveen Varia wanted. And so to support a GIS system you need points, lines, polygons, line groups, that sort of stuff. And it was clear that Ingress couldn’t do it because the data types we put into Ingress were the standard ones, integers, floats, text, strings, and you couldn’t support, you couldn’t efficiently support GIS types on top of that.

[10:20] So as a gis, the academic version of Ingress was a complete failure. And that was in the back of, of our mind. The other thing that happened, this is a little out of chronic chronological sequence, but it helps make the point is that the commercial version of Ingress, I think around 1985, you know, there was an, just proposed a date and time standard for relational databases. And so commercial Ingress implemented date and time, you know, using the standard Gregorian calendar.

[11:05] And so I was associated with the commercial version of Ingress as well, as I was still at the University of California as a professor. So I got a call from, from an Ingres customer who said, you know, you implemented date and time wrong. And I said, huh? We implemented the Gregorian calendar and you can subtract and, you know, if it has, you know, days have 30 or 31 months, except for February, except for leap years.

[11:40] So subtraction on dates works exactly the way you would expect it to. But he said, that’s not what I want in his particular world. He said he was dealing with bond financial instruments and for some reason, I mean, you got the same amount of interest on his financial bonds during each month, no matter how long the month was. So he had the date you bought the bond, the date you sold the bond. He wanted to do a subtraction, multiply it by the coupon rate and say, that’s what, that’s the interest we paid you.

[12:26] But of course, his version of subtraction was March 15th minus February 15th is 30 days, because that’s the definition of his calendar. And so he had to retrieve two dates out to user code, do the subtraction and user code, put the answer back, and it cost him a factor of two or three in efficiency. And he said, why can’t I just overload your definition of subtraction with what I want? And of course, with Ingress, it was hard coded.

[13:02] And the problem was this is a case where you wanted bond time just like you wanted points, lines and polygons. And so Postgres was engineered to have an extendable type system so you could have whatever data types you wanted. And they were very efficient. And that was the main gist of Postgres was that it had that flexibility. And as you know, in business data processing a lot most people were happy with the standard data types.

[13:41] But relational databases started to spread to all kinds of other places. What are called abstract data types or stored procedures, bunch of names they’re called, you know, had, had great applicability. And so Postgres, that was, that was the big thing in Postgres. We also, Postgres also supported what the AI guys at the time wanted in the way of inheritance. We also supported time travel, but the implementation absolutely sucked and it got taken out after a while.

[14:28] So there were a huge number of really nifty things in Postgres.

Ryan:

[14:32] You mentioned you want to hire extraordinary software engineers, and I think you’ve said before that, you have no trouble finding those people. How do you identify those people in your hiring? That they’re the extraordinary ones?

Mike:

[14:49] It’s usually pretty obvious. I mean, I have a good feel for how difficult stuff is. If they get 3x the, the amount done, you know, in school that I think is reasonable, then, then they’re incredible.

Ryan:

[15:05] On the flip side, you had this interesting quote I wrote, I wrote it down. He said, I can’t stand people who are, who aren’t really smart. It’s challenging to talk to them. How do you identify the people who aren’t smart?

Mike:

[15:20] Well, I mean it’s, it’s very easy. You talk to them and, and it, it rapidly, you can rapidly surface whether they’re smart or not. You know, what was your master’s thesis? What did you do? Well, how did it, how did it exactly work? Well, how did you deal with error conditions? How many processes did you have? Why didn’t you use threads? You asked them deep technical questions, you gave a talk.

15:55 — One size fits none

Ryan:

[15:55] And I think there’s also a paper behind it of this idea that one size fits all database systems not optimal, one size actually fits none. And that what you really want is database solutions that target specific needs. What database offerings you see today that are one size fits all.

Mike:

[16:16] In 2004 when I wrote the paper, we had an academic project which was building what became streambase. And so a stream processing engine looks nothing like a relational database. And we had the gist of an idea for column stores for data warehouses, which was popularized by Vertica, looks nothing like a roast or. So here were three wildly different implementations that had no resemblance to each other.

[16:49] And in each case they were an order of magnitude faster than the other guys. So it’s pretty clear that one side, you know that with those three instances you give up an order of magnitude when you’re running a database system that isn’t, that isn’t architected for your kind of stuff. I think that’s still true. I mean, I think Clickhouse is a column store. Pinecone is faster than user defined types on text based vector processing.

[17:31] And so I think it’s still very much the case. And I think there’s no difficulty putting a common parser on top of multiple implementations. Postgres has so far chosen not to do that. They don’t implement a column store. And so I think they are not, they are not competitive, you know, on sizable data warehouses. They also don’t have multi node support. Again, for people with big data warehouses. That’s table stakes.

[18:11] So I Think it’s just as true today as it ever was. I think that what is true is that if you want to get going, you have a database problem. You know, the answer is choose postgres. And there’s a huge programming community, all kinds of, all kinds of, you know, data type implementations. It’s free and you can find postgres talent easily and get going. And so I think it’s, it’s, it’s a great choice for lowest common denominator.

[18:51] And until you’re trying to do a million transactions a second, it works just fine. Until you’re trying to support a petabyte data warehouse, I say at the low end it’s absolutely the right one, size fits all. At the low end, it’s postgres. At the high end, that’s just not true.

Ryan:

[19:12] GPUs, do they make available some new opportunities to optimize databases?

Mike:

[19:19] Probably, but I think the big challenge is that GPUs are SIMD, single instruction multi data. And that’s the anathema of indexing. And so whenever indexing is the right answer, they’re probably not a good idea. I think also you’ve got to architect them so that the, so that the bandwidth, so that the bandwidth from storage is, is not, not the bottleneck. And so if they’re an add on to the CPU as often as not, the bus connecting it to the GPU to the CPU is a bottleneck.

Ryan:

[20:10] Can you explain why indexing would be not as effective when there’s simd?

Mike:

[20:17] So let’s, let’s say I’m, I’m looking for Ryan’s, I’m looking for Ryan’s salary and I have a B tree. So you go to the root of the B tree, you find, you find the divider that has both sides of Ryan. You follow the pointer, that’s a memory access for sure. Then you do it all again and you do this like three or four times. So that doesn’t parallelize. Well, so the answer is indexing doesn’t parallelize.

Ryan:

[20:54] Well, you mentioned B trees when you first implemented that first version of Ingress. Did you write all of that by hand? Because I imagine there’s probably not some existing B tree library or something.

Mike:

[21:07] Yeah, we wrote the original version of Ingress was all written from scratch.

Ryan:

[21:12] What was the hardest part of that implementation?

Mike:

[21:16] Query optimizer.

Ryan:

[21:18] And why is that hard?

Mike:

[21:20] It’s tough. It’s just algorithmically difficult. It’s still, if you ask most any senior database programmer, what’s the hardest, hardest part, they’ll still say the optimizer MapReduce

21:37 — Why he disagreed with Google

Ryan:

[21:37] came out at some point in the early 2000s and it kind of took the data world by storm. People were really impressed by it. They thought Google really knows what they’re doing. This is the best thing since sliced bread. But it seems like when I look at the literature and what you thought at the time, you kind of disagreed heavily. Why did you disagree so much with MapReduce?

Mike:

[22:02] Well, I think there were a lot of not very enlightened people who said, Google, Google is really smart, they must know what they’re doing. And so we’ll do whatever they say. And so they would, they would, they would engage in Hadoop or engage with Hadoop, but Hadoop is ridiculously inefficient. And so at the time, you know, others, you know, Dave DeWitt and others who were involved in our 2011 paper, we understood distributed databases and understood that you could beat the heck out of Hadoop with a distributed database system, which is basically what that 2011 paper says.

[22:54] And of course it was, it’s true. And, but that wasn’t the only, that wasn’t the only thing Google was stupid about. So Google also had the opinion that eventual consistency was the right way to do concurrency control. And so that was postulated from on high by Google all during that same period of time. And it wasn’t. And all the database people said, you know, you’re out of your friggin mind. Because it doesn’t, it solves one particular kind of problem, but only.

[23:39] And that very rarely occurs in practice.

Ryan:

[23:42] Why did they pursue eventual consistency?

Mike:

[23:45] Okay, well the idea is that have an east coast database and a West coast database and they’re replicas, so you want them to be the same. If you say I’m going to do a transaction, I’m going to decrement by one the number of widgets in the west coast warehouse, then I’m going to, before I commit that transaction, I’m going to update the east coast warehouse, pay a message over and back to update it.

[24:14] And then to make sure everything goes well, it takes, it takes another round trip of message to make sure that both of them actually do the commit correctly. So it’s expensive to do a distributed commit. It still is. And so the idea was, well, you do the, do the west coast update, you decrease the widgets by one, you just send a message asynchronously and not in a transaction so that eventually the east coast warehouse gets decremented by one.

[24:51] So meanwhile, if you’re on the East coast you, you decrement, you know, foodstuffs by one, you send an asynchronous message, eventually the west coast gets it and eventually everything settles out. So if you’re allowed to, to go below zero, then what will happen is if the east coast guy and the west coast guy simultaneously sell the last widget, then, then eventually the state of the warehouse will be minus one and somebody won’t get their widget, their widget.

[25:36] And so if you’re allowed, like Amazon to say, usually ships in 24 hours, then maybe you’re allowed to oversell. But most enterprises can’t do that. And so eventual consistency just doesn’t work. So we talked a million hours ago about referential integrity. So referential integrity in a sales system is integrity constraint is stock is greater than minus one. And that fails with eventual consistency.

[26:16] And so Jeff Dean final of Google finally figured that out and when they did Spanner, Spanner had a conventional transactional system. And so Google completely abandoned eventual consistency and completely abandoned MapReduce.

Ryan:

[26:37] So the trade offs basically correctness for performance.

Mike:

[26:42] So it’s performance versus data integrity. And if you don’t care about your data then you’re willing to deal with, with bad things happening.

Ryan:

[26:54] So did you ever talk to the Google team while they were doing those things that you thought was so wrong?

Mike:

[27:00] We talked to them before the 2011 paper and said why don’t we, why don’t we partner up and do some stuff? And they weren’t, they weren’t interested. So they declined.

Ryan:

[27:19] Have you seen other examples in other big tech companies where their databases are database solutions where you actively disagree with them, like maybe Amazon or Facebook?

Mike:

[27:31] Well, I gave a talk at Amazon maybe three years ago and I told them all the things I thought they were doing wrong. And I think Amazon’s problem is that they are supporting, you know, 15 different database systems and that’s about 12 too many. So, so I think they have their own culture. And I told, I, I said you’re supporting too many database systems. And at this point they haven’t chosen to retire any of them.

Ryan:

[28:09] Why do you say that the 15 should be 3?

Mike:

[28:12] Well, they’re supporting a graph based database system and it’s well understood that a graph based database system is almost never the performant option. And so if you want a graph, if you want, if you like the idea of having a user interface that deals with nodes and edges, that’s fine. Put, put a layer on top of a relational database system that gives you that user model. And so most of their database Systems, there’s some other of their database systems that better at what it does than, than it is.

[28:58] And so, so the answer is you should retire. You should retire any database system that isn’t performant in, in a big enough market to justify the maintenance.

29:14 — Why he chose academia over big tech

Ryan:

[29:14] You’ve influenced industry significantly from academia and my one thought that I had is what, why not work directly in industry? Or why, why do you prefer the position of being in academia and having influence in the way that you have versus just taking a job at AWS or something like that, being a very distinguished engineer there

Mike:

[29:43] because that gives you a boss and that gives you company rules. Limits your ability to publish, limits your ability to go talk at conferences, limits your, your ability to go, go poke at what, what various competitors are doing that they won’t tell their competitors. But mostly I really like being in startups and I, and I after the commercial version of Postgres got acquired by Informix, you know, I was working part time for Informix which was a 2000 person company and I didn’t feel like I could make a difference because it was bureaucratic and, and whatever the president wanted he got.

[30:38] So I think I’m just not cut out for, I’m not cut out for politicking. I don’t do that very well and I have a hard time interacting with people I think are dumb and that again. So I guess I have some problems with big companies.

30:58 — Replacing state in an OS with a DB

Ryan:

[30:58] I wanted to talk a little bit about dboss. I just thought it was a really interesting technical model. Can you explain what DBOSS is?

Mike:

[31:08] We started the academic project in 2019, 2020, something like that. And the gist of it was at that point Mate Zaharia, who is on the faculty at Stanford, was also one of the founders of Databricks, was the original creator of Spark. And so he said at the time Databricks basically was running people’s Spark jobs on the cloud. He said at any given time we might be orchestrating a million Spark jobs.

[31:54] We have to write a scheduler that’s going to decide who to run next at scale a million. And he said there was no, we tried all the, all the schedulers written by the OS folks and they didn’t scale. So we put all the scheduling data in a postgres database and basically a postgres application was doing scheduling and then it sort of clicked that. By and large most everything you do in an operating system is managing data at scale and you should do that using database technology.

[32:38] So why don’t we just replace at least the upper half of Linux with a database system. So that was the gist of the academic project and we worked on it at Berkeley and Stanford in the early, early twenties and it was, it was very successful. It clearly, it clearly worked. And in the process the Stanford folks wrote an extension to JavaScript so that you could program, you need some programming world that can, can talk to your implementation.

[33:23] So if you’re doing what amounts to a programming language and you’re running on top of what amounts to an operating system, that is a database, then the obvious thing to do is put all your state in the database. And that’s exactly what they did. And so we had an innovative programming language model, an innovative operating system model. And of course then the idea was, well, can we start a company?

[33:56] And so we Talked to the VCs, who to a person said, you’re dreaming if you think you’re going to displace Linux. However, this programming language stuff is really nifty. We had what amounted to extensions to JavaScript that would allow any, any program to have all of the nice features of a database system. You know, stuff was durable, you could have transactions, if it failed, you’d fail over, you know, it’s all that nifty stuff.

[34:37] So we got funded to start a company in 2023 and that was DBOSS Incorporated. And we decided that that was the name of the project, since it always been the name of the project, but we were, we were basically in the programming language business. And so at the current time, DBOSS has a version of TypeScript, a version of Java, a version of Joe Go and a version of Python, which, which are basically seamless.

[35:16] It runs what looks like vanilla programs in the world of the cloud. There’s every incentive for you to structure your, your, your application as a workflow. And so we decided that we would support a workflow system, period. And so the workflow that DBOSS supports in those four languages is the steps in a workflow. The individual micro ops, whatever you want to call them, are transactional workflows, are durable so that once you do a step, it’s not forgotten.

[36:05] And it’s clear that we can make workflows atomic if there was a market for it, which means the whole workflow would either finish or look like it never happened. So it has very, very nice properties and is a great deal faster and a great deal easier to use than the competition. So the company is selling and innovating in this area. And so, so the idea is that you want to make state of your application persistent when you put it in the database and then it, and then you figure out how to do it fast.

[36:53] And I think their business model as we were talking earlier, is very much get leaf level programmers interested. So it’s been very much tell us leaf level program or tell us what you need that we don’t have. Get it quickly and convince people to try it. And we’ve been very, very successful with others, with other startups who want to choose the best thing and we’re starting to be, to be successful with the big boys.

[37:36] So it’s an interesting market and I think the key thing so far is probably 2/3 of the customers are doing agentic AI, which means that they have a large language model surrounded by a bunch of stuff that adds more signal. And so far most of agentic AI is read only, meaning you want to produce a prediction for whether Ryan is going to be a good customer or not. And so that just runs some stuff and then produces a new thing that’s given to somebody.

[38:27] So basically read only, which means that you’re not actually updating Ryan’s credit rating or. And so I think fairly quickly this, the whole world is going to move to using, you know, agents to do read write applications and that’s going to make, that’s going to make them very, very databasey. And dboss does that stuff really, really well. And so you know, for instance, if you want to write an agent or two agents that move a hundred dollars from my account to your account and so you debit my account, you increment your account and these two agents have to agree to commit or you have to back everything out.

[39:29] Which is to say the workflow needs to be what I called atomic, which is it all happens or it looks like it never happened. And so I think the, the demands on, in this market will escalate with, with things with people wanting stuff to be read. Right. And so I think that that will bode well for the market and bode well for DWAs.

Ryan:

[39:57] And this, this what’s being offered in the market today to application developers differs from the original research project where that was actually swapping out the guts of an operating system with the database. I see that. I mean that’s really cool. I never imagined replacing all the state of a, of an operating system with a database. What’s the there, there’s gotta be some trade off there.

Mike:

[40:22] Well, a file system written on top of a DBMS is faster than the Linux file system. The scheduling engine is competitive with other scheduling engines. You can make everything fail over so you get high availability without having to do anything else. The answer is there’s really no downside.

Ryan:

[40:49] Then why wouldn’t Linux incorporate that and upgrade itself with this?

Mike:

[40:55] You hope they would. In other words, you should keep all the device driver junk down at the bottom because there’s a lot of it and no one wants to do that and replace everything else with the database implementation.

Ryan:

[41:12] Is that something that you’ve mentioned to Linux people and what’s their typical reaction?

Mike:

[41:17] Back in the academic project when I’d mentioned that to Operating system folks, they would get very, very threatened, which is this is the database guys trying to take over their turf and I think the programming language guys ditto. Which is the way to implement the runtime for a programming environment is with the database.

Ryan:

[41:46] That’s, that’s interesting. I mean if it’s objectively true, then maybe it will take over.

Mike:

[41:52] Well, I mean it took Java 10 years to become widely accepted. I just think the time constant is substantial.

42:02 — Future problems in databases

Ryan:

[42:02] I think we talked a lot about the past of databases and I’m curious your thoughts on unsolved problems in databases and what you think the future might look like.

Mike:

[42:13] Okay, so I think two different things that I’d like to talk about. The first one is like everyone else, three years ago we started to look at what were large language models good for. So we’ve been trying to get what’s now called text to SQL to. We’ve been, we’ve been trying to make it work on real world databases, especially real world data warehouses. So we’ve been trying the technology on four different production databases, warehouses where we’ve gotten the workload, the actual workload that’s run and you know, from the actual users using the system.

[43:11] And we’ve gotten them to reverse engineer the text that corresponds to that SQL. So we have text and SQL for. We have four benchmarks.

Ryan:

[43:26] When you say text to SQL, you mean like a human prompting model or something? Like I would just in English that text would be, you know, everyone over

Mike:

[43:36] four years old tell me all the professors at MIT who won the Turing Award. And so an LLM is supposedly good at that. And so the text, the SQL benchmarks, there’s one called Spider, another one called Bird, and the, the best LLM systems are pretty good at those benchmarks, you know, like 80% accuracy or better.

Ryan:

[44:04] So not superhuman?

Mike:

[44:06] Not superhuman, but they’re pretty good. Like you would consider using them and you know like current, current leaderboard is something like 85% accuracy, which, I mean it’s getting there. You say maybe it’s not quite ready for prime time, but it’s simply, it’s certainly looks looks pretty good. Well, on our benchmarks, large language models get 0% and if you enhance them with rag and all the tricks goes to 10%.

[44:41] And if you give as a prompt the from clause, in other words, all the actual tables that need to be accessed and all the actual join clauses that need to be joined, then accuracy goes to about 35%. So the definition of this stuff is not ready for prime time and not going to be for a while, if ever. So what’s the difference? Number one, data warehouse LLMs are trained on the pile. Data warehouse data is not in the pile.

[45:23] And there’s an adage that if you haven’t seen the data a couple times before, you have no chance of regurgitating it. That’s number one. Number two, query complexity on Spider and Bird is maybe 10 to 20 lines of SQL real world data warehouses, it’s 100 lines of SQL complexity is bigger. Number three, the schema in Spider and Bird is clean. The table names are mnemonic, the column names are mnemonic and there’s no duplication in data warehouses.

[46:08] People have materialized views all the time. It means there’s redundancy and column names are often underscore Z, upper score blah. And so they’re not mnemonic. That makes it a lot harder. And then they also have idiosyncratic data. So J term is popular thing at mit. It’s a one month term in January. Not unique to mit, but not very popular. So not in the pile. Idiosyncratic data, simple queries, schema.

[46:53] Schema is a mess, make it not work. And those are true of every data warehouse I know of. And so I think the technology simply doesn’t work and isn’t going to work anytime soon. So we’ve been. So what do you do? Well, first of all, we published our benchmark, it’s a thing called Beaver, which is an anonymized and abstracted version of these four data warehouses. And so if you think you’re really good at doing text to SQL, try a real benchmark, not a fake one.

[47:36] So number two, borrowing from what I just said, if you don’t have all the join terms and you don’t have the from clause, you’re toast. What’s more, if you don’t break down the query into simpler pieces, you’re toast. So that says to me that you want to give your retrieval system simpler pieces which include the from clause include join terms. That’s number one. Number two, the minute you want to talk to two different structured databases like your data warehouse and your CRM system, then it’s pretty clear to me that doing a structured data join using an LLM is a bad idea.

[48:37] It’s just you’re much better off leaving them as tables and doing a join in SQL. So our point of view is we are trying out turning everything into tables. You know, we’re, we’re working with the Department of Transportation in the city of Munich, Germany and they have six people full time who are answering citizens complaints queries which are of the form how come I, I don’t have enough time to cross this intersection next to my house before the light turns?

[49:18] All kinds of stuff. How come the trolley doesn’t stop for enough time for me to get on the trolley? You know, it’s how come the trolley doesn’t come more than once an hour? I mean it’s all this stuff. Their database is the trolley schedule, that’s SQL. The light sequencing, that’s SQL. The intersections, that’s cad. The federal, you know, country of Germany regulations of this stuff, that’s text.

[49:56] City of Munich regulations for this stuff, which is text. So you got to join SQL, SQL, cad, text and text. So our point of view is turn it all into SQL, all into tables and do a join with what amounts to a query optimizer. So that’s what we’re working on. I think other people will have other ideas, but I think it’s extremely fertile area because people really want to do it. So that’s number one. Number two, we talked earlier about agentic AI.

[50:33] The minute this becomes read write, it’s a distributed database problem and you want atomicity, consistency, all that stuff. I think a very interesting area.

Ryan:

[50:53] zero percent right now. What percent is human? If you took someone who really knows SQL and what would they score like the average human?

Mike:

[51:03] So once you disambiguate the text, a knowledgeable SQL user programmer with the schema will do, will get very high accuracy.

Ryan:

[51:15] Okay, like 90% or something at least. Okay, okay. Wow. I’m surprised that the LLM scores so lowly on this kind of benchmark. Maybe when this goes out, someone who works at Anthropic will reach out to you or something and say let’s.

Mike:

[51:31] I’d love to, I’d love to find out because I mean it’s a terrific

51:36 — Technical book recommendations to learn databases

Ryan:

[51:36] success story for people who want to deeply understand databases. And they’re looking for some material to study. Is there a book that you recommend that’s a top technical book to learn?

Mike:

[51:49] Databases, papers in the literature. I think Joey Hellerstein and I published a red book, what’s called the red book, which is called Readings and Database Systems. It’s now eight years old. I mean, I think that that would be a great set of readings for eight years ago. And beyond that, papers, popular papers from the literature.

52:20 — Advice for younger self

Ryan:

[52:20] If you could go back to yourself when you just graduated, give yourself some advice, knowing what you know today, what would you say?

Mike:

[52:27] Back when I first took the job at Berkeley, without thinking about it much, we said, let’s write a database system. And we, we knew nothing about databases, nothing about implementations. We were not skilled programmers like Bill Joy. So starting off doing something that was that crazy was really pretty crazy. And you effort and you make stuff work and you learn along the way. And so I think the answer is think outside the box, think crazy thoughts and try and do them.

[53:11] And I think to me it’s not at all obvious. The better question is if you were starting out today, what would you major in? Because I think computer science may well not be a growth industry going forward. And I’m not sure I would recommend 18 year olds to major in computer science. I think healthcare and the building trades are, are safe bets and everything else looks much riskier. If, if you’re about to get your PhD and are trying to decide what to do, then I think life is pretty easy.

[53:55] You know, take, take the most prestigious job you can get and find a mentor who’s willing to help you and then pick some area that isn’t, you know, like our, our stuff, you know, which is called Rubicon, is definitely not going with the flow. So choose something that’s not, that isn’t going with the flow and try and make it work. Both my wife and I said, follow your passion, somehow the money will work out.

[54:31] And I don’t believe that for a minute. But, but I think that’s what you have to tell your kids and your grandkids.

Ryan:

[54:38] If you don’t believe that, then why do you have to tell them that?

Mike:

[54:42] My wife is a good example. So she has a master’s degree in computer science, undergraduate degree in computer science, and she wanted to be a teacher, K12 teacher. And her parents said, you can’t do that, it doesn’t pay enough money. And so I think, I don’t think she, ever since that time has regretted that decision. She wasn’t passionate about doing computer science. It was simply a trade. And so I think find something you’re passionate about and and you will.

[55:22] You know, either you won’t starve, you may not make a lot of money, but I think chances are you’ll be happier than if you do something you’re not passionate about. Because I think a lot of people I know view their job as simply a job and life is what happens between 5pm and 8am I don’t feel that way at all. I really like what I do. Wouldn’t matter whether I made a lot of money or didn’t.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

1:27 - Finding problems that matter

11:42 - Learnings from 3000 postmortems

23:58 - Why caches are bad

29:37 - How AI will change software engineering

36:49 - Advice for junior engineers given AI

44:02 - Thoughts for senior engineers

49:59 - Why engineers should write

57:51 - Visibility and apparent expertise

1:04:23 - AWS engineers he admires

1:06:53 - Technical book recommendations

1:09:06 - Advice for his younger self

Transcript

1:27 — Finding problems that matter

Marc:

[1:27] Yeah, I think you have to go super broad. So I think there’s a set of those things that come in from customers from the world. Right. Like here is an unsolved problem. You know, I spend a lot of time meeting with AWS customers and listening to them talk about, you know, what are the things they still find difficult in our space? What are they, you know, what are they investing in? Where are they spending their time?

[1:48] Where would they prefer to be not spending their time and focus on their core business instead? And so that’s one rich seam of ideas and focus on what’s, you know, what’s interesting, I think completely at the other level is sort of looking at the technical trends and you can look at just the kind of speeds and feeds like, wow, networks have gotten faster, storage has gotten faster. You know, we’ve seen this huge explosion in multicore and now in GPUs and you know, and so there’s a bottom up innovation trend there too, which you can also look at and say, well, this enables all of these new, new things.

[2:27] And, and then broadly kind of across the world, like what, what are the big trends that are going on? What are the things that are changing in our industry? What are the things that are changing in, in the world? And really it is those kind of moments of change that have the, you know, bring with them the opportunity to, to, to build things and, and, and to recognize problems. And so to pick one, you know, concretely.

[2:52] You know, when I was in, working in the lambda team in 2020 and I was talking to a lot of customers about, you know, they were super excited about building on serverless, they were super excited about building on containers. There had been this massive shift and what people were seeing then was, wow, I love these serverless products, I love building this way. But the world of data, and especially relational data doesn’t fit super well into this, this paradigm, right?

[3:19] These relational databases are still very serverful, you know, fantastically powerful products, but not kind of operationally the same. And, you know, that thinking was, you know, just felt super important to me of like, wow, these customers have brought to me a gift of understanding something that’s really important. And so I joined the Aurora team. We built Aurora Serverless and then we built the SQL.

[3:45] You know, we’ve been investing deeply across all of our database products to make them a better fit for these, you know, serverless and container workloads. And that is an example of a trend that was brought by, you know, brought by a customer. But then also these trends that have been driven by kind of architecture or by other things going on, right? Faster networks, faster compute, faster connectivity.

[4:15] And so one of the big technical trends in the database world right now is this sort of block storage becoming the default backend, the default durability layer for databases of all kinds, from analytics workloads to online workloads. And there’s been this incredible explosion around that. And so if you look at what we did with Aurora DSQL, for example, that was very much learning from that trend and taking a lead on that trend and saying, well, well, we’re going to make S3 this block store that we built 20 years ago, sorry, object store that we built 20 years ago,.

[4:55] Underlying durability layer of this new database, but obviously it doesn’t have the latency properties or the rich interface that an online database needs. And so we’re going to build an architecture on top of that that deals with all of these other things in a much better way, but doesn’t have to worry about durability. And, you know, so that was this perfect collision of a set of things I was hearing from customers and a set of things that were technical trends coming together and thinking, wow, we’ve got this opportunity to build something now that is going to be a market leading product that would be hard to imagine without either of those input signals.

Ryan:

[5:39] I saw something that you wrote. You mentioned that you were on call for 15 years somewhere in there. And I’ve heard many stories of more senior engineers negotiating out of on call because per unit time it could be perceived as not that impactful. And so why did you stay on call for so long?

Marc:

[5:59] I would say that the majority of my in practice knowledge about how to build distributed systems has come from being on call and analyzing and deeply understanding these post mortems and COEs. You know, one of the challenges of running a company like AWS and running large scale systems is that folks come out of college with great, often great knowledge of computer science fundamentals, great programming skills, you know, great mathematical skills.

[6:34] All of that stuff is fantastic. But without the grounded knowledge of what it actually means to run and understand, you know, understand systems and you know, on call is one of the best ways to learn those things. Best ways to see, you know, how do systems really run? How do they really behave, you know, how do customers really use them? What happens when customers use systems in unexpected ways?

[7:01] How can we make systems more resilient to customers using them in different ways? And I think that should be almost a goal of on call, right? If you have folks in your teams who are on call and they’re just closing the same ticket over and over and over, well, you know, that’s where you need to just build some automation. And again, building automation is easier than ever. It’s more powerful than ever.

[7:24] Fantastic. But where you really want to spend the time of the deep experts on your team is, you know, here’s something unexpected or unusual that’s happened in the system. Let’s deeply understand that and let’s bring that knowledge back to both improving that system and communicating broadly to the company and the outside community what we’ve learned from that. And so one of the most, you know, one of the most powerful things we do at AWS is we have this mechanism of a very broad weekly meeting where we all get together, you know, engineers from across AWS leaders, senior leaders from across AWS and talk about COEs, these postmortems that we write and what we can learn from them and how we can apply those lessons across the whole company.

[8:22] And I think that particular mechanism, that particular kind of Wednesday morning meeting that we have, is one of the things that has been a core, almost causal factor behind AWS’s success because it has allowed us to and Forced us to spend leadership, bandwidth, to spend expertise, to spend the time of our best engineers deeply understanding how our systems operate and why they operate the way they do.

[8:58] And you know, that level of being just extremely grounded in reality helps you design better products, help helps you architect better systems, and it helps you think more clearly about the next round of things. Helps you fix, you know, helps you fix issues. And so it’s this fundamental kind of learning exercise. It’s a real blessing. So I would, you know, I would recommend oncall to anybody who wants to learn about the practice of distributed systems.

[9:29] And I would certainly recommend spending time reading coes, reading post mortems and, and deeply reflecting on not only what can we fix tactically, but what can we fix organizationally and strategically and what kind of tools might, might need to exist to, to prevent this kind of thing happening again. And you know, you asked earlier about, you know, where do ideas come from? This is another, you know, fantastic kind of flow of ideas of saying, wow, you know, we seem to be solving the same problem over and over in different ways and getting it slightly wrong every time.

[10:06] You know, can we extract a tool to do that? Can we build a service around that? Can we, can we build a feature around that to make it easier for us to get right and easier for our customers to get right?

Ryan:

[10:20] Yeah. It’s interesting because I think if you ask most engineers, they really avoid on call, but it sounds like you, you kind of go towards it and you’ve learned a lot from it because it’s a major source of customer problems. Yeah.

Marc:

[10:35] And again, you know, I think for me it comes down to optimizing for finding the most important things to work on. And you know, if you aren’t close to operating your actual system and you don’t know how it’s actually working, how are you supposed to identify what to fix? Right. You can come up with some theories about those, but they’re probably not going to be right. And again, like, I don’t think there’s a huge amount of value in the rote ticket closing work of oncall.

[11:06] I think automation, you know, should be doing those kinds of work. But I think there’s fantastic value in, you know, deep understanding, deep investigations and, and deep reflection on what you learn from postmortems and COEs. I tried to estimate a couple of months ago for a talk how many industry postmortems and Amazon coes I’d read over my career. The best estimate I could come up to, and this was about a year ago.

[11:33] Was between 3000 and 4000, and so know even a little bit of lesson from each one, and it tends to, you know, tends to stick.

11:42 — Learnings from 3000 postmortems

Ryan:

[11:42] Yeah, that was my next question, actually. I looked at the slides from that internal presentation and it said, I’ve read approximately 3,000 cloud system postmortems from across the industry. And my immediate thought was, I wanted to ask you what makes a good postmortem?

Marc:

[12:00] So I think, you know, what makes a really great postmortem is first really getting into the details and making sure that you deeply understand what happened rather than just assuming what happened based on the biases you bring in. And so there’s a kind of lesson one, there is, if you can’t understand what happened, well, that teaches you something about your logging and metrics and observability and simulations and all of these other things.

[12:30] And then once you deeply understand what happened, then the ability, then a great postmortem steps through the whys behind that at multiple levels. Right? Like, why? Well, yeah, there was a code bug. Okay, sure. Code bugs, yes, we can fix that, but we can’t stop there. Right? Like, why was that missed in testing and validation for these reasons? What can we improve? What can we build around those?

[12:58] Okay, next step. You know, why, you know, why was our testing and validation where it was? Or, you know, why did we assume a certain thing about the behavior of the system that we wouldn’t have assumed before? And so as you sort of get through these deeper and deeper layers, a great postmortem not only identifies kind of fixes to the proximal cause, but also identifies broader fixes to technology, to organizations, to products, and so on.

[13:31] And so that’s a kind of multiple levels thing, right? You can’t get stuck on what is the most proximal cause of an incident, but you also can’t get stuck on this. Well, things fail sometimes, and what are we going to do about it? And you have to come up with a set of, you know, really concrete action items to fix things at different levels. Fix this particular line in the software that caused something, you know, fix the testing processes that didn’t catch that, you know, fix the, you know, maybe social or team processes that led to those technical processes.

[14:14] And, you know, and then if you’re seeing patterns across multiple postmortems, sort of level those up and say, well, clearly there’s a hard underlying problem here. You know, can we build a service around that? Can we build a library around that? Can we build a, you know, community of practice around that, you know, are there technical changes we can make to avoid whole classes of things? So that’s quite a long winded answer.

[14:43] But I do think it all flows from understanding and understanding at multiple levels, like understanding immediately like what happened, but also understanding broadly what happened technologically and organizationally and in context, and then the ability to connect that particular event or postmortem with other ones and extract those patterns. One of the things that we did in DSQL was we spent a lot of time as we were designing that, looking around relational database related postmortems and thinking about both our own and our customers and thinking about how can we design a database that helps people avoid falling into these traps.

[15:31] A really common outage pattern, folks with relational databases is you have a client on a distributed system, starts a transaction and then goes out to lunch for whatever reason. And that could be a GC pause, or it could be a lossy network, or it could be a loss of connectivity and now it’s holding locks. And so if you look at relational databases, they don’t tend to be resilient to clients misbehaving in that way.

[15:59] And that’s a really common cause of operational issues for systems built on relational databases. And so as we were designing DSQL we were thinking how do we avoid broadly that class of problems? So folks can say, hey, I’m going to build on D SQL and just not have this whole class of problems. And I think that’s a really kind of powerful outer loop over the post mortem process is to say, how do we turn all of these lessons into new services and into service improvements?

Ryan:

[16:35] How do you prevent misbehaving clients from being a problem for the database?

Marc:

[16:40] Yeah, so in DSQL’s case, we have no pessimistic locking. And so within the scope of a transaction, everything that happens in that transaction, all of the reads happen using this mechanism called multi version concurrency control, where every row in the database we sort of store a history of versions. And so you can read an old version of a row without blocking writers and saying, hey, you can’t update this because I just read it.

[17:11] And then locally within the query processor that’s handling a connection, we spool the writes locally and then you get to commit time and we do this optimistic check of can I commit this transaction at, at the transaction commit time. And so combining those two mechanisms of having multi version concurrency control and, and the scale out storage that comes with it and the commit time optimistic checks, we can strongly say that, you know, there is no way that a Reader of a piece of data can block other writers.

[17:46] And there’s a no way that that a writer of data can block readers. Writers can block writers, but only by changing data, not just by looking at it. And so you can say, well, I can cause, sorry, writers can’t block writers, but they can prevent other writers transactions from eventually committing by making a bunch of changes. And that is inherent to the definition of the particular database isolation level.

Ryan:

[18:18] Out of curiosity, in practice, what percent overhead would you expect for keeping copies of old rows for the sake of those stale reads?

Marc:

[18:28] Yeah, it’s actually surprisingly small. And it’s surprisingly small because if you look at the access patterns for most online databases, even ones that do a lot of write traffic, that write traffic tends to be quite concentrated. And it’s quite unusual for an online database workload, or even an analytics workload to make a second version of every row in the database. Typically what it’s doing is making a, you know, first, second, third, hundredth version of this row and a 50th version of that row.

[18:59] But the vast majority of data isn’t changing. And so it’s super workload dependent, as is everything in the database world. But the overhead tends to be relatively small. I would say it’s unusual for a online database workload for that overhead on storage to be more than about 10%.

Ryan:

[19:23] From my experience, I’ve seen an interesting dichotomy between teams where some teams, they really understand postmortem culture. They tend to be infrastructure teams, they tend to take it really seriously. And everyone on those teams, the tech leads are asking you, hey, why? Why did that happen? And you know, really follow up and make sure it’s, it’s not a problem. Then I’ve also noticed on other teams that is less of a strong muscle for those teams that don’t take it too seriously.

[19:51] What would be your, your pitch for why they should take it seriously?

Marc:

[19:55] Yeah, it all comes down to where you want to spend your time, right? Do you want to spend your time improving your product and making it better, or do you want to spend your time fighting the same fire over and over? And you know, really the culture of building, you know, building great postmortem culture is to make sure that at the pros, at the product level and at the organizational level, you are fixing known issues and you are avoiding having the same problems multiple times.

[20:35] And typically when I see teams that have, you know, poor post mortem culture, I think they’re probably one of two failure modes there. You know, one of them is A lack of focus on just the outcomes, right? Like, you know, a lack of really, I wouldn’t say caring enough. I think that’s a little bit too personal. But being really focused on, on, you know, is this, is this product performing super well?

[21:06] Are we, you know, are we really making our customers happy? And that is fundamentally a cultural and leadership, cultural problem of, of setting the right standards. Oh, and by the way, like, I don’t think, you know, standards should be, you know, should be uniform, right? Like there are places where, you know, the details really, really matter, where things like durability are just critical and, and, and you do need to have super high standards in those places and you know, places where you want to optimize for other things and, and, and maybe have, you know, have, have a, a higher production defect rate.

[21:44] And I think that’s, that’s okay as long as that’s an intentional decision that’s being made. So that’s kind of case one, right? Like insufficient focus on the outcome. I think case two, and this is a harder one to change is normalization of kind of operational heroics. Like, we don’t need to fix these root causes because our on calls are superheroic and they’re going to stay up all night and they’re going to, you know, they’re going to hack around things and they don’t mind being paged 100 times a week.

[22:16] And they can feel from the inside like it’s a good culture, right? Like, oh, wow, these people are super strong owners. They’re super engaged, they really care, they’re really working hard on call call. And those are all good signals. But then when you look at it from the outside, it’s like, wow, we’re not actually fixing the causes of things. We’re just doing this fantastically expensive investment of taking all of these people and their strong ownership and their expertise and spending them just on, on, on this break fix cycle.

[22:46] And that’s where you need to kind of look at it from the outside and say, well, let’s take this energy of this team, fantastic energy, and focus it on improving the service, getting out of the cycle, finding new things to fix, finding new things to build. And that can be hard because it can be hard for those folks who’ve been in that mode to look at it and say, this feels so good. It feels really like we’re caring about our customers and caring about our product and caring about our business to realize that, oh no, we’re actually caring about it at the wrong level and we’re not Serving our business in the best possible way by being so narrowly and tactically focused on this break fix cycle.

[23:34] And that’s where you sort of need to pop them out and say, well, let’s spend more time thinking about the postmortem. Let’s spend more time thinking about the causes of things. Let’s spend more time addressing these things in a more strategic way. And wow, okay, now you’ve got so much more time to do that because you’ve broken the cycle and you can improve your product in different ways.

23:58 — Why caches are bad

Ryan:

[23:58] I mean, since you have worked on AWS for almost two decades, I’m sure you have a lot of experience building distributed systems. And I think one of the most common advice that you hear, I guess this is maybe in the context of system design, is I almost hear almost 100% of the time people will say, just throw a cache on it or you’ll have a system design. You say, how do you make it better? Let’s put a cache here, let’s put a cache there.

[24:26] And I saw you had a tweet that said that there are cases where caches are bad, despite people saying it’s best practice. I was curious if you could explain that.

Marc:

[24:35] Yeah, so caching is good, right? Like it’s, hey, I’m going to take these core ideas from computer science of temporal and spatial locality and I am going to exploit those to make my system faster, scale better, etc. And so, you know, obviously very attractive. But the downside of caches, especially in distributed systems, is they have this mode, right? Like they have this, you know, there’s a mode where the cache is full and the cache is full of the right data in time and space to perform very well.

[25:08] And there’s a mode where the cache is empty or contains the wrong data. And in the first mode, the system is fast and happy and healthy. In the second mode, the system is slow, often down, because now the back end isn’t scaled to deal with all of this uncached traffic. Customers are very disappointed. And often it is down in a stable way. And this is this kind of idea of metastable failures where the system has switched from state one to state two.

[25:43] And in state two, it’s still stable, right? Like it’s still, it’s down, but it’s not going to come back up under its own energy because, for example, all of this traffic is causing a huge amount of contention in my database or is saturating the network and so I can’t even refill the cache. It’s not even getting the right kind of data in. And so, you know, when I talk about the downsides of caches, it’s really about, you know, how do we avoid that modality between, you know, fast and, you know, that, that value of caches and the, you know, how do we avoid the state where we’re down?

[26:23] And so if I go back to D SQL, like our answer there is D SQL. What we call the storage tier is essentially a cache, but it is a complete cache. It contains every row in the database. And so it doesn’t have this mode where how do I recover from it being empty or containing the wrong data. It contains all of the data. Similarly, if you look at a more, let’s say classical relational database design like Aurora, the Aurora leader is constantly telling the potential failover targets, here’s something you should cache, here’s something you should cache, here’s something you should cache.

[27:01] So when a failover happens, the cache is warm on, you know, on the failover target. And so those are the kinds of things that you can do to avoid those modalities. But in general, you know, and I wouldn’t extract this as a rule or say that, you know, this applies 100% of the time, but in general I prefer to see the teams around me avoiding caching where possible. I prefer patterns where you have a, let’s say, complete materialized view of the data.

[27:35] If you need very fast access to it, especially if it’s slow moving, just pull it down onto your local machine and work with it in memory. If it’s only being updated once a week, who cares? Just make lots of copies of it. So that’s one pattern. Or use a scalable backend, DSQL or DynamoDB or whatever your favorite scalable database is, and keep your database vendor honest about getting to the scale and performance you need rather than putting a cache in front of things.

[28:06] So caching isn’t a bad pattern, but it is a pattern with some significant downsides that are really best avoided in practice.

Ryan:

[28:18] How often do you see that metastable failure though?

Marc:

[28:22] Yeah, it’s not super common. You might go years without seeing something like that. But if you look across the biggest, most impactful, you know, system postmortems across the industry, I would say that these kinds of metastable failures have been an underlying cause in probably a majority of them. And it’s super important that, you know, as an industry and as a community of practice, we understand those things deeply.

[28:52] Because also those cases where these do happen, you know, tend to be larger scale issues, longer recovery time issues and more complex to fix issues. Right. Where you have to often turn it off and turn it back on again, which is this very, very painful thing for a team or an organization to do. And you know, and so again, like you might go years operating a system with seeing nothing like this. And but if you look at the most impactful issues, it’s actually fairly common as an underlying cause for those issues.

[29:31] And so it’s kind of both of these things of being quite uncommon and being rather common.

29:37 — How AI will change software engineering

Ryan:

[29:37] I was reading your blog and you have a series of posts on how AI may impact the future of software engineering. And I kind of want to pick your brain on that. So what’s your perspective on how you think AI will impact software engineering and how it’ll change things?

Marc:

[29:54] Yeah, I mean, it’s maybe harder than ever to tell the future. And so this is a set of maybe guesses and predictions about the future. So I’ll say the first thing I deeply believe about software is we have only just started to see the impact that software is going to have on the world. There is such an opportunity for more software to exist, bigger software, better software, more personal software, all of these things.

[30:29] And so Software has throughout its 60 ish year history been supply constrained. And I think that’s going to remain true. I think the opportunity for software in the world is just almost unbounded. And that’s really exciting, right? It’s really exciting to be at a moment when the economics of building software are changing and are changing rather quickly. And that gives us an opportunity to think about what could we do in the world with a lot more software, you know, a lot more software personalization, a lot more just the right software in the right place at the right time.

[31:15] And you know, that gives me a huge amount of excitement about the future of this industry because we have a massive opportunity ahead of us driven by these changing economics of software development. Now also with those changes, there are going to be needs for us as software practitioners, people who build software, people who love software, to, to adapt. And you know that that means that software careers are going to look different.

[31:53] They’re going to look different early on, they’re going to look different later on. I think the software business is going to look different. And the success of people in organizations over the next, you know, next, who knows, five years, decades is going to be largely predicated on their ability to adapt to that change and to lead that change.

Ryan:

[32:17] You told this story about this guy who bet on analog circuits when obviously we know Digital became kind of the more dominant way, yet he made good money. For the people who maybe don’t want to adapt, you could still get by and succeed. It’s not going to be like a crazy thing. Is that kind of the takeaway and why you brought up that story?

Marc:

[32:41] Yeah, I think that’s, that’s the right takeaway. And so if I sort of break down, you know, the world into two, three tiers, you know, I think there’s going to remain a huge amount of joy in the craft of software. You know, like the craft of joinery with, you know, with hand saws. Right. Like it’s, it’s a nice way to spend time. It’s not a particularly economically interesting activity anymore. But not everything we do has to be an economically interesting opportunity.

[33:12] It can just be something I do because I enjoy it, because I enjoy the product of it, because I enjoy talking to people about it. Right. And so there’s, you know, I don’t think that is going to go away. I think we’re going to see, you know, a lot of interest in that. Like, there’s been interest in retro computing and, you know, people who run an Apple II as their desktop and like. Well, again, it’s wildly impractical.

[33:34] It’s not economically interesting, but it’s fun and something I could do as a hobby. And so, you know, that’s going to be a remaining part of the world of software for probably forever. And then there’s this kind of story that I told in the blog post and I think this relates to driving change in the real world is always harder than it looks from the outside. As you get into the details, things become more difficult.

[34:04] They become more dependent on people, they become more dependent on politics and policy and, you know, our various irrationalities as humans. And so driven by that, you know, there is going to be a huge amount of, and a shrinking over time, amount, but a huge amount of the software industry that is run in what I might call the old way, right? Past techniques, past languages, past technologies. And there’s real economic opportunity in engaging with that part of the, you know, part of the world.

[34:43] You know, as we saw with, with analog electronics. Analog electronics still very much exist. In fact, there are parts of the world like, you know, like radio and power systems, where there’s been incredible technological advancement in those fields, but they have become more niche. And so, you know, digital became the mainstream. We wouldn’t be talking like we are today if it wasn’t for this, you know, 12 orders of magnitude or whatever explosion in digital transistor counts.

[35:17] But there’s interesting opportunity there. And I think that interesting opportunity is going to change shape and become more and more specialized and more and more niche and great careers to be built there. And then there is the mainstream, which I think is going to adopt these new technologies from agentic development to AI powered development to specification driven development and a whole lot of other new things whose names we don’t even know yet, to build software at a speed and a cost that is unimaginable to do with the old techniques.

[35:57] And I think that is where, correctly, the majority of the industry is going to be going. I think that’s where the majority of careers are going to be built. I think that’s where the majority of economic opportunity is. It’s the space I’d be in if I was building a company today. It’s the space I’m in in my role and you know, the one I would sort of personally be most excited about. But yeah, it isn’t the only one.

[36:22] I think there’s going to be the spectrum of software practice and especially where software engages with the physical world. There are going to be some really interesting questions about how do we bring these new technologies, how do we bring these new practices into the various, many niches that software is going to and has over six decades kind of wormed its way into.

36:49 — Advice for junior engineers given AI

Ryan:

[36:49] It’s interesting you mentioned joinery. I wonder if down the road we will see apps on the App Store that people pay extra for because it’s marketed as this was written by a human or it was written by hand. So bespoke custom app. Crazy how the world is going to change. But so it sounds like, you know, change is obviously the common case. It’s the one that we should be thinking about. Maybe we can break up the conversation into two parts.

[37:18] One is for junior engineers, what is important given that code is kind of flowing like water.

Marc:

[37:26] Now, at risk of being a bit meta about our past conversation, it really is about finding those problems that matter and doing that early in a career. And that requires an understanding of customers, it requires an understanding of the business, it requires an understanding of economics and of systems. And that can, I think that’s going to move from being, you know, almost kind of senior engineer work of like, oh, well, you know, now you’re going to go and talk to customers and actually understand the context of the stuff you’re building to being more and more part of even the earliest steps of an engineering career.

[38:09] Right. Like, here’s the context, here’s the Problem, here’s the customer. Let’s go off and work together and solve problems, you know, and solve this problem with all of this context. And I think that’s going to be super exciting for one set of folks and a little bit frustrating for people who have come into, you know, looking for a pure software development career. Right? Looking for a career where they sit down, open their idea, start typing, and don’t stop for eight hours.

[38:21] I think that’s going to be a mode that we’re going to see fewer people in and a mode that’s going to be harder and harder to build a career around. Now, the other mode of, oh, I’m excited to go off and learn from my customers about what they’re building and what they need. I think that’s going to be ever more highly valuable and so super exciting opportunity to build careers there. And then maybe.

[39:09] And this might come across as being a little bit paradoxical. I think there’s also a ton of opportunity for folks who are extremely technically deep, who are deep on optimization problems or deep on infrastructure problems, or deep on various scientific things, or deep on databases, or deep on, you know, one of the many, many topics that are behind our industry, because I think the ability to ask the right questions is also much more valuable than it was has ever been.

[39:49] And so I think there is a ton of opportunity for people coming into the industry with deep technical or scientific knowledge to now leverage that in ways that, you know, maybe were, were hard before. Right. There was too much sort of boilerplate to really, you know, to really use that leverage that you have. And so I think we’re going to see a lot more of, of, of those kinds of careers of really kind of building expertise in a technical topic, in a scientific topic, and then be able to turn that into software and software products in a way that was really difficult before and in some cases wasn’t possible before and is now, you know, vastly easier.

Ryan:

[40:31] If I was to look at a career ladder’s expectations, some of what you described, of maybe engaging with the customers and understanding the business context, uniquely in software engineering, it feels like the earliest levels are insulated from all of that. You have your tech lead, tech leads handing out tasks, and then the early level engineers just given tasks just converted into code. And it sounds like that part’s relatively solved.

[41:00] If not now, maybe I’d be surprised if a year or two from now wasn’t completely solved. And I think that could scare a lot of junior engineers because they would think you’re going to expect me to graduate from college or start working as a software engineer, and then I would have the senior engineer expectations. What would you say to the scared software engineer that’s just entering the industry with all this change?

Marc:

[41:30] Yeah, I think. Well, I would remind them that we as people who hire and build organizations of software engineers and they, as people who are building software engineering careers, have really aligned incentives. It’s not valuable to hire a bunch of people and set them up to fail like that’s, nobody wants that. It’s not an outcome that is good for anybody. And so, yeah, we’re going to need to figure out how do you support people on that path?

[42:06] How do you help people learn those things? How do you give them the right guardrails? Hey, that first time that you go out and talk to a customer, yeah, it’s going to be scary. My first time talking to an AWS customer was, you know, was, it was, was super scary. But you know, I, I got a bunch of help with that and I got a bunch of advice and I got a bunch of mentorship and I got a bunch of feedback and I got better and better at that over time.

[42:30] And I think that’s exactly what these things look like is, you know, you start off and you, you start small and, and, and, and you learn, you know, as you go and, and, and so that feedback loop goes faster. And so I don’t expect that people coming in from college or, you know, will, will. Will come in with all of this knowledge. I think, you know, it’s never been true that people coming into technical or engineering career straight out of college know everything or any career for that matter.

[42:59] Right. Like you talk to, to teachers about, you know, what they’ve learned on their job versus what they learned, you know, studying, you know, they learn a huge amount in, in, in things like internships and, and so on and over the course of a career or doctors or anybody in a field like that. And so, yeah, it is going to be about learning. And I think the emphasis on what people learn is going to be different.

[43:25] I think it is going to require leaders like me who care deeply about hiring and developing folks early in their career to be really thoughtful about what does that new letter look like. And, and you know, we’re doing a lot of that thinking. I think people are doing that kind of thinking across the industry. And yeah, it’s changing fast. It’s uncertain. It’s an interesting time to be graduating. But again, like, it’s a super exciting time.

[43:57] I think that just the scale of the opportunity is bigger than it’s ever been.

44:02 — Thoughts for senior engineers

Ryan:

[44:02] Sounds like your advice for senior engineers is different from that of junior engineers. What is your thinking there?

Marc:

[44:09] Yeah, I mean, I think for folks there, the challenge is how do you retain the value of this incredible experience and knowledge that you’ve gained over a career while not falling behind, while learning how to best use the tools? And when I look at senior folks, this is a challenge ahead of them. I think a lot of people have found themselves in influence and leadership type positions where they aren’t hands on building every day.

[44:44] And I think it’s going to be harder and harder to be in that kind of role and be able to influence and advise in a relevant way, in a positive way. And so really I think my advice for folks is you kind of got to get building, you got to get back in, get back into it. You need to deeply understand how the practice of building software and the practice of designing software has changed and is continuing to change.

[45:22] And so the challenge is, how do I really take advantage of all of this knowledge and expertise that I’ve built up in my career and be super curious and be super hands on and really be in the details? And the good news for that? Well, I think there’s two bits of good news. One of them is because of these new tools. You know, time spent as a practitioner is so much more leveraged than it is today. You can build such cool stuff, you know, during that period of time, the amount of kind of wasted time and boilerplate and so on is so much smaller.

[46:00] And so you really do have this opportunity. And the other one is again, like, why did you know, why did we get into the space? Well, I didn’t get into it so I could go to meetings and sound smart. I got into it because I love learning and because I love building technology and because I love solving my customers problems and because I love learning about new technologies and learning new things. And there’s more opportunity to do that than ever before again because of this new set of tools and the leverage that comes with them.

[46:31] And so really is getting back to, you know, why are you here? Why did you get into this career? And I think it really gets us as technology focused people closer to our original answer to that. It’s really obvious to me right now when I speak to, you know, practitioners, you know, who and who isn’t using a, you know, modern set of agentic powered developer practices, right? And the people who are have these really interesting things to say about the strengths and weaknesses of those approaches and the work that still needs to be done and the integrations that still need to be done and the things that are working and aren’t, and the people who are, you know, using them hands on have such a poor mental model of how they work, what they’re good at, what they’re not good at, that the things they say about them tend to be essentially fiction.

[47:31] And so I think we are in this minute that if you aren’t doing it hands on, your opinion about it is very likely to be completely wrong. And that takes a level of humility to admit. That is tough. It’s tough for folks with fancy titles and it’s tough for folks with, with distinguished careers, but I think it’s a must.

Ryan:

[47:55] I feel like there’s a common sentiment among software engineers when they work with someone who is a quote unquote, tech lead, but they’re not really hands on. So they’ve kind of been in the docs for the last five years or so and there’s these minor things they can tell that this person doesn’t actually understand the underlying thing. And sounds like that gap will widen with these new tools, which is if you’re, you’re looking at things from a thousand feet up and you’re not actually using the tools, that’s just another thing that separates you from the people who are actually building, where you’ll be very out of touch.

Marc:

[48:35] And I think, you know, when I look at, I think that’s always been true. I think it is wider than, you know, ever before. And. But when I look at the, you know, engineering leaders that I’ve really respected and learned a huge amount from over my career, you know, for example, some of the folks who built S3, you know, 20 years ago, that was such a successful product because those folks were so deep in the details and so grounded on the use cases and so deep in the economics and really just did, you know, really thought about both the kind of strategic world of like, how is this cloud thing going to change the way people want to interact with storage?

[49:21] But also the, you know, minute to minute details of what’s fast now, what’s slow, what’s good, what’s bad. And I think, you know, when you think about a extremely enduring product like S3 or EC2, I think it’s been that groundedness in the details from early on, from all levels of leadership that has made those things so successful, where other products seemingly with the same amount of early promise didn’t turn out to be as successful.

49:59 — Why engineers should write

Ryan:

[49:59] I think one of the last topics that I wanted to ask you about was writing. You have a ton of awesome posts on your blog. The style of writing is incredibly clear. And I was curious, why do you write so much as an engineer?

Marc:

[50:19] Writing and speaking, but especially writing, have this incredible power and for technical folks, it’s this incredible multiplier in being able to take these ideas that’s in your head and share them with the world. And, you know, you can take a set of technical ideas in your head and share them with the world by building a great product. And that’s a fantastic thing to do. You can share them in the world kind of one on one, you know, mentorship, teach people, learn small groups, also a great way to spend time.

[50:54] But the multiplication factor of doing a talk or even more of writing something is so much higher, right? Like there are so many more people that you can share that with and it lasts for a much longer period of time. And so just having something written on my blog, even that I wrote like a decade ago that I can share with someone and say, you know, here’s, you know, here’s how to think about this problem.

[51:21] Here’s an insight that I wanted to share with you or have people discover that organically is just super powerful. And so writing lets you scale out the impact of your expertise in space and time in a way that’s really hard to do in other media. I think with video and with podcasts and so on, we’ve seen other ways to do that. But I think writing remains kind of uniquely powerful. And then there’s also this idea which is this kind of core belief culturally at Amazon.

[51:53] And I’ve obviously been affected by this over the years that, you know, writing forces a level of mental clarity that speaking, making slide decks, etc doesn’t. And, you know, that’s something that has also really been my experience of sitting down to write something down forces me to think that through at a depth that I wouldn’t have been been forced to think it through without that. And so I saw one of your early conversations with, was with Leslie Lamport, who kind of takes that a step further and say, hey, you know, it’s formal mathematics that is the next step there.

[52:31] And I, I love that point, but I, I, I think writing is this really accessible thing for, for people to do that does force a level of thinking. And so I do a lot of writing sometimes just for myself, right. Like I’ll, I’ll write a doc, not ever intending to share it with anybody, but just to sharpen my own thinking on a, on A particular point. And so it’s some of that combination of three things, right?

[52:56] Like I just have something to say and I want to say it. You know, I have something to say and I want to scale it out in time and space and I want to sharpen my own thinking on a subject or the thinking of a small group on a subject in a way that writing is just a super powerful tool to do.

Ryan:

[53:20] Definitely.

Marc:

[53:20] Yeah.

Ryan:

[53:21] I remember being surprised early in my career I had a manager or tech lead who we would write these docs on either designs or the strategy. And he said even if you just wrote it and you threw it away, it would still be worthwhile because you’ll realize things as you’re writing and that clarity will save you a lot of time down the road. And it’s interesting to me because a lot of engineers, they complain about writing docs docs and all the stuff around the code.

[53:53] They kind of hate that. It’s a sign of slow big company processes. And what would you say to an engineer like that who’s saying just let me write the code?

Marc:

[54:03] Yeah, and that’s a great point. And I think it really depends on the level of problem you’re trying to solve. And so, you know, if I look at, I’m going to pick on UML for a minute here, right? Like it’s a sort of semi formal software design process and not one that I’ve ever found useful because I think it just happens at the wrong semantic level. I think it’s bothered with details at a level that aren’t helpful.

[54:30] And I think a lot of the let’s go off and document this has, has a similar problem, right? Like does this actually require, require that level of reflection and thinking? And so I think what you know, for me separates a valuable doc writing and thinking process from a busy work process is understanding what you’re getting out of it and what you’re getting out of it might be an artifact to share with the future, which is super valuable.

[55:02] Either your future self if you’ve got a terrible memory like me or you know, new teams, new people, people you know, or I want to share something with customers or I want to share something with the world and so that’s super valuable. Or I want to write down something so I can think through a really difficult, often one way door, kind of hard to change technical decision or API design decision. And I’m not going to do that every time I make a technical decision.

[55:34] It’s not worth it because a lot of those technical decisions are either Easy or not as critical or can be just be taken back if we figure out they’re wrong. But I am going to spend my time that way when there are key decisions to make, when there are key insights to find. And I think it is that what is the purpose of writing that separates well spent time from poorly spent time. Now there are people who still don’t like writing, even when it’s well spent time, even when it’s like, you know, you have to explain this piece of technology to, you know, to a future team.

[56:23] I think that’s a skill worth developing. You know, sometimes you do need to, you know, eat your vegetables, you know, and it’s, it is a skill worth getting good at. And especially in documenting the core kind of technical decisions behind a design is so useful. And that’s useful in two ways, by the way. Like, one of them is, as we think about building a big system, we make thousands of decisions.

[56:54] And some of those decisions are very carefully chosen, very particular and very impactful. And some of those decisions are the best thing we could guess in the moment based on having no data to make that decision. And it’s super useful for people who are coming in to improve that system down the line to be able to look at the design and say which of these things were very carefully chosen and thought through and which of these things were arbitrary.

[57:25] And because the arbitrary things like, okay, well I’m going to change that and I’m going to just go ahead and change that because I have better data now, I’ve watched the system run, I can go and change those. And these other ones are like, well, let me really engage with the reason that we made this decision. Maybe it was non obvious, maybe there was some more advanced thinking. And so being able to kind of understand the amount of thought that went into a decision is almost as important as understanding what that thought was.

57:51 — Visibility and apparent expertise

Ryan:

[57:51] You had a really interesting blog post. This was from a while back. It’s titled the Four Hobbies and Apparent Expertise. And you introduced this really interesting idea. It’s a two by two matrix and on one side there’s doing versus discussing and on the other side there’s the hobby and the gear. Maybe I can overlay it for people who want to see. And then later you kind of liken that to your career and how.

[58:20] I guess maybe we could imagine the hobby is actually coding and maybe the gear is. Let’s just say it’s like your dev setup or something like that. You talked about these two aspects of being in depending on which quadrant you are, which is there’s this trade off between expertise and visibility where imagine you’re really into coding and you’re really into doing. You’re going to be phenomenal in terms of expertise, but maybe not as visible because you’re not talking with everyone about how cool your setup is and all of that.

[58:54] And on the flip side, if you’re really into the gear, or maybe you’re set up in this case and you’re really into discussing, you’re on all the messaging posts and that you might not actually be that good at coding, but you’re very visible and you have this apparent competence. And I thought that trade off was really interesting because I’ve seen that so much in software engineering too is there might be someone who’s really quiet coder, they never write anything, but they know everything because they’ve just been in the weeds all the time.

[59:26] And then there are people on the complete opposite end of the spectrum that writing all the time, speaking all the time, but maybe not actually practicing as much. And my question to you is, how do you strike that balance? Because obviously too far in either direction is not optimal. So how do you strike that balance?

Marc:

[59:44] Yeah, that’s, you know, that’s, that’s something that I reflect on a, on a lot, you know, and I do explicitly think that sort of being 100% on either of those ends is a, is a, is a failure mode. And I think, you know, I will say that I have a lot more personal enjoyment working with the people that are 100% on the doing side and 0% on the talking side. I appreciate and deeply, you know, deeply love their expertise, but I do think that, you know, they, they could have more impact and leverage if they, you know, swung a little bit away from that.

[1:00:22] You know, I tend to not enjoy as much interacting with the people who are 100% on the speaking side, but I think they would, you know, have a lot more relevant things to say, you know, if, if, if they, you know, swung a little bit back towards the center. The other challenge of being on 100% on the doing side, it sort of gets back to that. How do you find the really important problems? And you know, if your head’s down in your ide all day, you could very likely be working on the wrong thing.

[1:00:59] You know, something that, that isn’t as, as important, isn’t as impactful, you know, doesn’t have these properties that people want. So, you know, how do you find the optimal balance? I don’t have a recipe for, you know, what really is optimal? I tend to do about, let’s say 75, 25 kind of practitioner versus, you know, teaching and communicating, maybe 80, 20 at times. I found that about what feels right for me.

[1:01:35] I would say that, you know, they’re great people I work with, you know, from sort of 9010 on that scale up to about 50, 50 on that scale. I think, you know, outside of those, you know, folks tend to, you know, tend to get into trouble as practitioners, right? Like, you know, there are people whose job it is to be, you know, communicators. And that’s great as long as they have the curiosity and, and are clear about what they, you know, know and don’t know.

[1:02:04] But, you know, I found that sweet spot to that sort of 75, 25 point in my career and, and that’s what’s what’s worked for me, I think. And I think in this moment where things are changing so fast, there’s so much to learn, you know, swinging a little bit more towards the practitioner side, I think generally will help people. But again, you don’t want to go too far that way because then you lose the, you know, what’s important for you that comes with interacting with the outside world.

Ryan:

[1:02:37] On the doing versus discussing axis. I kind of view the doing one as if you were too far, you would be underrated, and if you were too far out, the discussing, you would be overrated. And if for someone who’s structuring their career, would you say it’s better to be overrated or underrated?

Marc:

[1:03:01] I think long term, if you’re using that terminology, it’s probably better to be underrated. I think being overrated can feel great in the moment, but it’s rarely sustainable and really gets you to where you need to be. I really enjoy, you know, things like sports and, you know, these sort of creative hobbies and, you know, crafts, because it does, you know, turn that, let’s say perception and reality knob to very much reality.

[1:03:42] Right. Like as a sports person, you can’t fool the world for very long. It very quickly becomes very obvious, you know, who can and who can’t. You know, I think as a, as a craftsperson, the same, right? It very quickly becomes obvious who can and who can’t. And I think it takes a little bit longer in a field like ours where there is so much kind of qualitative stuff that goes on. But I think long term, when I look at, you know, careers that I really admire and people I really admire, they tend to be people who are personally very honest about their level of knowledge and understanding and skill.

1:04:23 — AWS engineers he admires

Ryan:

[1:04:23] So people who walk the walk, not necessarily talk the talk. I see, yeah. About engineers that you admire. I’d be curious because you have worked at AWS for such a long time and you have seen so many legendary engineers. Who, at aws, do you look up to and why?

Marc:

[1:04:43] Yeah, I mean, just fantastic. One of the blessings of working at a place like AWS is I get to work with so many great people. You know, maybe because he’s retired, I’ll. I’ll talk a little bit about. So Allan Vermeulen was one of the sort of early engineers at AWS and original, say, huge contributor to the design of S3, a really big contributor to the design of a lot of. A lot of our database services over time.

[1:05:11] Al was actually the CTO of Amazon for. For a period of time when he realized, I think that wasn’t the job he wanted to do. But what I really admired about Al from early in my career is very clearly he was somebody who deeply understood the things he was doing and he could work in these two modes. I have a great memory of a 2010ish, you know, arguing with Al about some of the edge cases in the Paxos paper.

[1:05:49] And, you know, he was super deep at that level, but could also get up to the really kind of executive level and talk about, you know, cloud strategy and the way we should be explaining things to people and some of the, you know, sort of fundamental things that we need to be building. And I really admired that ability to work sort of almost at every level. And I was like, wow, you know, this is. This is something I aspire to and, you know, want to model my.

[1:06:16] Want to model my own career after. And so, you know, that’s. I. I think that is, you know, the kind of person I’ve really, you know, really enjoyed working with is. Is people who do have that, you know, do have that breadth. And I think, you know, one of the other things that is really admirable about a lot of these folks is, you know, they. They don’t want to be celebrities, right?

1:06:53 — Technical book recommendations

Ryan:

[1:06:53] want to continue their engineering education and really remain on top of things, deeply understand the technology. Do you have any top technical book recommendations?

Marc:

[1:07:04] You know, anybody who’s building distributed system things? I highly recommend Martin Kleppman’s book. I think there’s a second edition of that coming out soon. There’s a new edition of Quantitative Systems Design book, which I also think is great. Hennessy and Patterson’s Computer Architecture book. This is a super useful one that covers a ton of ground. I read a ton of fiction and nonfiction and mostly papers when I’m reading technical things.

[1:07:37] I find I find engaging at that level more useful for me. And by the way, that’s become way more accessible now. One of the great ways to dive into a paper is, hey, Claude, summarize this for me and then I can dive into it and read the author’s words. And I find that mode is great and it’s super accessible for people who haven’t been able to read papers in the past. But, you know, and then there’s also a ton of insight in, in some really old stuff too.

[1:08:16] For example, you know, some of the algorithms that we used in Lambda to manage traffic and manage bursts of traffic come from erlang’s work like 100 years ago on managing telephone call centers and his book about that. And so folks also shouldn’t think that, oh well, the industry is changing super fast.

Ryan:

[1:08:59] more

Marc:

[1:09:00] again, more, maybe more leveraged than ever before. You know, deeply understanding those topics.

1:09:06 — Advice for his younger self

Ryan:

[1:09:06] And then last question for you is if you could go back to your younger self when you just joined AWS and give yourself some advice, what would you say?

Marc:

[1:09:16] I think maybe be a little bit bolder. I really love the team that I worked with and, you know, especially in EC2 and in the early days in EBS, and I think I was a little bit more hesitant than was optimal about leaving those teams and looking for the next thing. You know, as, you know, my own learning and impact kind of, you know, tapered off a little bit in those places. And so, you know, I think I’ve changed organizations kind of in a big way four times in my career and maybe five or six would have been optimal.

[1:09:53] Not a lot more, but some more. And so, you know, don’t, don’t hesitate to think about, you know, what am I learning and who am I learning from and is there a better environment to do that, you know, more quickly and to learn more things? And, you know, I’m highly, personally highly motivated by being able to follow my curiosity. And every time I’ve done that in my career, I found that a valuable move and something that I’ve personally enjoyed.

Ryan:

[1:10:27] Awesome. Okay, well thank you so much for your time. I really appreciate it. Mark, thank you for sharing with the audience.

Marc:

[1:10:34] This has been super fun. Thanks so much.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

0:57 - Kicked out of college

5:35 - FBI showed up at his house

7:43 - Pre-IPO Netscape experience

25:19 - Joining Instagram as head of eng

29:12 - Why shrinking teams improves velocity

32:59 - Working with Mike Krieger

37:16 - Leading Cryptocurrency project at Meta

42:30 - What he is working on now

54:29 - Career regrets

56:27 - Advice for his younger self

Transcript

0:57 — Kicked out of college

James:

[0:57] I kind of got kicked out of college when I was just starting out when I was 18, but they hired me two weeks later as a full time computer scientist at Netscape. No, this was like early in my career. I was 18, I was in college. I didn’t go to my classes. I was just programming my computer. The library computer information at Penn State started using my software that I uploaded and they learned I was local.

[1:26] So they actually had me come in and they wanted me to come in and actually help them with migrating all their mainframe stuff to PCs. Meanwhile the computer science program. I didn’t go to any of my classes, so they kicked me out. But totally like the left hand not knowing what the right hand was doing. They were like, hey, we want to offer you a full time staff position. And I was like the first person hired at the university in over 50 years without a degree and a full staff position.

[1:53] I had to call up my mom and tell her this, you know, like, hey, I got bad news and good news. She start crying, you know, like when she got kicked out.

Ryan:

[2:02] So am I understanding correctly? They, they booted you as a student and then they hired you as a full time.

James:

[2:09] They full time kicked me out. They sent me a letter saying, you need to reevaluate your educational goals so you can reapply later. But like you, I literally got a 0. 0. Oh my God. I literally didn’t go to any of my classes. Like I was just programming at home and totally, you know, was, was, was working on this software that I was like obsessed with. I was all night working And I was uploading it.

[2:33] It was open source at the time. It was called shareware. And the university just randomly got a hold of it and was using it. And they were like, this guy’s local. Let’s, let’s get him in. We don’t have anyone with this expert expertise. So I ended up working for the university for years.

Ryan:

[2:49] That’s so interesting. When you, when you got booted and hired at the same time, was that a positive, like a net positive for

James:

[2:56] you or is that, oh, unbelievably so. It was like, you know, I basically was able to go and like I, I did was interested in learning. So like I ended up taking pretty much all the computer science classes that the university had over the next few years. And I never got a degree because I never took any of the electives. But I, I had like 150 credits of like advanced comp Sci because I took them for free.

[3:20] I was staff. I could just go take. They couldn’t even argue with me. They couldn’t even say like, you can’t go into this class. Some of them were honors classes and you had to be an honor student or staff.

Ryan:

[3:30] You didn’t get the degree, but you got the education.

James:

[3:33] I did. It was more of a, yeah, I did, but I can’t say that I learned the most from school. It was more from, you know, from going off and pursuing my software, my curiosity outside of the university because I worked there for two years and people kept sending me money for this open source software. I’m like, I’m going to quit and start my own company and just go do this. So I did that. I left university and started my first company when I was 20.

Ryan:

[4:04] You said open source and in my mind that’s GitHub.

James:

[4:08] Oh yeah. No, that this was a very different world back in the early 80s. This was like you write code and you would use a modem and you’d connect to a bulletin board and you’d upload your code and it was called shareware freeware at the time. And then people would go on these bulletin boards. It was like, you know, one user per bulletin board. You had to sit there and redial until you could get your chance to connect and you could browse.

[4:34] You know, it was like, I’m talking like 1200 baud. You know, like you can almost read the characters and then you’d find the file and then it would take, you know, sometimes a half a day to download. You know, I mean a very different time.

Ryan:

[4:46] I see. Oh, so you Would connect to some shared server that was horrendously slow.

James:

[4:51] It wasn’t even shared. It was like bulletin boards were sort of these. Like, you know, I ran one too. So it was like this system that was running on someone’s computer in their house with a single modem and a single phone line on it. And you would connect to this and then you could use it. And then the way these. If you wanted to send a message, some of these were like networks. And the way they would work is they would all shut down at like one in the morning and start calling each other and syncing their messages.

[5:22] So like, there was like a few hours a night where they were just doing that. This was like early. This was like wild west. Yeah.

Ryan:

[5:29] I mean, I can just imagine all of the security potential security vulnerabilities from.

5:35 — FBI showed up at his house

James:

[5:35] Oh, well, it was a whole different time. Yeah, of course. Yeah, it was. It was. Security wasn’t a thing back then. And, you know, like, that was. I was a hacker. Right. That’s how I got into it too. As I was writing software that would take over the phone systems When I was a teenager, the FBI showed up at my house and asked me to stop.

Ryan:

[5:55] How did they find you?

James:

[5:57] They traced my number. I was like a kid.

Ryan:

[5:59] Oh.

James:

[5:59] There was two things. One, they traced my number. But the other thing was I wrote hacking software. And I was like 16 when I wrote this. And I was proud of my software. And so I put my name on it.

Ryan:

[6:13] Okay. Okay. That’s how they find you.

James:

[6:15] You can go still search the Internet and you’ll find it.

Ryan:

[6:18] What did it do?

James:

[6:19] It allowed you to completely take over an operator’s phone line. Like you could blow using frequencies. At the time, it was called freaking. And like, I had a Commodore and the Commodore had this like multi sound chip. So you could generate two tones at once. And you could basically call and blast 2600HZ into an operator line if they were on the right system. And it would literally blow off the line.

[6:43] And then the keypads generated different tones than a regular touch tone. So you had to have those. And there were a couple extra keys, KP and st. So you had to have these extra keys and know then the sequences. My software emulated all those frequencies, did the 2600 tone and then had stuff built into it to allow you to do anything an operator would do. Like you wanna listen in on somebody’s conversation, you could do that.

[7:09] You could do free phone calls.

Ryan:

[7:11] When the FBI showed up on your doorstep, is that who answered the door? Is that you or your parents? Okay. Did she know what was going on?

James:

[7:22] She knew it was for me. And so, you know, I was concerned. Like, you know, she was the. She didn’t understand what a computer was like. I didn’t either at the time, to be honest. I thought I was wasting my life. And she was like that you’re down on that video game, you’re not doing your schoolwork. I was like, oh, well, I did not know. She was wrong.

7:43 — Pre-IPO Netscape experience

Ryan:

[7:43] So at the beginning of your career, something stood out to me. I saw Netscape and I know Netscape is super famous for the first iteration of the browser wars. And I was curious, because you joined so early, what’s the story behind you joining and what was it like at the time?

James:

[8:00] Like, it sort of goes back to like how I got to California in the first place is like there was a company called Borland back in the day that was, it was sort of like where all the good engineers were in the valley and they wrote Borland C and Turbo Pascal. They were a languages company. They were probably the main competitor back in the early 90s to Microsoft. And I learned on Borland Tools. So, you know, I got a call because of my open source software that I had.

[8:31] They had seen that I was building developer tools, so they, they recruited me to come out and interview. I didn’t think they’d hire me, but they did. And so I went in there and I worked there for years. And when Netscape, when Netscape first came on my radar, it was through some Borland colleagues that went over. Mainly my close friend Lloyd Tabb, who went on to found Looker. And he recruited me over and it was probably like some of the worst judgment that worked out for the positive for me ever, because Microsoft was just beating us up bad at Borland.

[9:17] They were, you know, Visual Studio came out and like they were competing with everything and we were just losing market share, layoff after layoff. And I’m like, I gotta go get out of Microsoft’s way. Like, I, I’ll go join this little Internet startup. You know, Microsoft wants nothing to do with that. Turns out that was pretty, that was wrong. We kind of got square, more square in like the bullseye of Microsoft.

[9:44] They turned the entire company against us, right? Like, they, they were, they were off, you know, building, building the browser into the operating system and all these things that were just almost impossible to compete with. And it ended up being a pretty intense time. But it was Borland. You know, like, one of the things I think that’s Interesting about my career, and I think it’s a good pattern for others is like, you end up building great relationships with people and companies and that opens up doors and creates opportunity for you later in your career.

[10:20] So Borland, you know, building great relationships there, took me to Netscape, which honestly, even my current company, Guild, that I have three people that have been along that journey all the way since Netscape with me, and we’ve done five, six companies together.

Ryan:

[10:39] Wow, six companies. Yeah. I think when you’re like kind of going through your career, a lot of people, they give you that advice, they say, yeah, you should network or you should kind of. But in the immediate action is like, oh, do I. Do I start talking to people? Do I start, I don’t know, coffee chats? I don’t know exactly what. But if you were to sum up your understanding of what good networking is for someone who’s in tech, what does that look like?

James:

[11:09] Yeah, it looks different now to me than when I was younger. And I wish I had this filter when I was younger, but one, I think people that ask a lot of questions going in and are curious, it’s a great way to build relationships in a company when you’re first starting out your career. And I was the same. It’s like, I wanted good mentors around me and my good buddy Lloyd, who I already mentioned, he has the best advice.

[11:39] And it’s like, look, if you want to find a mentor, don’t go and ask someone, hey, I need a mentor. Go offer them help. Like, hey, how can I help you? And it will naturally evolve into a good mentorship for you. So try to be useful to others, be curious about others work. Don’t be afraid to speak up in meetings. Like, just start, just start talking and listening. And I think that it’s important to build this network and, or at least, you know, don’t.

[12:13] Don’t make enemies. Right? It’s like, I think that there was a quote. I can’t remember who said it. It might have been Bill Gates, but he might have been re quoting as, like, you know, friends come and go, but enemies collect. And, you know, I think that’s something to that, like, you know, build good relationships and don’t burn bridges. You know, I remember sitting at Netscape, the people sitting around me, like, even, you know, John Gianandrea sitting right next to me building the plugin interface.

[12:41] He now reports to Tim Cook running all AI at Apple and, you know, he ran all search at Google and, you know, the next person, you know, my, my first one of my first college hires, went on to found a company and sold it for 400 million a few years later. Like, the people that you’re sitting next to are going to look a lot different in 20 years.

Ryan:

[13:00] You mentioned Netscape, and in my research, I saw that the company went from just starting to IPO in 16 months. In today’s market, I feel like companies wait a lot longer for ipo. What was it like at the time and why did it IPO so quickly?

James:

[13:21] That was a crazy ride. Like, first off, the rapid scaling was insanity even by any standards today. I think when I started there, there was about 150 people total. But the engineering team was incredibly small compared to that. It was not majority engineering at that point. I think within a year we were over 3,000 people.

Ryan:

[13:46] Within a year. Yeah.

James:

[13:47] So I, I worked seven days a week and like, there was a time where I took two days off and I came back in and there were people sitting around me. I had no idea who they were. I’m like, it was just like a, a parade of people coming in, sitting. It was complete chaos. So, yeah, so that went very quick. But like, on the IPO question, I believe that was September 9th as like sort of a day that sticks in my head of 1995.

[14:14] I was there pre IPO, looking forward. Like, there are a lot of. It was a lot less regulation back then. You know, you didn’t have Sarbanes, Oxley or any of these things that, like, would prevent you from going public. It was a way to raise more capital. If you had a good story, you could go out. I mean, there was a crazy stream of like, the dot com bubble is crazy IPO after ipo, right? Like, and everyone was just buying stock that was a web company.

[14:47] They didn’t care what it was because they wanted to get in for the ipo. And a lot of people didn’t do well with that. You know, if you bought webvan or, you know, one of these, I even saw, I even saw a company that was formed to, to ship potatoes. Like, it was like potatoes direct or something. It’s like, wait, you’re. You’re going to charge 100 bucks to FedEx a bag of potatoes? That’s your business.

[15:09] But people were like investing in this stuff, and so they decided to come in and put some regulation around that, which was a good thing. So, you know, now and the competition, there’s a lot more companies out there competing, and people know that you need to have real revenue. You need to be able to predict what that is going to look like in a year. You have to show steady growth. So that’s evolved into a space that is a little more organized, a little more governed and a little more sane these days.

Ryan:

[15:42] You mentioned that you left your previous company to get away from the competition with Microsoft. And then I understand that Netscape’s like one of the most famous competitors with early Microsoft. And I want to know, when you got to Netscape, at what point were you realizing, oh crap, we’re competing with Microsoft?

James:

[16:02] It was pretty quick. We got in there and suddenly Microsoft put out this big like communication saying, like, this is an existential threat. You know, they looked at us like an operating system and people were going to start using web applications instead of shrink wrapped, you know, operating systems and software that they sold. You know, interestingly though, it ended up the, the thing that they were afraid of ended up actually making them one of the biggest companies in the world because it expanded.

[16:36] Like they adapted well. It expanded the market so dramatically that they, you know, they were able to offer services, reinvent and do all that stuff. But it came pretty quick. And you know, our mistake was my big mistake. A big learning for me was I was working on this browser and I kept thinking like, look, we’re going to build a better browser. The technology is superior. And it was, for a long time it was like, and it was like, so people are going to use a superior solution, right?

[17:08] Nope. Microsoft’s like, watch this. It’s not a technology war, it’s a distribution war. And we’re going to just bundle this and give it away and people will not use your stuff anymore. And that’s literally what happened, right? Like people didn’t care once, especially when like the quality became pretty equal, right? Like if you have two products that the quality is pretty equal, people are going to just use the one that’s the default.

[17:34] There’s no reason to switch. And that’s how Netscape made money. We sold these browsers, right? That was our original model. Like we sold these, they were like 35 bucks. We were growing fast. And then Microsoft gave it away and instantly killed all the revenue going to Netscape for the browsers.

Ryan:

[17:53] What was the internal reaction at Netscape among you and your peers when the competition was happening?

James:

[18:00] At first we felt like we were on a mission. The engineers, we felt like we were on this mission of interconnecting the world and bringing it. So we were really focused on that. So in one way it was validating, like, oh, this is a big space. Microsoft sees it but on the other hand, it was completely depressing and stressful. It’s like, okay, they just gave away our product. What do we do? Luckily, you know, we had the Netscape browser would default to netscape.

[18:00] com that was like the landing page. And we had a tremendous amount of traffic. So we started monetizing Netscape. com as the first like, web service and added like, email to it, like webmail and things like that. And so that started actually filling that gap. But we had to invent that on the fly. And it was a stressful time to get there.

Ryan:

[18:53] I was reading the Wikipedia page about the browser wars, and I saw, like, a really interesting part about the competition. It said, In 1997, September, they launched Internet Explorer 4. And on the release, they. They put a giant E logo somewhere in San Francisco. And it says, Netscape employees showing up to work the following morning found the logo on their front lawn paired with a greeting card signed best wishes.

[19:27] The IE team did. Did you work in the office? And see that?

James:

[19:31] That was literally. That wasn’t in San Francisco. They put that on our front lawn of our office, the one I was in. So. So, yeah, they went and put this big. I mean, it was. It was friendly competition in one way, like, you know, because the Valley engineers move around. So there was a bunch of my Borland colleagues were at Microsoft and there was some friendly rivalry, but, like, you know, it was interesting that they did that.

[19:57] And I actually know exactly the person who did that. And yeah, I believe that was Hadi Partovi. And when I went to after Netscape, I went to tell me networks. Hadi and I were, you know, I worked with Hadi directly. Like, he. We were both running parts of engineering. I actually reported in initially to Hadi and we would just basically tell stories about, like, yeah, we were doing this and he was like.

[20:26] And he was running IE and I was running the Netscape browser. So we got to tell these stories back and forth.

Ryan:

[20:33] Yeah, so I guess the story goes is, I mean, the superior technology didn’t work for Netscape, and the distribution advantage that Microsoft had was too much. And so ultimately, Microsoft ended up winning. I saw that Mozilla was created because Netscape just gave away the technology, gave away the browser. Why would Netscape choose to give the source code away?

James:

[20:58] Sure, it was a bold move at the time. And I believe the person who originally suggested it was Jamie Zelinsky, also known as jwz, and he’s pretty well known out there. So here was the thing. We were out resourced we were a small company compared to Microsoft. We had dozens of engineers working on the browser. They had thousands. It was very hard to compete. So we’re like, how do we completely change the game?

[21:27] And why don’t we invite the world to help us? We’re on this righteous mission to. Make people connected and share information. So why don’t we give away the browser? Let’s give it away. Let’s open source it and invite the world to come in and help. That was the only way we could see of engaging thousands of developers to come help. And it was a lot of work to get there. Right? And it did work, right?

[21:57] Like Mozilla became the foundation of lots of components that had made it into Chrome and there was Firefox and still going strong. So in one way that did work, but it was hard getting there. The story is even open sourcing the code. You couldn’t even distribute SSL source code back then. It was deemed a munition by the government. So we had the not only remove SSL from the open source version, we couldn’t even like show how the APIs would work.

[22:27] So we had to obfuscate APIs. And then we were a bunch of young idiots working on the source code. So we had like swear words in it and like inappropriate variable names. So we had to like print out all the code and go through it with like highlighters and like censor our own code.

Ryan:

[22:44] Why didn’t you just grep the code for bad words?

James:

[22:48] Because you don’t know what they are. If you have a, a variable named G G netlib breeding like bunnies, like how do you grep for that? Literally that was a variable name. So you know, so like we had to go through and, and, and make sure like it was something that wasn’t going to get us trouble sharing. Funny story. I think JW’s Jamie, he saved all the censored code and then he released it all. So like it’s, you can go out there and find all of the censored code from, from that.

Ryan:

[23:19] Oh, that’s funny. You mentioned the government deemed SSL like a munition. What does that mean?

James:

[23:27] It meant that it was like a dangerous technology. Like if people could encrypt stuff and the government or legal authorities couldn’t ever encrypt it. So back then, even like very small levels of encryption were concerning and we couldn’t distribute it because people could weaponize it if they got a hold of it. That was the theory.

Ryan:

[23:56] Before we leave. This part of your career is There any, I guess, lesson learned or favorite other story that you’d want to share? Otherwise, we can go on.

James:

[24:07] I think the lessons learned there is like, when you’re joining a company, understand what, what the actual, who the competition is and what the war is. I think that’s interesting, especially today, right? Like, everyone’s fighting to win the AI wars. Like, what does that mean? And what is the war? Is it a technology war? Is it a data war? Is it a distribution war?

Ryan:

[24:37] if I’m understanding correctly, you’re saying like, as an engineer even you should be thinking almost like an investor. Like where, which company you go to because there’s much more outside of the technology.

James:

[24:50] That’s right. Think about. I think that’s a good way to look at it is like, you know, look, if, you know, when we’re young, we probably don’t have a lot of funds to start going investing in companies. So we invest with our time, you know, we get stock often as a compensation, you know, for our time. So, like, look at your portfolio and see where you’re overweighted. And if that’s your goal, I think that’s a reasonable way to look at it.

25:19 — Joining Instagram as head of eng

Ryan:

[25:19] Later in your career, I saw that you joined Instagram and actually when I worked at Instagram in the past, you were the head of engineering when I joined as a new grad. And yeah, I kind of want to ask a little bit about that. So how’d you get hired there? Was it based off your network like you’re saying, or did you go through some other process?

James:

[25:38] Well, this is a great example of sort of failure leading you to a place that’s better, right? So like I went and I, I founded a company called Luminate. And Luminate I worked at for six and a half years. And basically it was Instagram shopping, by the way. It was, it was that product but for publisher sites. And we, we worked a long time and we couldn’t get it working. Now one of my board members was.

[26:03] This was Elliot Schragg. Elliot reported to, I believe to Cheryl at the time, and he was on my board when I sold the company. I sold it to Yahoo, but Instagram actually tried to buy it. So I met Emily White and Mike Krieger and stuff briefly. I committed to only one year at Yahoo. I was very good friends with the chairman of the board and so he convinced me to go there and help. One year came up and Elliot actually reached out and said, hey, Instagram’s still looking.

[26:39] Would you please just come and talk? And he didn’t know it, but I had been talking to other companies and I had verbally accepted a pretty large position. But I had to go meet just because it was Elliot and he wanted me to talk. So I went in and I met Mikey again and some of the other people. I’m like, oh, this is the job I want. This is like, this is the call. Like, the values were clear. I was passionate.

[27:10] I realized I used the product. I’m like, I actually use Instagram. The company I was almost going to go to, I don’t even use. I think at the end of the process I probably would have paid to take that job, but I wasn’t going to tell them that. But it was a job I really wanted. So. So it was my network, it was the timing. It was like my previous company that I couldn’t make run, that ran out the Runway to make that timing perfect for me.

[27:36] It was my experience working on image stuff at the previous company that sort of was also helpful.

Ryan:

[27:43] When I think about an employee in big tech, oftentimes when people consider starting their own company, the thinking is either that company is going to succeed and obviously that’s great, or the company will maybe not do so hot and I’ll just come back to where I was. But it seems like I’ve seen in other many careers, it seems like yours as well, that even if it’s not a incredible IPO success, there’s this middle ground where companies get acquired and then you can come in at very high roles.

[28:15] Like someone might have left, I guess the big tech companies at like maybe middle role or something like that. They start their company, it goes not good enough for breakout success. But they’re a leader at that company and they get, you know, hired back in at like a senior leadership position in big tech. And I was curious, what’s the mechanism for like the level that you come in at when you enter back after starting your own thing.

James:

[28:42] So there’s a lot of those, but like you can’t. I mean, people aren’t going to bring you in at a senior level unless you can back it up, unless you can show your experience. Even though I came from a smaller startup and went into Yahoo and I went from 15 engineers to 1,200 or something, I had previously ran large organizations at Netscape and tell me was even pretty large. So I had that background already.

29:12 — Why shrinking teams improves velocity

Ryan:

[29:12] I interviewed Ryan Olson, he was one of the original.

James:

[29:15] I know Ryan well Okay, great.

Ryan:

[29:17] Yeah. And so we were talking about Instagram Stories and he mentioned something really interesting in that project, which was that actually when the project started, you immediately kind of cut the size of the team almost in half or significantly just down to like, Ryan Olson, a few other people, so that the project could move faster. And he said that was a big part of the project success. And you were the head of engineering at the time, so I was curious if you had any thoughts on what.

[29:49] Did you see that that made that a good decision at the time?

James:

[29:53] You know, Instagram Stories was an interesting thing. Now, like, your natural instinct is like, hey, Snapchat’s gaining traction and like, we need to go do what Microsoft did, turn the whole company against this. But really, this is where your entrepreneurial skills come in. It’s like, hey, what we need is a small team that can move really fast. So, like, you want to set it up so that you have really capable people, you have the right types of engineers who are very proactive, and then you remove any roadblocks from them.

[30:24] So you give them a separate, you give them a separate area where it’s not dependent upon like, you know, deep, deep APIs and blocking points into other organizations. So you set it up so that they can move fast and that if that’s. So we did that purposefully. Right. And I think it was like three months we had. It was a three month project. We went from zero to releasing stories, you know, and it was a pretty big success.

Ryan:

[30:55] But I, I think maybe a lot of managers might naively think, oh, competitors coming, we gotta move fast. That means throw a lot of engineering resources at it. So here’s, here’s a 20 person team. Go, go, go. As opposed to. It sounds like you have a different philosophy here, which is a few. A small talent, dense team.

James:

[31:17] That’s right. With founders leading it directly. Kevin and Mikey were heavily involved and brilliant product thinking and execution at that level as well. When you get a small team that’s capable and you give them clear ownership and you remove dependencies, that is one way to increase speed. So if you get a large team, they tend to be interdependent. You start federating large pieces of architecture and pretty soon those dependencies end up slowing you down.

[31:52] So I think sometimes it’s better to start with something small and build on it rather than trying to build the big thing at the beginning and evolve it over time. And that’s sort of what this was, is like, let’s get it working, let’s get it stood up. Let’s. Let’s get this thing working and then plug it into the product and we’ll scale it instead of like, what does this look like at large scale? And let’s build that from assuming that that ends up often being a way less efficient way to get something done.

[32:25] And it’s definitely a lot slower. Mikey Krieger had, like, and it was the name of my conference room, it was called this value that he said, do the simple thing first. I don’t know if you recall this, but, like, hey, if we’re going to go off and we have a thesis before we go off and just pour the whole company in it, what’s the simplest, hackiest thing we can do to go prove this thesis True, that was part of the values of Instagram is like, what’s the simplest thing we can do to go prove this thesis before we scale it?

32:59 — Working with Mike Krieger

Ryan:

[32:59] You were hired as the head of engineering, and at the time, Mike Krieger was still the CTO there. What are the differences between that CTO role and the head of engineering role? And how did you partner with Mike Krieger? And what went well? What didn’t?

James:

[33:15] Yeah, so Mikey brought me in. The engineering was hitting about 100 people. It’s a common place where teams start to break. And Mikey’s brilliant. He was actually a really gifted manager too, but he just didn’t want to do that. And he wanted to focus on the technology and even coding. That was his true joy. So he brought me in. So the initial confusing things to the team was like, what are our different roles?

[33:44] Like, you know, who do I listen to? I’m used to getting directions from Mikey, so you have to be pretty clear. So the first thing that we did is like, we wrote our job descriptions and we published them to the Org. And so, you know, and classically, a CTO is like, works with customers, new technologies, works on the product, and the technology roadmap experiments, all of these things, builds the external engineering brand.

[34:12] And the VP or the head of engineering is usually a people management thing. It’s like hiring, execution, org structure, performance reviews, all that fun stuff, right? And so think technology, roadmap for cto. Think process people staffing, execution with a VP of Eng. So we published that and we even came up with like, flowcharts like, hey, I want to. I want, you know, we. We put it out to the team.

[34:44] We’re like, hey, I want to. I want five more headcount. Go do this project. And it’s like, come to me. Hey, you know, I Want to use this new technology? I think it would be cool. Go to Mikey. So, like, we. We also put, like, a roadmap out there, and then it took some evangelism, you know, like, hey, you know, to get people to learn where to go for what they needed. People often confuse this, by the way.

[35:06] You know, it’s like people confuse the difference between a CTO and a VP of Eng. Like, I get called into a lot of companies saying, hey, we need a new cto. And I’m like, why? What’s breaking? And they almost always describe VP of Inch problems. I’m like, I don’t think you need a cto. I think you need a VP of Inch. It’s a very different person.

Ryan:

[35:26] As we kind of wrap up this Instagram leg of your career, is there anything that you think I might have missed or kind of, like, lessons that you learned from this part of your career?

James:

[35:35] Yeah, no. You know, I. I think that the one thing that I remember is, like, you know, when I came into Instagram, it was a different company. Everyone’s nervous when they go into a new job. And they brought me in right when engineering teams start to break. But, like, these weren’t breaking. And I was very confused at the beginning, and I was starting to feel like I was a dinosaur. I’m like, wait, you know, this is all working.

[36:01] This isn’t breaking. Like, what kind of management voodoo is going on here that I don’t understand? And then about 150 to 200 engineers just started breaking, and I’m like, oh, wait, these people are so smart, they’re able to IQ their way past, like, that normal barrier. Because what usually breaks is communication lines. Like, you have to hold a lot of stuff in your head, use face, familiarity with people.

[36:25] It’s called, you know, Dunbar’s number, all that stuff. But, like, the people were just so gifted and smart at Instagram, they were able to IQ past that. Plus, the great thing, the reinforcing thing about Instagram was the best software is built by people who use it. And, like, everyone in Instagram was a power user, right? Like, and even if you go back to Boomerang, do you know. Do you remember Boomerang?

[36:51] When it first came about, it was. I think it was an intern. Tim Leonardo, I think, was the intern. He went and built it as a hackathon in, like, three days. Like, one of the most engaging features the Instagram released was a hackathon built by an intern in three days. Like, it’s just amazing to work in a team where you’re the primary customer and you’re building stuff for yourself. So anytime you can do that, like, that’s awesome.

37:16 — Leading Cryptocurrency project at Meta

Ryan:

[37:16] Moving to the next part of your career. I saw you. I remember when I joined Instagram, all these very high profile people were leaving Instagram to work on Novi. It was the hot, hot project at the time. I was thinking, oh, wow, this, this new crypto thing must be really big. And so, yeah, what drew you to working on Novi or the crypto project at Meta? And what was the impact you saw on it?

James:

[37:42] So inside Meta, I had met David Marcus. David was running messenger at the time and we had just met, but he’s like, hey, I want to run something by you. And I went. And he was like, don’t tell anyone, but I’m going to leave messenger and this is what I’m going to do. And this is my first meeting with David. And he’s like, I’ve heard good things about you. Would you consider coming and running the engineering aspect of this?

[38:08] And he was selling me this crazy crypto story, like, and I loved Instagram, right? And I was like, you’re insane. Like, that’s a crazy thing. Why would I go join that? And like, I just couldn’t get it out of my head. And then like the next day it started hitting me like, oh, wait, this is an area for impact. And so I went back the next day and David was kind of grinning and he was like, I knew you’d be back.

[38:35] And he’s like, he called it, he’s like, I call this Dementel virus. He goes, it takes about 24 hours for to infect people. People. And then they get it. And I was like, darn you, David. And long story short, I’m like, okay, I’m gonna come start this. I was heavily invested in the mission, which was financial inclusion and bringing safety and low cost money movement to people around the world that needed it.

[39:03] And I loved that idea. And I remember telling my product partner, Kevin Weil at the time, hey, I think I’m gonna go do this. And he, I explained it to him and he was kind of like, had that same look I had. And then like the next day he throws a 30 minute meeting on my calendar. And I’m like, oh, he’s gonna try and talk me out of this. And he comes in, he goes, I think I gotta go do this too.

Ryan:

[39:30] What?

James:

[39:32] Like, oh, this is gonna be tricky. So I connected him with David and I’m like, great, this is gonna be a tricky one to navigate, but it well, and you know, we went over and started the Libra project at the time.

Ryan:

[39:44] How did the project go and you know, what were the pivotal points in it?

James:

[39:47] Oh man, I think it’s pretty public how that project went. Like we, we weren’t able to ship it right. Like it was. There was a lot of pressure on, on Facebook at the time, just political, political and regulatory. And like they weren’t excited about us, what their view was as thinking we’re going to rewrite currency. It’s like not what we were doing, but like there was not a lot of strong understanding on the government.

[40:11] They’re not the best, deepest technologists, but they didn’t want us to release this. And it was a shame. David tried to do this like the right thing. He’s like, we announced what we were going to do, that we were going to work with regulators and everything. In retrospect, we probably should have just released it and asked for forgiveness. But that sort of put us into a tailspin where we had to keep re architecting the project to be regulatory compliant and took us years to do that.

[40:42] And ultimately we got it to the point where even the government said this was the platinum version of stablecoin, but they still didn’t want us to release it. And I think that’s when we gave up. And we were a pretty big org by then. I think we were like 3,500 people. And yeah, they put all payments and everything under David at that point. And I think in a funny moment David came to me. Well, I’m like, I can’t keep doing this.

[41:13] I’m going to go do something different. And I loved David and, you know, felt like he was a friend. I didn’t want to let him down. So it took me a couple of weeks to get the courage up and practice my resignation. And I was ready. And I’m like, I’m gonna go in our one on one, here’s my story. I’m gonna give them all the time, but tell them I need to move on. I go in there and I sit down and David goes, well, before we get into whatever you want to get into, I have some news to share with you.

[41:40] I’m like, what? And he’s like, I’m gonna leave. That’s what I was gonna tell you. I’m like, what are you gonna do? And he’s like, I think I’m gonna go get into my entrepreneurial roots. I’m like, how? And like, one thing led to another. We kept talking and I ended up leaving Meta for a couple of years. And I went and started lightspark with David because I was invested in this mission of. And that’s what like, Spark’s doing.

[42:03] And I love it. It’s just an amazing mission of, like, you know, the way I think about it is finance is the last area for disruption, you know, But I do think, like, the cost and latency of moving value globally needs to go to zero. Because when it does that, when you can move money, like packets of information, like you can stream money, you can move at low cost to areas that need it, it’s just going to be revolution.

[42:27] I have no doubt that that’s going to happen at some point.

42:30 — What he is working on now

Ryan:

[42:30] And so you were working in the crypto space and now you’re working in AI. What made you leave the crypto space if you know there’s impact there?

James:

[42:41] Yeah, first off, I’m like a crypto skeptic. I never was a buyer into all of this stuff, but I do think that it’s really good for money movement and as a base la. So what happened there was like, I was working and I co founded lightspark and David wanted to do it in la, so I figured I’d move down there. I commuted for two years down there and I formed a team and built it. And I was kind of like at a point where the team was formed.

[43:11] I’m commuting three days a week. I didn’t feel like I was adding a lot of value until we were at the next level of scaling. So I talked to David and, you know, David and I agreed, like, hey, yeah, fine, I’ll step back. And, you know, we’re, we’re, we’re very good friends. So it was like an easy conversation. I groomed a successor and stepped back. And at that time, Meta learned I was free. And they were like, hey, you know, we have a role we want to run by you, we hear you’re free.

[43:42] And I’m like, I don’t know what I would, what that role would be that I’d be interested in. And they’re like, well, what about Dev infra, you know, like running all the developer infrastructure at Meta? And I’m like, oh, hold on, that’s my background. Like, I’m a developer tools person. That was the first 10 years of my career. Like, and it hit me like, oh, wait, what was I even doing at Instagram and all these things?

[43:58] I’m a developer tools person. Like, that’s really what I want to do. And with AI coming into a picture, it was like, awesome. We get to go write science fiction stories about how developer productivity is going to be and go try to make that happen with unlimited resources. I’m like, sign me up for that. So I came back and I love that team. Dev Infra is amazing. And such a senior team, like 1,000 engineers building all of the developer tooling for inside Meta.

[44:35] It was just amazing. I love that job. Did it for almost two years. And the only reason I left is because I had that observation and I had to go start this company.

Ryan:

[44:49] And what was the observation?

James:

[44:51] Well, so we built something internal called devmate. Devmate was an agentic platform that allowed engineers to sort of go learn from each other and then scale these agents. And these agents were working on tooling, not just coding. They were driving tools and doing all types of things. And it went viral internally. We couldn’t keep up with the demand. Within months, it needed an entire dev server per devmate instance.

[45:22] And there are 40,000 plus engineers at Meta and pretty soon everybody wanted an instance and some of them wanted three instances. So you’re like, how do we allocate all dev server? But it was such strong pool. You’re like, oh, I see the dynamics that work here. Like 40,000 developers is fun, but it would be fun to go do this for 40 million. So we took a lot of these lessons and we’re like, yeah, we kind of think we see around the corner of what the world needs for this to work inside their companies.

[45:57] And that led to the formation of Guild.

Ryan:

[45:59] When I look at the AI space, it feels like a bloodbath or it’s incredibly competitive. When you saw the space, I guess what, what gives you the confidence that you will compete well or what is the niche that you’re going to focus on?

James:

[46:15] First off, I don’t have the confidence, but I have the drive and I’ll tell you that much. It’s like I deeply believe that what we are building is needed. So there’s that. But I’m always looking out of the corners of my eyes of who else is building this and who might now execute. That’s just what an entrepreneur does. So if you look at a lot of the startups and a lot of the companies, I think a lot of them are building for the future past.

[46:48] What I mean by that is they’re building for things that fill in gaps of what LLMs will probably be able to do. Like look at cursor, it’s autocomplete. Now LLMs are starting to get to be able to completely write code without an IDE it’s like, okay, you wrote you build a product on a bus stop, not where the end of the line is. You know, so what’s that end of the line and like what is needed regardless of all of that.

[47:15] So what we’re building is a control plane and it’s a, it’s a layer of infrastructure that allows you to manage these agents inside a company. Like you know, your agents, if they’re rolling around in your infrastructure, you want to treat them like employees, right? Like some of them they, some of them you want to have access to things, some you want to restrict, you want to report on what they’re doing.

[47:35] You need to observe auto. You don’t want one agent just like tanking your token budget. You need circuit breakers, you need a governance layer for that. Regardless of where this ends or which model company wins, you need this. Right? So we believe that we’re building something that’s needed and in any like scenario where anthropic or OpenAI or meta or somebody else wins, you just need this layer.

[48:07] Let me paint you a vision of where I think like things are going to go. Okay? So I think a few years, like right now inside the company, you have a bunch of internal web applications, you have your HRIS systems and you know, your intranet and like developer like CICD tools and you have a lot of internal apps. I think there’s going to be a layer of like thousands of agents that will sit on top of this and help drive them.

[48:33] Right. For example, we build an agent that we use on our own system here that basically doesn’t make duplicate bugs possible. Right. It’s really simple thing. It’s like you submit a get issue, it fires a webhook, our agent wakes up and goes, let me look and see if there’s one that’s like this and it just marks a duplicate and goes back to sleep. That’s different than a coding agent. Right. So you can imagine anywhere where there’s a tool integration prompts and evals.

[49:04] You can go into design, you can go into product, you can go to hr. It’s just basically workflow management in a company. And that’s the thing where we think it’s going. But if that’s the case, then, you know, you’re just like people. You probably don’t want your engineers having access to your finance systems, you don’t want your finance people having access to your code base. Like, you know, all of these things you’re going to need to set up policies around and even like regulatory policies.

[49:32] You’re going to need that, especially in a large public company. So you need layers to be able to enforce that. AI is non deterministic. Right. Like, it’s like you can’t guarantee what the outcome is based on any inputs. If you’re building a stable infrastructure layer, you need it to be deterministic. So you need to put a deterministic layer on top of this non deterministic technology. It’s the only way right now.

Ryan:

[50:00] Okay, so it’s almost like this harness that. Well, yeah, control plane is a great way to put it. Control plane for the agents within, maybe a corporate environment.

James:

[50:10] Centralize it. Like right now you see people using these AI agents. We had one customer that they have their developers running agents on their laptops. They don’t know where they’re running. This is one that blew through their token budget. One agent in 12 hours and they didn’t know. So you centralize that. Okay, here’s a central runtime. You make it so agents can work together in a secure way. So you put bubble wraps around them.

[50:41] You don’t want these agents just yellowing at your infrastructure. Right. So you have to put control around them. I use this analogy. There’s this movie, Gremlins. You remember Gremlin?

Ryan:

[50:53] I haven’t seen it, but Gremlins.

James:

[50:55] Yeah, it was like this easy Spielberg movie where these little furry creatures started multiplying quick and they just were causing chaos in town, ticking over the electricity, power grids. Like AI agents are like that. Right. It’s like they’re just like, once they start multiplying and they’re like, you can’t predict what they’re doing. Like that’s a problem. So you need to be able to put a layer there that provides some predictability and control around that.

[51:18] That’s what we’re building.

Ryan:

[51:19] Yeah. At this point I feel like you have so much management experience. So I wanted to discuss this high level management philosophy that I see in the industry, which is that I see that this Elon style, Elon Musk style of management is kind of like getting in the details, micromanaging. And I hear a lot of people who are against that as well and curious what side of that you’re on. And maybe we can discuss the pros and cons of each of the approaches.

James:

[51:52] Yeah, sure. Well, look, I think that there’s different tools for different jobs. And so what you’re describing when you talk about these two different management styles, they’re in two very different environments. And so I’m not a micromanager. But I’ve also been fortunate enough to have incredibly capable people to work with. So, you know, you can think of an org as a top down org, like where you have a leader with a very clear vision and he’s hired, you’re hiring a workforce, he or she’s hiring a workforce to help achieve that mission.

[52:24] Or you know, you can be the opposite of that and consider yourself at the bottom of the org chart and you’re trying to act as a tool to help that team achieve their vision. That’s the one. I am. Now that being said, there are times where if your team isn’t performing and I’ve inherited some teams that were not working well and things like that, where I’ve had to go in and micromanage, but not for long.

[52:56] You want to only use it as a tool in my opinion, to fix things and then step back out of that. I think that our job in technology is a creative. This is a creative job. You don’t know what it looks like. Often you just know what outcome you want. And if you’re going to do that, you have to free people. You don’t want to hire smart people and think for them. You want to just focus them on the outcomes.

[53:26] You want to like clear what’s blocking them. You want to be able to probe and like ask great questions and course correct them. But if you start micromanaging really smart people, you’re going to lose them. Now if you look at some of these classic examples that you mentioned, like Elon and stuff like that, think there was a lot of, lot of lost a lot of smart people that weren’t willing to work in that environment.

[53:51] But I didn’t understand the environment. Maybe it was necessary. I can’t speak to that. I’ve built my career around hiring really smart people much smarter than me and getting out of their way and just supporting them, you know, trying to make them, give them the space to be the best that they’re at. And so I think, you know, not saying there isn’t room for this micromanagement, like get down into the details, you need to do that every once in a while.

[54:21] But like that’s not. If you just, that’s your only mode of operation, that’s your only tool. It’s only good for a very specific thing.

54:29 — Career regrets

Ryan:

[54:29] When you look back on your, your whole career, is there any career regret that comes to mind that you think other people could learn from?

James:

[54:37] Oh man. So yeah, there’s a lot actually. And I wouldn’t say they’re regrets. Because, like, I’m pretty happy where I’m at and, like, everywhere led me here. So, like, you have to also look back and go, I’m not going to punish myself over mistakes if you’re happy where you’re at. I’ll tell you things though, that, like, I wish I would have known or I wish I would appreciated when I was younger is like, I’m.

[55:01] I’m not an intern age anymore. I’m a little older and, you know, time is my scarce commodity now. Like, so back when I was younger, it was okay to like, beat my head against a wall for six, seven years. I’m like, I have all the time in the world. I would like to have that time back and spend it well. So, like, be sure you’re doing what you love and be sure you’re having good days, because that’s really your scarce commodity, you know, so that’s one thing.

[55:29] Go where the smart people are. Like, I learned this by accident. Like, I was fortunate enough to get recruited into Netscape and Borland, like, where all the smart people were, like, going there really helped build me a network that’s just, you know, people, the early engineers at Netscape and Borland are kind of like owning the Valley now and Marc Andreessen and, you know, so. So that, that’s also good.

[55:55] And that’s where I got the best mentors and learned the most. And I’d say one simple tactical one. If you’re just starting at a company like this may sound silly, but I would say I wish I would have just maxed out my 401k on day one. Like, just go, max out your 401k. It sounds like, like, you’re going to get a reduction in salary and you may not be starting out at high, but trust me, you’ll forget it.

[56:17] But, like, when you’re my age, that there’s going to be a giant amount of money. I wish I would have done that earlier. So that’s the other tactical thing I would do.

56:27 — Advice for his younger self

Ryan:

[56:27] If you could go back to the beginning of your career and give yourself some advice, knowing everything you know now, what would you say?

James:

[56:34] I would speak up more. Like, you know, I. I remember thinking, being afraid to speak up because, like, I was worried that people would think I was an idiot. But it turns out that, like, those are the people now that I notice are the people that even if what they say is off, they’re not afraid to speak up. So speak up. Like, be curious. Like, I wish I would have networked more and asked people like, more about what they were doing.

[57:01] And, you know, one simple thing is, like, it didn’t occur to me. It’s sort of a version of go where the smart people. Or like, I was seven years in Pennsylvania working. That’s where I’m from. It didn’t occur to me to, like, go to, like, I got recruited into Borland. But, like, in retrospect, it was like if I were an actor and like, it didn’t occur to me in seven years to go try Hollywood. It’s like, you know, it made all the difference was getting into the Valley where there was so much opportunity and resources.

[57:31] I wish I would have done that earlier.

Ryan:

[57:34] That makes sense. Awesome. Well, thank you so much for your time, James. I really appreciate it. It was great to meet you.

James:

[57:39] Yeah, great. Thanks for taking the time. It’s a fun conversation.

A clip from our conversation got millions of views. In it, Ethan says:

“One of the hardest things for people to understand is I’ve identified a legitimate weakness in my boss. I go to my skip. Why doesn’t he do something?

Although many people were disgusted, Ethan was a high performer at Amazon and shared the incentive structure with unusual transparency. He’s not commenting on what is right or wrong. He’s saying what is.

From my experience, as my career grew I worked with more high performers at Meta. Some people play the game selfishly while others play the game while also focusing on what is right. But common among all high performers is they understand “the game” and its incentives.

When you understand the incentives, you can focus your limited time on what is rewarded. When I was a junior engineer, I spent so much time working on random code cleanups and miscellaneous projects that were not impactful. Later, I still worked hard but I focused on projects that mattered.

In the example where Ethan is the skip-level manager and an engineer has a problem with one of his front-line managers. Ethan explains how the incentive structure naturally leads him to not prioritize the needs of the engineer.

You can hate him, but he’s just playing the game. You probably just hate that game.

If you hate the game

I wouldn’t try to change the game since the effort is usually not worth it. The best move is often to find a different team, role, or company.

It’s unfortunate, but it’s the reality. The dream is a job where the incentives match the work you care about. Early in my career, I got lucky. I loved coding and optimizing infrastructure, which Meta rewarded well. Those situations do exist in tech, but you may need to do some searching to find them.

I asked Ethan how to avoid office politics, and he summed up your options well.

One more thing: if you hate the game but try to play it based on how you think the incentives should be, you will get burned. Best case, you’ll do okay but not as good as you could have. Worst case, you’ll get fired for being difficult or doing work that doesn’t matter.

As Ethan said, “do yourself a favor and get good at this [politics].”

Full conversation is here if you’re interested:

I plan to write more about the most interesting takeaways I learn from my podcast guests. I won’t write every week, but I will write whenever I have something interesting to share.

A few personal updates for those following what I’m working on:

1. Keyboard – We launched on Kickstarter and hit our funding goal in 8 hours after launch, thank you if you backed it 🙏

2. Podcast – It is still cash flow negative, but I tested a sponsorship with my friends Evan and Stefan from Hello Interview (it’s here if you’re curious). It went well, so I might try more in ways that don’t take away from the content.

Also, I’m thinking of making this newsletter’s branding more general. That way I can write about anything interesting, not limited to only software engineering career growth. If you have thoughts on this, let me know!

Thanks for reading,
Ryan Peterman

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:02:33 - Empire Building

00:31:54 - Stealing scope

00:44:03 - Managing out via reorgs

00:51:01 - Bad managers and mutiny

01:01:43 - Political messaging

01:11:48 - Handling politically skilled operators

01:20:01 - Orgs trying to steal scope

01:30:26 - Handling difficult people from other orgs

01:36:19 - Handling weak managers

01:46:30 - Backchanneling

01:52:04 - Influence without authority

01:58:04 - Sexual harassment

02:00:49 - Skip overruling firing

02:05:39 - How to fire managers

02:11:31 - Leverage when people are getting fired

02:24:01 - How to grow past senior eng

02:43:01 - How to avoid politics

02:48:15 - Advice for younger self

Transcript

00:02:33 — Empire Building

Ryan:

[00:02:33] The first topic I want to go over is empire building.

Ethan:

[00:02:39] Empire building, I think, is a bad thing in that, as defined, it means focusing more on getting raw number of people than on what you’re doing or the impact you’re having. So why does it exist? Empire building exists because it’s rewarded. And it’s rewarded because counting people is the easiest thing to do. And so when a leader or a set of leaders are deciding who’s ready to promote, how much impact you had is subjective.

[00:03:13] One person thinks the project was hard, another thinks, oh, that was pretty easy. But no one can debate, well, Ethan has 42 people and Ryan has 17. Ethan is the bigger leader, is the conclusion they’re making. And that encourages empire building because it’s what’s visibly measurable. And so what I found is that drives leaders to want more people because that’s going to be their tool. And some companies absolutely have written thresholds of to be eligible for this promotion, you have to have this many people.

[00:03:53] So of course, if you give high performers a goal, they’re going to figure out a way to hit that goal. And if that requires claiming they need people or seeking to take over other groups, they’re ambitious, they’re going to find a way to rationalize that to themselves. I need those people.

Ryan:

[00:04:11] I saw that there’s this desire to change the incentive structure, at least at the company that I was at, where managers could be promoted, not necessarily just directly proportional to how many recursive reports you have. Is that possible to do?

Ethan:

[00:04:27] Of course it is possible. It takes more effort because impact is harder to assess. Right. The impact an engineering manager has is somewhat, somewhat fuzzy. What’s most important, is it operational performance? Is it new features shipped? Is it whether or not those new features resonate in the market and make money and you end up with difficulties. Like maybe my product directly generates money, whatever I’m doing gets directly sold and customers pay for it.

[00:04:57] And what you’re doing is infrastructure. Well, now, how do we compare that? Like, you’re keeping the systems up, but I’m actually the one bringing in all the money. Who has more impact? Again, that’s where people get sucked into. Why have 42 people and you have 17? I must be a bigger manager. That’s how that happens. It is, of course, possible to make value judgments, but then they’re also harder to defend.

[00:05:26] I have to look at you and say, ryan, we love the work you’ve done and you’re on the right path, but Ethan was more impactful. And that just makes you mad. Probably, unless you happen to agree like, oh yeah, that guy’s a star. It’s just all these things make assessing impact difficult and dangerous and that’s why it doesn’t get done as much as it should.

Ryan:

[00:05:52] You mentioned the written thresholds for promotion and since you were VP at Amazon, I know it’s been some time, but off the top of your head, do you remember the rough written thresholds for how many reports you would need to be considered for senior management director and vp?

Ethan:

[00:06:10] So interestingly, those thresholds did not exist while I was there and they existed in whisper numbers, but they were not written down. They came about after I left and people I was coaching, still at Amazon or talking to told me very clearly that the number that got established to become a director in some orgs was 80 people and in some was 90. So I became a director in the early days of the company with 22 people because it was different times and the company was smaller.

[00:06:42] And that crept up and up until when I was at the end of my career, we did have like a sense that the company wanted us to look for people with about 75 or 80 people, but it got written down as 90, at least in some organizations. And the thing that’s funny about that is one of the Amazon leadership principles says there’s no bonus for additional headcount. But now that it’s written down that I can’t get to this very well paying level without 90 people, there damn sure is a bonus, right?

[00:07:16] There’s like a several hundred thousand dollars a year bonus quite literally for accumulating 90 people. And I’ve talked to leaders who. Their boss trying to get them across the threshold was like, we’re going to put these people under you for six months, check the box and give them back to the person we took them from. Because that’s the game they have to play if they want to, you know, reward a person in their team.

[00:07:41] And of course that’s insanity where we’re moving people around and changing their lives just to hit a number.

Ryan:

[00:07:49] But if you did that, let’s say I’m a VP and I have a very loud director or very loud senior manager who wants to become a director. And so I go, okay, to retain them, I’m going to give them some reports from over here. And now they’re going to be eligible for director after they get promoted, though, they need to sustain those reports to sustain that position. Right. Like you couldn’t just move them back.

Ethan:

[00:08:15] I mean, you can again, it depends on the company. But you can play all sorts of shell where for example at Amazon if you drop below the number of reports, there’s like a grace period, six months, nine months where if you can tell a story about how they’re going to get back to that level or how future growth, it’s okay if they’re below it temporarily. The other thing is no company, no company I know of really systematically down levels people, what they would do is they’d say okay, well Ryan, you’re a director, your team is no longer director scope.

[00:08:51] You’re going to have to find a director level role. And so then it’s up to me if I can do that in my org or I move you. But I’ve still achieved the goal for you. And you absolutely see I have seen not just in Amazon but other places when headcount is the main issue. I’ve seen things moved around and I certainly when people normally come to me, it’s with the complaint. My boss actually told me that he’s putting me under my peers so that my peers team will look better.

[00:09:22] So the reason I’m being reorged under this other person is not for any business reason, but because they don’t have the scope to get the promotion. And so I need to go be their scope. That’s what I’m being told.

Ryan:

[00:09:38] When I was an engineer and reorgs were happening, I think you don’t know all the behind the scenes of how orgs are managed and so you’re told there’s a certain reason for the reorg however, and which may be true, but there may be a secondary an ulterior motive of this person also needs to get promoted. Do you ever have a experience like that where you saw reorg happening in other org you didn’t know the exact person who needs to get promoted or what, but you could just tell based on how it happened and what happened that oh, there’s something else going on behind the scenes.

Ethan:

[00:10:15] Yeah. And I would say there’s always something else going on. Meaning all managers know, all leaders know that too many reorgs cost two people and so you don’t want to do them too often. What that means though is like you wait as long as you can and then when you need to do one, you’re asking yourself how do we get everything we want? How do we, how do we. If there’s someone we don’t like, how do we give them less so they’ll move on?

[00:10:42] If there’s someone that we really think is high performance, how do we set them up for success? How do we retain key people? And yes, hopefully the start of the reorg is how do we make the business better? But as soon as we figure out what we think we need to do for the business, then we’re trying to figure out how much else, you know, this ship is leaving the dock. How much else can we throw on the deck as it’s pulling away that’s going to help us.

Ryan:

[00:11:07] I just trying to figure out the exact mechanics, like maybe a pastry org that you’ve done, where you tried to cram a bunch of different ulterior things into it. And how does that actually play out?

Ethan:

[00:11:20] Well, a couple things happen there. First, there are usually multiple people involved. And so sometimes you have to either get them on board with your second motive. You know, you have to tell them, like, yeah, I want to help this person get promoted as a part of this reorg, or I want to set them up. Are you supportive of that? Sometimes you have to tell a story like, no, I’m not doing that. It’s really this reason.

[00:11:47] And so there is some cleverness to justifying because even for a senior leader, usually a reorg is a big enough change that somebody will be auditing it, and you’ve got to be able to explain it to them. So you’re coming up with your narrative. You’re a storyteller, and you’re coming up with your narrative about why this is the best possible reorg that rewards the deserving people and accomplishes the business goals.

[00:12:13] And if anybody is suffering, you’ve made sure that that’s people that aren’t the highest performers. So if anybody does leave, it’s okay. And when I think about specific reorgs I’ve done, they’re very much about retention. If you’re making a change, you’re very much thinking, people don’t like change. Who is it I absolutely need to keep and who is it? If they leave, I’ll either be okay or even maybe happy.

[00:12:45] And I can’t immediately think of a place where I personally put someone in a role where I’m like, and by putting them there, they’ll quit. But I can absolutely think of the reverse where people I really wanted to keep, maybe who I felt I was at risk at losing, I was using the reorg to give them something juicier.

Ryan:

[00:13:06] So then how does it usually get started? I mean, a reorg, let’s say. Yeah. What’s the triggering condition?

Ethan:

[00:13:14] Well, usually the triggering condition is a change in the business. So something about the business has changed. And the organization is not set up to address it, so it needs to take on a new market, or you’ve acquired a company and you need to blend them in or you want to start a new project. That’s the first reason for a reorg’s change in the business. The second reason for a reorg is somebody else quit or got moved.

[00:13:41] So often reorgs come due to cascades. And many times, for example, several times in my career, my boss either left or was moved to another job, and so I had to go somewhere else. I could go up if they thought I was ready, I could go under somebody different, or they could choose to backfill, you know, just leave that position open. But at least in my personal experience at Amazon, I never really saw the backfill happen.

[00:14:14] I always was moved under another leader, and several times I’m pretty sure I was put under them to stretch them. When I ran the prime video team, my leader left and I was reorgang from being under an engineering VP to being placed under a business leader. And that business leader, who turned out to be one of my best managers ever, some of his first words to me were, I don’t really want your team.

[00:14:47] I don’t want to run engineering. I don’t know anything about it. And of course, as an engineer, I’m like, oh, but he was a good leader. And he said, we will figure it out. And we worked together and actually had a great run. But I had that happen a couple times where I was put under people, and it was, they had never led engineering. And I was like, well, let’s give him Ethan’s team and let him figure out engineering, which could be really demoralizing.

[00:15:14] Right. Like, okay, here’s the person who’s not technical and I’m their guinea pig in conducting a reorg.

Ryan:

[00:15:21] You mentioned that you’re the storyteller and you’re building a narrative. And you said in some cases you’d be direct and you’d say, we’re trying to retain this person, so we’re going to move things around. And in other cases, you said you, you would tell a story that this is solving a problem for the business and that that other part would be hidden. It’s really interesting to me because there’s.

[00:15:48] It’s almost like this hidden chest going on.

Ethan:

[00:15:50] Absolutely.

Ryan:

[00:15:50] You’re aware of there’s multiple things and you’re aware of the messenger of this is what this person wants to hear. And here are the things that I’m trying to achieve. What pieces of what I’M trying to do. Can I sell to them to make the reorg happen? And also you’ll get maybe other things as a result. And I think that turns off a lot of people, especially engineers. They go, I don’t want to be managing all these things.

[00:16:18] I just want to say it as it is. Is that just a inevitable thing that you have to deal with if you’re in management?

Ethan:

[00:16:26] Well, inevitable, you can certainly work around it. Meaning you can just do a reorg purely based on business goals and not let any concern for people enter. But probably that’s going to cost you people because some people are more ambitious than others or just more demanding. And there is some truth to the old saying, the squeaky wheel gets the grease. This is another harsh truth people don’t like.

[00:16:57] Sometimes, let’s say two people perform roughly equally, but one of them has made really clear that they’re thinking about leaving. Well, a lot of managers will ask themselves, you know, Ryan’s this great guy, always sticks with us. He’ll wait another six months. And I can use those six months to save this other guy who’s also valuable. And by being quiet and dutiful, sometimes I can make the decision.

[00:17:30] It’s not that I mean to screw you. It’s more I’m focused on saving this other person. And I know you’ll put up with it, and so you can end up behind just because you’re such a nice guy. And I often tell people when I’m coaching them, I will often tell my clients, you are a very nice person. And when I tell them that, what they don’t know is I’m then going to say, and that is not a compliment in this context.

[00:17:58] Right? Again, I don’t think you need to be a jerk. But jerks up to a point, pushy people get more. Now they can push too far, and then they get less because they’ve antagonized. But if you never ask for anything, you’ll never get anything. And so in a reorg, you ask about what’s inevitable. Well, what’s inevitable is I have some people who have been more vocal about what they want in their career than others.

[00:18:30] Perhaps I’m arranging the reorg just because I know what some people want and I don’t know what others want. And so I assume they’ll be fine. That’s the most benign version. It may be even more that for some people I know they’re very pushy and demanding, and for some people I know they’re not. And so I’m Serving the pushy people because then I get to keep my whole team.

Ryan:

[00:18:55] So it seems like almost always it’s better to be pushy and forthcoming with what you want in your career. But there is an upper bound though where you go too far. And so to, to empire building, for instance, a concrete example. I mean, have you ever had someone who is a manager underneath you and clearly asking for reports and you could, you could see immediately that you don’t need these reports you’re pushing because this is going to lead to your eventual promotion?

[00:19:24] And how do you handle that as a manager or as a senior leader when you can see someone’s being too pushy and it’s obviously not good for the business and not defendable?

Ethan:

[00:19:34] Yeah. So I think a manager, if I’m in the leadership seat first, I better have a good trusting relationship with that person if possible, where we can have a heart to heart and I can get in a room where they don’t feel anybody’s listening or watching and say, look, let’s talk for a minute why you really want these people. Okay, let’s both be honest that you’re empire building and I get that you’re trying to get to a goal, but this is a bridge too far.

[00:20:00] Let’s talk about how I’m going to get you there or how we’re going to get you there by some defendable path or some other path. And so as opposed to tell them just no, you can’t have this, which will be very frustrating and they’ll probably argue, I try to bend the conversation to let’s problem solve together. What’s another way to do this? And I won’t say I’m always successful. Some people want what they want right now and they can’t see or they have zero patience to wait.

[00:20:30] And at that point, again, that’s that fine line. If I can, I’ll share a related example that I know people resonate with where one time I wanted to tell my manager that I really expected to be promoted, but I wanted to do it in a diplomatic way that couldn’t get me in trouble and wouldn’t irritate him. Not just couldn’t get me in deep trouble, but like would bring out his support even though I was threatening a little.

[00:20:54] And so what I did is with this manager, I wanted to be a director at Amazon. I was a senior manager. It’s when I had the 22 people I referred to earlier, I went to him and his name was Neil and I said, Neil, I need to understand how important My career is to Amazon because my career is very important to me. And see, that’s a statement no one can argue with. No one can argue and say, well, your career shouldn’t be important to you, or it’s illogical.

[00:21:25] So I said, my career is very important to me. I need to understand how important it is to Amazon, because if it’s not as important to Amazon as it is to me, I need to think about that. And, you know, I need to think about that isn’t a direct threat. But you laughed. Everyone understands that I’m kind of serving notice, but I’m not doing it in a way that is easily easy for him to push back on. See, if I say, well, if it’s not as important to you, making it personal with him, it’s not important to you as it is to me, then I might quit, that’s a threat.

[00:22:03] And all managers are going to be like, hey, you can’t extort me, right? That isn’t how it works. So I came up with these words that clearly conveyed the message, but in a way, you can’t argue it’s important to me. You can’t argue really, that if it’s not as important to Amazon, I didn’t personalize it, that I have to think about that. So I was able to put it on the table. And, you know, he promoted me, right?

[00:22:36] He pushed the promotion through. And I think there’s always a way to say things in that middle ground that makes clear what you want and makes clear that you’re willing to consider your other options, but without being a threatening jerk about it.

Ryan:

[00:22:55] That wording is, it’s, it’s kind of. I mean, because when you say the wording, I feel, I mean, if I was. I feel the effects of extortion a bit, right? But I couldn’t, I could not label it extortion. If I went to someone else and I, I went to my management, I said, hey, Ethan told me this, they would, they would probably also immediately get, oh, there’s a, there’s a threat here. But I don’t think anyone in any room would say, this guy’s being mean with it or we need to get rid of him.

[00:23:30] They’d say, that’s a well done way to present yourself, right?

Ethan:

[00:23:35] That’s bold. He’s clear on what he wants. And that’s, you know, one of the advantages. Do find a place where you fit. Meaning Amazon was a relatively sharp, elbowed, pushy place. It has that reputation today. It was true long ago. You have to tune Your message to. To the workplace that you’re in, that message would be offensive in some workplaces, and it was totally not offensive at Amazon because there were people sort of pounding the table of, I need to move up now.

[00:24:10] And, you know, so. So part of it’s just finding the place where you look like a wonderful, sweet person in the mix.

Ryan:

[00:24:19] You mentioned that there’s this. This dichotomy, I guess there’s these people who are team players, I guess, and then there’s these people who are more pushy with what they want, and they’re, they’re very forthcoming and obviously that has benefits for their career. I’ve heard people tell me that when they want to express to their manager, oh, I want to get promoted, or something like that, they have this feeling that they don’t want to be too pushy because it could irk the manager.

Ethan:

[00:24:50] Correct.

Ryan:

[00:24:51] Imagine you’re a high performer. You’re a high performer, but also you keep demanding things of your manager that could also go wrong depending on the manager. I’m curious your perspective as a senior leader at Amazon when you had these demanding high performers. Is that something that you were. Were you happy to service their requests? Or was this. Did it ever irk you? And you said, I’m going to actually not give you what you want because you’re a little too.

[00:24:55] You’re a high performer, but you’re also too demanding.

Ethan:

[00:25:25] So I want to tell you a couple of things about that. Yes. First, the great thing about the really demanding people is if they really are bothering you to the point where you decide they’re a net negative, if you just tell them no, they will quit because they, they are so driven about getting what they want that if you deny them what they want, your problem will go away. Shortly. They will even leave for a lesser job just because they believe, well, at this place, I’m going to have different results.

[00:25:53] So, yeah, you, you can calculate and say, okay, what? There is some calculus that says what answer gets me the result I want. And if someone’s too pushy and not enough value telling them, no, you can’t have that, they will then go away. The second thing, though, is I tended to try and make a deal. I’m a huge believer in deal making, and I made these deals with my own leadership, too. And the deal essentially was, okay, you want this thing, I will help you get it.

[00:26:23] But you’re a high performer. You’re going to show incredible performance on the things I need, and then I will deliver. And there are managers of Course, who make that deal and don’t follow through. I can say with total pride that I feel I always followed through. There’s actually one exception early in my career I’ve shared where I failed to do that. And I still feel bad about it 20 years later.

[00:26:51] But I make a deal where I say, okay, you want this? Here’s what it’s going to take. I need this and this and this. So as an example, I can give you. My manager gave me a requirement I really didn’t like. He was my vice president, I was a director at the time. And he told me, your team is too expensive. We need to put a bunch of it offshore in India with cheaper labor. You need to open. Well, he didn’t even say India.

[00:27:17] He said, you need to open a low cost offshore dev center. And of course I didn’t want to do any of that. Like, that meant traveling around the world and having teams around the world. And I didn’t know how to do it and where to do it. He left it completely. He made the whole problem mine. He didn’t say, go to India, he said, go somewhere. And so the long and short of this is I found a senior manager I knew who wanted to get to director.

[00:27:39] He was Indian, living in the us And I went and said, hey, would you go back to India and start this dev center? If you can build me a dev center in India, I think we can use that as a platform to get you to director. And that’s exactly what happened. He moved home to India. He got to take his son home to learn Hindi and see his grandparents. He moved to India for three years. And during that I was then able to use his independent action there.

[00:28:08] And the fact that he was doing all this stuff remotely and to the point of narrative, I was able to create a story about how this was clearly a director level accomplishment and he was promoted to director. But I made that deal years before it happened. You go to India, I will get you to director. So that’s what I try and do with the demanding people.

Ryan:

[00:28:30] Well, I can imagine also a lot of things could change over the course of a few years. Maybe you’re at a different org and you had that guy went to India and then you’re in and he’s no longer reporting to you two years later. And you can’t deliver on that. But I mean, you can’t make decisions based off that.

Ethan:

[00:28:48] No. If I had been reorged or fired, you know, it depends, right? Like there are things that could have happened where I would have had to say to him, look, I’ll do what I can to support you from the new role. But it isn’t I didn’t do what I was saying, I. It’s that I no longer have the power to and I could feel all right about that. I can’t immediately think of a circumstance where that happened to me where I wasn’t able to follow through for someone.

[00:29:17] But I can see where that would happen. The good news is people are, most leaders are professionals in that they under, they can see the difference between you didn’t come through and you know, shit happened. Those are two different categories.

Ryan:

[00:29:37] Yeah, it’s interesting how much back of the envelope I guess deals there can be between managers and reports or I’m sure between managers as well. Like maybe let’s say you’re trying to get someone promoted in your org and you’re working with your manager on making that happen eventually. And he says, he or she says okay, I have someone who’s even more urgent now but that one’s going to happen in two halves.

Ethan:

[00:30:13] there’s absolutely a line meaning by a line I mean a line up a queue where even HR has a policy in big companies to say what does the forward looking slate is what we call it, the forward looking slate of promotions. And they would track and say by halves who’s in 6 months, 12 months, 18 months, maybe as far as 24. And you could list different names and so there was a sense of like whose turn is it we’re going to do?

[00:30:43] Not that there was in my time a quota. I understand for example at Google today talking to someone I coach there, there’s a strict quota like this quarter or this half two people can get from LA to L9 in this org. And you know the guy I was coaching is like I think I’m number four, I’m not going to make it. And again that can be toxic a little bit. But people understand the idea of there’s a line.

[00:31:14] And so absolutely you have those discussions now again for early in your career there’s usually no quota to go from like entry level, college engineer, whatever that’s called where you’re at to sort of regular, you know, engineer, engineer maybe even to senior engineer. Things get to have more of this quota system when the positions are rare and expensive. And there is a question how many staff or principal engineers does this company Need?

[00:31:45] How many distinguished engineers does it need? By the way, those positions cost a million dollars a piece, so we don’t just want to hand them out like popcorn.

00:31:54 — Stealing scope

Ryan:

[00:31:54] So you mentioned when we were talking about empire building, there’s this notion that managers career progression is tied to the number of their recursive reports. And so on the one hand there’s an incentive for managers to say I want more new headcount. But I could also see there being cases where managers want existing headcount. So maybe someone underneath you is vying to take reports from someone else and basically steal scope for their own promotion.

[00:32:28] And as since you were such a senior leader at Amazon, I’m curious, how did you handle cases where you noticed somewhere in your org, someone is trying to steal Scope from someone else in your org?

Ethan:

[00:32:40] So first I would say this has become more common as growth is slowed. In other words, when a company is growing very quickly, as Amazon was through much of my tenure there, you see less of this because it’s easier to go get new people than it is to fight with someone and it causes less bad blood. Not that I never saw that. But now that companies aren’t growing, you know, if you look at the Magnificent Seven or however you look at them, most of them are flat headcount, couple exceptions.

[00:33:13] There’s a lot more of this, what I would call cannibalism. Since I can’t go out and get new, I’m going to have to eat what’s here. And that, that’s very cannibalistic. You see a lot of maneuvering for that. How did I deal with it? You can’t allow people on your team to go rogue and be pirates in your own organization. Like capture somebody else’s ship and you know, kill the captain, sail away because of what it’s going to do to your org, you’re no longer in charge then they’re in charge, right?

[00:33:45] They’re raiding and pillaging. And you seem clueless, so you can’t allow that. But what you then have to do is either find them a path, convince them to wait, or frankly, if they turn to septic, they become your answer, meaning they’re going to leave, or you get them to leave and you give their team away. So if it really is cannibalistic, what happens is if they don’t pull off, you know, it’s, it’s like any coup attempt, I guess, in, in like politics, if you attempt a coup and you win, you get to be president or, you know, whatever.

[00:34:19] And if you fail, you get hung or shot. And so this is a little bit, you know, it’s not quite that bloody. But one thing that holds back the worst behavior is people do realize if I get caught being too obviously self serving, it’s going to burn me here. And so they usually are pretty careful. But do they want it? Sure. People all the time are asking, could I take over this, could I take over that?

[00:34:48] What I encourage people to do is I tell them, look, you never know when a reorg’s coming. You never know when somebody’s going to quit or get fired or laid off. So talk to your manager and say, hey, if we have an org change and there’s an opening, I just want you to know I’m ready to step up, I’m interested and I will stand by you. This is one thing people want to know in reorg is reorg, shake the ship.

[00:35:15] And so some people are like, oh, this company’s going down or it’s not stable, I’m going to quit if I can. If you tell your manager what they don’t want to do is put someone in a role who’s then going to leave. And so you tell them, look, I’m actually looking to stay. So when you need a stable leader, think of me. And then when that opening comes up, they’re like, oh, right, Ryan said he’d stay. Perfect Ryan.

[00:35:39] Ryan just became most qualified, not because he’s best, but because he’s most reliable.

Ryan:

[00:35:44] Right, I see.

Ethan:

[00:35:45] And that is best in. In my situation.

Ryan:

[00:35:48] So that’s what. And what you just recommended there to me almost sounds like a better narrative for asking for scope. So of course, yeah. I mean, rather than saying, give me, give me this, you can say, I want what’s best for the org and if I need to step up, I’m here, but without being the pirate. And you might actually get what you want.

Ethan:

[00:36:14] Absolutely. And that other piece of the messaging you stumbled on is really important, which is, of course we know you want this so that your org is bigger and so that you have career growth. But there’s a polite fiction that can be true, which is I also want what is best for the org. In other words, I understand that my best path up is to do things that are valuable to you and to the organization.

[00:36:41] And if I tell you, look, when a reorg happens, doesn’t matter what the team is, if you have a need, I’ll be there for you. That sounds very selfless and org centric, even though the underlying message could be phrased as I don’t really care which team’s available. I just want more scope. I’ll take whatever you have. But the way I phrased it sounds so much better because one sounds like the pirate ship and the other one sounds like the selfless servant.

[00:37:09] But they’re both really, hey, whatever’s available, I’ll take it.

Ryan:

[00:37:13] That phrase you said, the polite fiction, I feel like that’s the thing that I could feel as while we’re talking about it too. It’s like if it feels weird, it feels uncomfortable, it feels conniving almost, because it’s, it’s. It’s. You have. You secretly. You have this, and also everyone else knows you have this, but you say this, but you. I mean, both can be true.

Ethan:

[00:37:38] That’s, That’s. Yeah, that’s the key is it can be conniving, but it doesn’t have to be. They can be honest. I can go to my manager and say, look, I want to grow, and I will do whatever you need in order to grow. Like, whatever’s available, I will take it and do my utter best on it. And that can be both things. One story. When I was a director working my way towards vp, I once again had been put under a VP who had no other engineering leaders.

[00:38:11] But then they kept giving him more engineering stuff, and I was the first, and I built trust with him. And he just automatically came to me with everything, like, well, they handed me this. Can you take it? And at one point, I was managing, I think, eight different functions, and they were as diverse all over the place. I’ll give you three examples. I led the technical team for reverse logistics, meaning when people traded in video game cartridges, we had to grade them and sell them used.

[00:38:43] I was an advisor to the team that was grading returned video games or used video games. I was selling new video games, like software downloads, video game downloads. And then I was starting to build Amazon’s app store. That’s three examples. A fourth example is I also owned our B2B business where, like, Amazon was trying to displace and go into the market of Office Depot and, like, ship pallets of paper to people.

[00:39:15] And these all had ended up under my boss. I had this, you know, you call it whatever you want, but this random mixture of stuff, but my boss loved it. And over time, that totaled 200 people from all these different functions. And that just came from saying yes to everything. Now, did I really want to work on grading video games or shipping pallets of paper to schools? Not really. So it was the win win, though.

Ryan:

[00:39:43] Well, this reminds me of the offshore India story that you had, which is your manager has work he doesn’t want to do and so he will pass it to you because he’s identified you as the high performer that’ll solve his problem. So it is win win for you because you wanted the larger org and he does not want to deal with these problems.

Ethan:

[00:40:05] Exactly. And so someone I really respect, he’s now a senior vice president at Walmart. He, he said he was taught in turn by a senior vice president at Amazon. The simple way to get promoted is quote, solve problems for your boss. Like if you just need to get it down to five words, solve problems for your boss. If you’re solving their problems, they will value you. And if they value you, if you then express what you would like, they don’t want to lose you.

[00:40:37] Because not everyone’s helping them that way. Most people are not. They’re just punching the clock or fighting to do their own things. And so even a so called bad boss will often take care of someone who’s helping them. Because even if their calculus is, well, imagine I’m evil and I don’t really care about anybody but myself. But I’ve got this team of people and some of them are helping me and some of them aren’t.

[00:40:59] Who do I want to keep? Even if I’m totally selfish, I want to keep the ones that are helping me. So even then it can work out.

Ryan:

[00:41:07] What about the case where, because in engineering there are cases where we can create scope where someone goes off, they find a problem that we didn’t know existed and they build a new system that changes the industry or does something really impressive and then the impact they had is just incredible. But they didn’t necessarily solve a problem that that boss specifically came to them with.

Ethan:

[00:41:35] Sure,

Ryan:

[00:41:37] that could also work though as well and that manager would push that promo through. But I guess that’s a different path

Ethan:

[00:41:46] and I recommend both of these. What I mean is ask your manager what you can do, how you can help them, but also be thinking about what can you do on your own? Or what can you go to your manager and say, I think we should do this and here’s why. So an example I’ll give Amazon now sells over $1 billion a year of funny T shirts, custom printed. You know, you can put your slogan on a T shirt and put it up on Amazon and buy it yourself or get other people to do it.

[00:42:15] I help start that business and specifically I funded the initial team on it and, and my manager was saying, hey, you run Amazon’s app store. Like, what does that have to do with T shirt printing? This has nothing to do with it. Why are you doing this? And I said, well, I think there’s a big business here, but I need 10 people, and I have 800. So if you’re gonna tell me that I can’t spend basically 1% of my resources on this thing I believe in, we have another discussion to have because you’re micromanaging me like, this should be within.

[00:42:49] And so he’s like, fine, I disagree, but go do. You’re right, it’s only a few people. Go do your thing. And then it turned into a big business. And of course, I’m simplifying a ton when I say, and then it turned into a big business. There were, like, years of work in that phrase, but I was thinking about, what else can I do? What else makes sense that either with my own resources or with lightweight approval, I can go work on?

[00:43:20] And so I think with your manager, of course, you’re trying to ask, what do they need? And that’s what I was doing with 790 out of my 800 team. At 10, I was doing something that he didn’t think was a good idea. But I was able to say, like, well, let me have my 10. He’s like, all right, that, you know, you’re a big enough leader if, you know, you’re now responsible for making these people useful. So I did, and if I had failed, I started projects that also didn’t work out.

[00:43:50] You’ve got to shut them down quickly and, you know, have a good enough hit rate. But I think there’s multiple paths to the top. I tried them. All right, be. Be the helpful person, but also be the inventor.

00:44:03 — Managing out via reorgs

Ryan:

[00:44:03] We talked about reorgs earlier, and you mentioned that it can be used to manage certain leaders out.

Ethan:

[00:44:10] Sure.

Ryan:

[00:44:11] What is that? What does that look like in practice? Like, you give me example.

Ethan:

[00:44:15] Well, what it looks like in practice is you put them on something they don’t want to work on and you claim, well, that was the only seat. That was the seat we needed you in. I mean, it’s. I’d like to claim it’s more complicated than that, but I’ll tell a story of someone who got reorged and I thought they’d quit so much. The manager even thought they were going to quit and was very upset with this.

[00:44:39] When I first joined Amazon, we had a few groups and we organized ourselves, and I owned what was going to be Prime Video, and someone else owned what became Prime Music. And a third person owned what became the Kindle, the E Reader. And then there was a guy who had a clear ownership over the ingestion pipeline. So the way to digitize movies, music, et cetera was all a unified pipeline. And then there was one more leader, and he got given database maintenance, quality assurance, and like, a couple small things.

[00:45:13] And even he joked, well, my group’s other, it’s the garbage can, right? And our VP was like, don’t call it that. You cannot call it that. What about the people in it? Like. And I’m sure he was thinking like, the leader who stuck with other isn’t going to stick around. Well, as far as I know, that guy is still at Amazon 20 years later because he was one of those very calm, nice guys who is willing to do what was needed.

[00:45:40] He was the team player who was going to do what was needed. But the point is, if I wanted to get rid of a high performer, not a high performer, why would I want to get rid of a high performer? But if I wanted to get rid of a pushy, overly ambitious person, just give them other, like, six weeks later, that’s a vacancy. I can open the role right now, right? Because, like, I’m a bit of a confrontational person.

[00:46:04] So when I needed to fire someone, I tended to just go straight at it. But this putting someone in a seat they don’t like is classic. And there’s even a. This is a widely understood. You know, this isn’t an Amazon thing. If you go to Japanese culture, completely other country, there’s this concept called a window seat. I don’t know if you ever heard of this. So in Japanese culture, which is very collectivist and I’m not an expert, so maybe someone from Japan will say this isn’t true, but I don’t think so.

[00:46:37] You want to be in the center of the building. Even though window seats are very desirable in American culture, the hot, you want to be in the middle of things there. You want to be close to the CEO and close to the action. So actually being on a window means you’re not that important. And so ironically, what we would think of as great, being given a corner office in Japan is a signal of you’ve been moved as far from what’s happening as possible.

[00:47:03] You’ve been, you’ve been exiled to the land of irrelevance, where you know you’re not. You don’t even have coworkers on one side, you have glass. And that would be a way to signal to somebody like your career here. Is now over. You’re in a window seat. So this idea isn’t. You know, I don’t think it’s. Some people would think, oh well that’s only Amazon because they have a history of being combative.

[00:47:27] I don’t think that’s true. I think this idea is well known.

Ryan:

[00:47:30] You mentioned that one guy who gets other or the garbage can, he’s the nicest guy or he’s. He’s the team player and this and that. And I, I guess I see the incentive structure that makes it that nice guy gets garbage can. But why? I don’t know. Some part of me is wishing the world wasn’t that way. Sure.

Ethan:

[00:48:06] pressure you can to some degree. First note that the org I described where I had all those random things was also the garbage can. So I built the garbage can, kind of opting into it like I will take all of your junk. And that is my role. He was not doing that in the same way, but there was some of that. The problem with only rewarding the nice guys is number one, sometimes the nice folks don’t actually want more.

[00:48:38] In other words, ambition tends to come out and if people are really ambitious, they do usually end up. Not always, but they usually end up more vocal. Second though, I have the problem that I do have all these vocal people and what they’re going to do usually is not become less vocal just because I have an example of play nice and you’ll move up. They’re going to look for where can I go that’s going to reward me right away.

[00:49:09] And so of course maybe a better leader than I can build this culture where all the nice people are rewarded and all the sort of less nice people see that, see the error of their ways and become naturally generous. But I’ve just. That isn’t human nature. And so I think there is some balancing. What I often do, as you heard from my coaching, and I’ll be honest, is I tell the nice folks being that nice isn’t helping you.

[00:49:37] You don’t have to be a jerk. You have to do what I did. Where you say my career is very important to me. It’s okay, it’s okay to have. Where people fall down is they come particularly in tech from non us cultures that are collectivist or have different mores and where asking for what you want or Speaking up culturally doesn’t fit. I see this a lot, particularly with women from Asia, different parts of Asia.

[00:50:13] At home, they’re expected culturally to take one role, which is often more family oriented and supporting of parents and grandparents and so on. But then to get ahead and work, they need to be a little bit more outspoken. And it’s a big contrast. I would just say it’s okay to ask for what you want that isn’t wrong.

Ryan:

[00:50:38] And

Ethan:

[00:50:40] managers, even good managers, they’re not in the business of guessing. If you’ve never told me what you want, sure, in theory I should ask you, but I’m very busy. And so sometimes I get around to that and sometimes I don’t. Meanwhile, this other person every three days is knocking on my door. I’m super clear on what they want.

00:51:01 — Bad managers and mutiny

Ryan:

[00:51:01] When it comes to engineers, there’s one layer to think of. And so if someone’s low performing, there may be your peer, but if you have a low performing manager, there’s this vertical stacking that we have. You might be reporting to someone who you think is not great at their job. And so the natural thought is go to their manager, which would be your skit, which I guess you could consider this as mutiny.

[00:51:26] If you went to them and you said, hey, my manager is bad. I don’t want to report to this person. Do you have any examples of this kind of escalation coming to you where someone in your recursive chain was considered incompetent by their reports and they came directly to you and they said, please manage this person out, or I just don’t want to report to this person all the time.

Ethan:

[00:51:50] All the time. One of the hardest things for people to understand is I’ve identified a legitimate weakness in my boss. I go to my skip, why doesn’t he do something? Well, if you come to me with a weakness in one of my employees, there is subconsciously this process that goes on that says I have two choices. I can believe that you’re overly sensitive and high maintenance, in which case I don’t really have a problem, you’re the problem.

[00:52:21] And you know, you’re two levels down from me. So if you quit, the manager has to do the backfill. And I can tell the manager, you know, Ryan was here, he said this, that and the other. Maybe you can work with him and that’s exactly what you don’t want, is me ratting you out. But I can make it my manager’s problem. On the other hand, if I agree with you and I’m like, You know what? This manager I have really isn’t that good.

[00:52:44] Now I have three problems. This is really bad for me. One, I have to decide what to do with my manager. Maybe I have to manage them out. Two, if I do manage them out, I have to hire and train somebody else. And three, while they’re gone, I have to do all their work myself. So you can see why, even if it’s subconscious, I have a lot of reasons not to listen or not to believe very easily. So mutiny.

[00:53:09] Let’s talk about mutiny. If you want to see it as a mutiny. We’ve talked about pirates already. Never mutiny alone. If you want that to work, if you really want to say this manager’s bad and we have a problem, take two or three people or send two or three people in sequence. Find some of your co workers who are willing to speak up. Because I will give you an example. It really bothers me. I had a leader who was treating the women on his team like crap.

[00:53:42] Now, he wasn’t doing that in front of me, but it turns out it was true. Well, it took a long time for the different rumors to bubble up to me enough to where I’m like, I’m hearing bad things. And then I went and talked to several of the women and like, so in working with this person, I’m hearing and they’re like, oh yeah. Well then I was like, oh, okay. I’ve been blind to a problem, I need to act on it.

[00:54:15] Had any one of them come to me early and said, this person’s a problem. They’re doing this, that, and the other I probably wouldn’t have listened to because it was a lone report. But when several of those reports came up, then I’m like, oh shit, I’ve been missing something. So if you want to hold a mutiny, go together. Right? Even if it’s Zelda, it’s dangerous to go alone. Take this, take other people.

Ryan:

[00:54:41] I see. Okay, so if, let’s say there’s a high performing IC who is reporting to a manager and say, the fact is that that manager has some shortcomings in their abilities, you would say, don’t go immediately to the skip first. Build allies with some of the other people and say, hey, are you also seeing this with this person?

Ethan:

[00:55:01] And if you can, yeah, you want to talk to them and say, hey, do you see what I see? Well, one thing is that’s a way to sanity check. I’m really struggling with our boss in this area. Do you, Are you having that problem? And if they say no, I get along with them great, and here’s how, then maybe you can learn and adjust. That’s the style thing. Whereas if they’re like, oh, yeah, then the hardest next hurdle is are they willing to speak up?

[00:55:27] Because a lot of people aren’t. But if they know that you’re going to go first and play ringleader, like, I will go talk to our skip, but I’m going to tell our skip they can verify my story by talking to you or, and also to Sally or Fred or Rajeev. Most managers, if you go to them and say, look, I know that my saying the boss is a weak link is uncomfortable and difficult for you, but these three people are all willing to share similar stories, would you consider at least talking to them?

[00:56:05] Most managers will do that. Most, you know, not all. There’s no guarantees when you’re dealing with people.

Ryan:

[00:56:12] I mean, that’s. Yeah, that’s the immediate thing I imagine is, let’s say I was that IC and then I went to other people and we all kind of knew that that person had a shortcoming. It. There’s the difference between that, that water cooler talk and saying, yeah, this person is disorganized. Yeah, he keeps doing that. Compared to, hey, I’m going to go to his boss now and we’re going to, we’re going to make, take this up a notch because also there’s this, this delicate balance which is all these people still report to that person.

[00:56:43] So now if you were to get caught at any point in this process, you might worry about retribution or something if you’re wrong. What about the path of. Because this is a lot. I mean, your imagination needs to be pretty bad if you do this. What about the approach of you say, hey, it’s just me and him, and you want to go to the skip and say, hey, Ethan, you know, nothing, nothing personal with the person that I’m reporting to, but I see a better business case for me to go to another team or, you know, something like that where you word and say, I don’t, don’t think poorly of this manager, but I’ll be more productive somewhere else.

Ethan:

[00:57:30] approaches, the first of which is, remember we talked about the story you tell versus the story that may be true if you can actually make a strong case that you’re equally valuable or more valuable somewhere else, a plausible case. Don’t even bring up the manager. Just say, hey, I was looking at this other role and I think I could do so much more for you in the org over here because of A, B and C.

[00:57:58] And you try that run first, and if that works, you’re done. You never have to bring it up. The second thing you can say, you’ve just. You’ve gotta. You’ve gotta play chess, not checkers, meaning several moves in advance. You can totally say, look, candidly, you keep it blame free. I’m not compatible with our leader. I’m not. Maybe it’s him, maybe it’s me. Of course, I naturally feel that maybe I’m doing good work and he could improve.

[00:58:30] But I’m not here to throw your manager under the bus. I’m just here to tell you I need a change. And I did this once with a leader at Amazon, where I wanted to make a change. I was doing a good job for him in the role I was in. So he wanted me to stay there. And what I had to help him understand is I said, you are seeing this as I take this new role or I stay in this role, and you’d prefer I stay in this role because that’s good for you.

[00:59:01] And I get that. But what I’m trying to tell you is either I take this new role in your org that I’m asking for, or I’m going somewhere else entirely. So there is no choice. Keep me where I’m at. There’s, do you want me on your team or on someone else’s team? Now, I had both options available, and it was really interesting because after weeks and weeks of resistance, I was in the new role the next morning.

[00:59:25] Like, once he got the clarity of, oh, I thought I had a choice between you stay in the role I want you in and you go to the role I don’t really want you in. But the choice I actually have is I keep you in my team or I lose you. It was totally different the next morning.

Ryan:

[00:59:44] But how did you word that in a way where that person you’re reporting to did not feel upset that you kind of subverted their power and forced their hand?

Ethan:

[00:59:55] That’s a good question. And so I can exemplify that. I framed it in terms of what they would do. This person was a vice president at that point. I was a vice president reporting to them. So I was like a smaller vice president, whatever, lower in level. And I had gotten a job offer from a senior vice president in a different organization. That was my other job. So I went to him and I said, I have an offer from this svp.

[01:00:26] Now, you know that if I give my Word to that guy. I have to go. Like, that’s what you would do? If I give my word, it’s over. You know, I really want this other role. I have to give this guy an answer. You’ve told me no on this other role several times. What do you want me to do? Bam. The door. Like, because once he understood, he understood he would never break his word to an svp. So he understood that the moment I gave my word, it was done.

[01:00:58] And so that’s the same. Like, what would you do if you can tap into the manager feeling like, okay, I don’t like it, but I get it. That’s what I was doing is he was like, oh, I don’t like this, but okay, I get it.

Ryan:

[01:01:11] There were three roles under consideration.

Ethan:

[01:01:14] Well, there was the one I was in and really wanted and the alternative I had gotten on the table.

Ryan:

[01:01:19] So you brought in a new option that was better than your current. And you said, my alternative now is I may leave you unless you give me what I want. But you didn’t say that directly.

Ethan:

[01:01:29] I didn’t say that what I want. Instead, I said, I actually, I’m very careful not to make it that threatening. I said, you know, once I give my word, I’m. I can’t go back on that. Like, that’s the.

01:01:43 — Political messaging

Ryan:

[01:01:43] The. The wording. I mean, as soon as you say, I think. I mean, if I was that manager, I. There’s. I’m on your side. Even though I see the. What was the. What was the word? The polite fiction, I guess I see the polite fiction immediately, but I laugh because it’s just so.

Ethan:

[01:02:03] And I think this is a skill I haven’t figured out how to teach people. But maybe this talk can help. Several times I’ve given you these polite fictions, these careful wordings. I think a mistake people make that they can do so much better is think through your wording before you’re in the room. Because most of these conversations are not spur of the moment. You either. They’re either on the calendar, this is my performance review, or whatever, or you at least know they’re gonna come up.

[01:02:36] And so spend the time to have that key phrase ready so that you can just, with true sincerity say, yeah, you know, once I give my word, I can’t go back on that, and I deliver that. I look like I did mean it. In no way was I lying, but at the same time, I deliver it like I’m an angel just fallen from heaven. You know, poor, innocent victim. Both are true. And when I put the screws to that other boss long ago and said, well, I’m going to have to consider if my career is not as important to Amazon as is to me, I’m going to have to think about that again.

[01:03:19] I said that with a warm smile, like, hey, of course I’m gonna have to think about that. And I just look like the kitten, you know, the anime kitten with the giant eyes. Well, body language matters, because if I say that to you, see, I can say that differently. I can say, well, Ryan, you know, if my career is not as important to Amazon as is to me, I’m gonna have to think about that. Then you’re like, well, screw you.

[01:03:45] Whereas if I say, well, of course I’m going to have to think about what that means for me. Totally different delivery, same line, same words. Now, there’s some people watching this, engineers, who are like, I hate you. I hate the politics. I hear you. But this is human nature and human interaction. You’re dealing with other humans and their emotions are at play and their defensiveness and their own stress.

[01:04:09] Do yourself a favor and get good at this.

Ryan:

[01:04:12] Well, yeah, and to me, that, that is clearly your superpower. I mean, I can imagine so many other people in these situations not getting the outcomes they want because they word it slightly differently and it doesn’t come off, you know, the, the way they say it and, and kind of the, the, the interplay about what that person wants and the things that you could say. And how do you do that? Like, if we were to try to reverse engineer coming up with those polite fictions so that people can get more of what they want out of their careers, I imagine this would be useful even outside of your career.

Ethan:

[01:04:49] Oh, it’s all through your life.

Ryan:

[01:04:50] Yeah, all throughout your life.

Ethan:

[01:04:51] It’s all. Any negotiation, any deal making, I call it. I’ve tried to reverse engineer, and we’ll do some more right here, right now. I call it being interpersonally warm. So being friendly, smiling, not getting agitated while being professionally firm, saying what I need to about what I really want, but in a way where I don’t bring anger or you versus me into it as much as I can. And so there are some key phrases.

[01:05:33] A great phrase is help, right? How can you help me? I want to get to the next level. How can you help me? Because everyone loves to think of themselves as helpful. And so if I ask you for help, that’s just as opposed to I ask, what are you going to do for me? It’s that sort of phrasing. So Part of it is the attitude. I read a book I really love. I can share the book. It’s called Leadership and Self Deception, and it’s written by some research institute called the Arbinger Institute.

[01:06:12] But what makes this book so interesting, what it taught me is don’t see the other person as an obstacle to what you want. See them as a human being trying to do the best they can in their situation, who has all kinds of problems. And what makes a huge difference here is when I start seeing my boss, who I want something from, not as well. This person controls my pay or my promotion, and I have to yank it out of them.

[01:06:37] That’s seeing them as a thing. They’re an obstacle, like a barrier across the road. If I see them as a human trying to get by, then I can emote to them and say, look, here’s what I need. What do you need so we can make this work? I want to help you because I want your help, and that’s an honest offer most people can understand, and so it comes from that. How do I see them? I think there’s a huge thing is we objectify others.

[01:07:09] This happens anytime you’re working with another team. As an engineer, you did this all the time. I just need them to do this thing. How can I. Your thought is, how can I get them to do this thing that’s making them an object? I need to give you the right inputs and yank on the right levers so that the right thing comes out. I’m treating them like an API. I make the right call, I get the right result.

[01:07:30] What you want to do is treat them like a person and say, look, I know you have a busy roadmap and I understand you have a lot of pressures, but I do need this. So what can we do together? How can I help you so that you can make the time or let me make this change myself? And the key is, once people feel like you’re caring about them as a person, they usually respond that way. Now, nothing works 100% of the time, and I can’t instantly recall one, but I’ve probably rolled out these polite phrases before and not one.

[01:08:08] Like, I was debating politics with my college roommate, and I thought to myself, this is very recent. I’m going to use all I teach all day about how smooth I am and how influential I am. I’m going to use all my skills and I’m going to convince them of this point in modern American politics, which is very contentious. I failed. The best I can say is that we Left the conversation, nobody angry, friendly.

[01:08:39] But I kind of got nowhere because sometimes their motives and what’s important to them just can’t be aligned with yours. So I’m not. My only point here is these techniques will get you way more success. I don’t want to convince anybody that like I have some magic recipe that’s 100% likely to always work out.

Ryan:

[01:09:02] You said a few things and it sounds like you had some high level rules you could, you could give, but it’s hard to reverse engineer because it’s, it’s not a recipe inevitably it’s, it’s two people interacting and every single person’s different and seems like the, the unifying thing is to really imagine what is that person feeling and when you say things to them, what do they want and how will how you present yourself make them feel in the exchange.

[01:09:35] And hopefully that goes in the direction you want, but obviously not always because it’s subjective and you can miscalculate.

Ethan:

[01:09:43] So I think an analogy that may work for at least people who know chess a little bit about chess is I think of this a little bit like I’m not an expert chess player, but I understand there’s two pieces to playing chess. Well, one is of course knowing the board and how to move. The other is called a book opening, where you’ve memorized a set of opening moves. And I think what you can do in a lot of these conversations is open with a set of preplanned moves where you’ve thought about how am I going to make my first ask?

[01:10:20] And you’ve thought about when am I going to do it? Like, don’t catch your manager wanting to have a deep conversation about career five minutes after a high severity problem on a Friday. Like they’re not in the mindset, right? If their hair’s on fire and they’re trying to get out the door to pick up their kids, this is not when you’re going to get the best hearing. So in the play chess analogy, set up your opening moves.

[01:10:48] But then yes, at some point there are some general rules about chess. Again, I’m no expert, but like in chess, you want to hold the middle of the board. Being towards the middle is. So if you have a chance to move towards the middle, move towards the middle. Well, with a manager or any leader that you want something from, try to move towards that common ground where we can agree. But yeah, it’s about learning how to play well.

[01:11:12] And I’ve spent a career learning how to play well in verbal conversation. And I think the biggest thing, I used to have quite a temperature. I actually got fired two separate times, put on layoff lists. Because I was volatile and critical early in my career, I put the temper away and I learned how not to go there. And that made a huge difference because people don’t react well to anger or criticism, even if you’re right.

01:11:48 — Handling politically skilled operators

Ryan:

[01:11:48] So, I mean, obviously you’re very adept at this interpersonal way to influence people. When you were working in corporate America and you met someone else who you identified, had that skill set, let’s say some manager underneath you, or maybe managers in other orgs, or it could be an engineer as well.

Ethan:

[01:12:07] Yep.

Ryan:

[01:12:08] Is that something that, in your mind, that’s a positive thing, or is that something where you’re thinking, I have to be careful because that person can mold things around them strongly Both.

Ethan:

[01:12:19] What you’re trying to do next is assess what’s their motive. People often ask me, what’s the difference between influence and politics? Like, when does influence become politics or manipulation? And for me, the answer is the skill sets are the same. They’re around motive. And so one thing I teach executive presence, and one thing I point out about executive presence is Darth Vader had it in spade.

[01:12:47] Like, he had tons of executive presence and so did Palpatine. Right. They’re these powerful, commanding figures. These skills, being influential, can be used for good or evil. So with the example you’re talking about, the thing I’m now trying to figure out is, okay, this person is a skilled operator. You know, are they U. S. special forces or are they Russian special Forces? Right. Like. Like which side, you know, who am I dealing with here?

[01:13:14] Right. Am I. Am I dealing with someone who’s gotta, like, rescue me and get me out of a hostage situation, or am I dealing with someone who’s across the line and is about to drop a bomb on me? And so I’m trying to figure out motive. And there’s kind of three cases even there. There’s people whose motive is to do good, easy to work with. Great. We both have high social skill. We’re going to be so effective together.

[01:13:41] There’s people who are completely out for themselves now. You know, going back to Star wars, it’s very much like Sith versus Jedi. I’ve got to get my lightsaber up and I’m going to be fighting for my life. And then there’s this middle ground, though, that’s interesting, which is, well, this person has motives I can work with. They’re not altruistic. They’re not just trying to do well. But in this case they kind of want what I want.

[01:14:10] Or I can give them enough wins and I can negotiate. I can be like, hey, you want this, I want that and that’s the deal making of, of we’re never going to be best friends. I don’t necessarily think you’re the best person. But in this limited scope we can both make money together or we can both ship something together.

Ryan:

[01:14:33] Let’s say there’s someone who is extreme high performer but you can tell they, they will only do what will help them and you see a path to aligning what you both want. So it can be good. But you could tell that if this is ever off their path, they will not be on your side. Or in fact if there’s something that could get them ahead and put you behind, they would gladly do it. Would you still choose to ally with that person or would you kind of separate them from you and your org somewhere?

Ethan:

[01:15:07] Well, if I could separate, I’d love to. But remember at work we don’t get to choose always the teams we work with. Like sometimes we need another team. Sometimes we need a team that’s run by a terrible person. Sometimes we need a team that’s run by a self serving person. I think if you want to succeed in your career, most sure get away or marginalize or minimize contact with difficult or unethical people.

[01:15:36] But also learn to work even with the very unethical because sometimes you’re going to have to. Now of course there’s a limit. I’m not saying break laws or do things, you know, but, but sometimes you don’t get to choose who your teammates are or what system you need. And so I do think it’s worth being equipped. How am I going to do the best I can? And there are, there’s, you know, we don’t have time.

[01:16:07] There’s a lot of techniques to protect yourself. There’s bring allies with you. For example, most slippery people, they love to be slippery in the dark where they’re not going to get caught. So if you can bring a few allies so it looks like, oh, everyone thinks this often suddenly they’ll say well I think that too. Because they, you know, part of their power is slippery. People usually can’t afford to have everyone know that they’re unethical because once we know, once everyone knows someone’s unethical, they do get shunned and, and so they have to keep it largely under wraps or defensible.

[01:16:57] What There’s a phrase from politics called plausibly deniable, where they can deny that they were really as bad as maybe you’re saying. What you want to do is put them in a situation where it’s easier to go along with you than to fight you. So construct the situation where they’re like, okay, this isn’t the best for me. Like, I’d rather stab Ethan in the back, but, boy, there’s a lot of witnesses.

[01:17:22] Right? That’s. And so you. And it’s unfortunate, you know, again, I know many engineers will be frustrated. Why do I have to do all this crap? Can’t I just do good work? Why? Why? Sorry. Humanity going back. By the way, read any, you know, ancient text of philosophy or religion. Humans haven’t changed. They’ve been mean to each other since the first ape clubbed the other ape. This doesn’t change.

[01:17:46] So it’s. I’m sorry that it frustrates you, but you’re gonna have to do it.

Ryan:

[01:17:51] So it sounds like the best way to prevent yourself from backstabbing is to have a really strong, I guess, soft power in your org where there are people who will defend you. Or just the setup is that you would be too powerful for them to come after you.

Ethan:

[01:18:08] Not worth it.

Ryan:

[01:18:09] Not worth it, right?

Ethan:

[01:18:10] It’s, it’s, it’s like all the animals in the animal kingdom, you know, like porcupine. All right, plenty of animals can eat a porcupine if they’re willing to get a mouthful of quills, but it’s not worth it. Like, you know, and I’m sure there are many other examples, but you, you want to be not worth it. It’s just easier because remember, this is important other than cartoons, like a Darth Vader. People don’t wake up in the morning.

[01:18:40] That slippery person does not wake up in the morning and say, I’m an evil jerk. And that’s all I ever want to be. Their self narrative is more like, I’m doing what I need to. I’m practical. Other people are wishy washy and have all these optimistic things. I’m practical. And some people get upset that I’m so practical. Well, that’s not my fault. That’s what they’re telling themselves. And so what you want to do is help them with their narrative and say, you know what?

[01:19:11] You’re right, you’re very practical. And in this situation, the practical thing is to go ahead and help me out because I’ve got a lot of support and, you know, I’m a rising star, and it’s, you help me out and I’ll remember. And they’re like, oh, that is very practical. You know, even though. And that’s just. It’s important to remember that the other person’s narrative, we think of them as like, that person is so unethical.

[01:19:42] They must wake up in the morning and think about how they can rob their grandmother. That’s not what they say about themselves. They say, you know, grandma’s getting old and I need to manage her funds because she’s not really up to it anymore and she doesn’t need all that anyway. But that is what they tell themselves, right? It’s not I’m a thief.

01:20:01 — Orgs trying to steal scope

Ryan:

[01:20:01] What about. I mean, I think a lot of what we talked about is kind of implying within your org. Have you ever seen cases where other orgs were plotting to maybe take scope from your org or kind of, you know, battling between the org orgs?

Ethan:

[01:20:18] Oh, geez, I’ve seen this all the time. And again, the narrative people are telling themselves is not usually I’m going to pillage scope from that other org. It’s they know their own org and their own mission really well. So they have a really clear idea why having your scope would help them. Meanwhile, they don’t know your mission and they don’t know everything you’re really doing. And so it’s very easy for them to imagine that your org is not that important.

[01:20:51] A very smart engineering leader once said, respect between two engineering teams is inversely proportional to their distance. So the further apart they are, the more they belittle each other because they can’t see the work. And you hear this all the time. Like I think of Amazon S3, right? This service that stores now hundreds of trillions of objects. I’ve heard it described as, well, it’s just a big disk drive.

[01:21:19] So this is very belittling. Like of course, at some level it is just a big storage system. That’s true. But the boil down, it’s really just a big disk drive is so dismissive. And you see this all the time. And the closer you get to home, the more you’ve seen the work or worked with the people or seen the complexities and all the edge cases. You’re like, oh, actually that is very complicated. It. Well, I see this with orgs, sure there are some leaders who are just like, yeah, we should take that org because I want it.

[01:21:49] But a lot of times what they’re doing is like, well, we can do so much more with it. Again, that internal narrative. Very few humans wake up and say, I’m going to Be evil today. What they do is they want something and they construct a line of logic which we’re very good at about why that’s actually a great thing anyway. And that’s where you get the org versus org fights.

Ryan:

[01:22:12] Let’s say I’m an engineering leader. How do I protect my org from other orgs?

Ethan:

[01:22:18] Well, first thing is you’ve got to get all this is the skunk trick. I guess we’ll use animals here. The skunk trick is make yourself unattractive. So trick number one is list all the things that they don’t know about that you have to maintain that are unsexy and they don’t want and have the customers for those show up and say how important they are. Oh, yeah, well, we actually run, you know, the Yugoslavia, and Yugoslavia is not even a country anymore, but we’ll use that.

[01:22:45] We run the Yugoslavian tax engine. And so, boy, we get a lot of tickets for that. And it’s going to have to be owned. So are you guys ready to take on that? That’s one thing you could do. The other thing is you want to make yourself thorny again. There is a little bit. Who’s the least attractive target? And so they’re out, sort of. Do they really want your org or do they want an org? And so one of the ways to be thorny, honestly, is to overreact and attack back and just make yourself like, oh, ouch.

[01:23:22] They. They went public and volatile really soon. Maybe I’ll go for a softer target. Now, you have to be careful at how you do that because if you, if you behave in a way that makes it easy to discredit you, then you’re an easier target. Oh, see, Ethan’s not a very mature leader. We should definitely move that under me. I’ll make sure we don’t have this sort of unseemly behavior that’s weakening myself.

[01:23:48] But if I can immediately say counterattack and say, you know, it’s really good that you’re thinking about merging our orgs. Your org is going to be so great as a part of our org and I promise I’ll take care of you. Then they’re like, oh, maybe. Maybe I messed with the bull and got the horns so you can. So there’s I, you know, I’m uncomfortable talking about sort of combative tactics because I see when you get to that, it’s a failure, in my opinion.

[01:24:16] But if someone’s trying to maneuver and you don’t want that, it’s a matter of shining a light on it. And sort of exposing their motives if you can, while making a strong case for why the work you do is very important. Now where you’re vulnerable is if your work is behind, if you have weaknesses, if your work is behind, if it’s not making a lot of money, if it’s performing poorly, those are hard cases to make.

[01:24:45] Whereas if you’re on a strong platform of high performance, then you have a lot of arguments hopefully about don’t distract us. Another great one is we know reorg cause a six month hit in productivity, we’re killing it on profitability, new customers, whatever is valuable. We should, you know, we shouldn’t even talk about this for six months. We need to hit these goals and get to year end because you know, in six months they’ll have moved on.

[01:25:16] So one thing, that’s another delaying tactic and I hope what you’re really getting from this answer is, is how quickly I’m coming up with like here’s three or four different things. You just have to look at the situation and say how can I put this off, change it, scare them off, make myself unattractive, whatever I need to do. I’m just looking for what’s again narrative. What’s the best story? People love to think that facts make decisions, but what we know from psychology, people come to an emotional conclusion and then rationalize it.

[01:25:57] If I am a better storyteller, I give you that narrative that you’re like, yeah, that sounds right. That feels emotionally where you wanted to go. And then you have a story. I’ve helped you wrap your own story around it. And then once it’s your story, your internal story of how this should be, it’s very hard for anyone to shake that. So be the better storyteller.

Ryan:

[01:26:20] You mentioned a lot of tactics on defense. What about the flip side? Let’s say you’re high performing manager and you see there’s a great reason for another org nearby to be absorbed into your org. What are the attacker tactics?

Ethan:

[01:26:39] Wow, Ryan, when you said we would just go deep on transparency, you were not kidding. Ah, attacker tactics. So the best attack, which I would argue isn’t even necessarily an attack, but it has that effect, is a real story of how that’s going to create more value. If you can make a point that whatever metric you’re using, this group puts out one release a year, I can get it to two. This group makes $100 million, I can get it to 200 and here’s how then that’s actually a very good argument.

[01:27:21] If we go to Maybe slightly less good arguments. They usually have to do with seniority, efficiency. I can save costs. I can. By consolidating this, we’re either going to get more done with less or, or we’re going to have simplicity in our communication or you’re going to have fewer direct reports, which is going to make your life easier. There are arguments that aren’t necessarily false, but they’re less directly about business growth and they’re more about making things smoother for the leader who’s making the decision, or nicer or cheaper.

[01:28:05] They’re like second order effects. I wouldn’t, I hope no one would do this, but I know people do. The third attack tactic is clearly to go highlight flaws in the other or right to go show, well, this leader has this problem and this team has this problem and they have high turnover and they have high cost and look at this outage. And you’re basically trying to show, well, it’s a dumpster fire and you should let me put that out.

Ryan:

[01:28:35] But what if that’s true, though?

Ethan:

[01:28:37] Okay, if it’s true again, if it’s true, it’s a dumpster fire, then you’re offering to make things better. It’s where you’re kind of highlighting things that maybe aren’t as bad, you know, just like only pointing at the bad. Yeah, if it’s, if it’s true, an org is struggling. The truth is you can talk to your manager and say, look, the way I would do that though, is not say, I should take over them, I’ll fix them.

[01:29:02] I would say, hey, I go to whoever owns it or the common boss, I say, hey, I can’t help but notice that you’re having a lot of trouble here. Is there anything I can do to help? That’s my opening gambit for I want to take it over because I can go in there and start helping. And then later they’re like, wow, it’s so much better under Ethan’s leadership. And at that point I can casually mention, well, you know, if it would really help you, you know, we could just consolidate.

[01:29:33] Now that I’m here, I see all these efficiencies. That’s, that’s like another way to go about the same thing. I feel like giving all these answers, people are going to think like, wow, that Ethan is a real sadistic.

Ryan:

[01:29:47] what actually happens though, I think, like, what you’re saying. So it’s kind of. I don’t.

Ethan:

[01:29:52] I have seen it all. I’ve seen things, you know, that make me want to close my eyes for sure.

Ryan:

[01:29:57] I kind of want to talk a little bit about, like, some of that interpersonal stuff or maybe, you know, the backstabbing, some of the politics. If there is someone within your org that is a people problem, they’re doing something. Actually, I don’t know, you. You can just immediately get them fired because they’re in your recursive reports.

Ethan:

[01:30:21] have the evidence for sure. But yeah, you can. Absolutely. You can move very quickly if it’s clear.

01:30:26 — Handling difficult people from other orgs

Ryan:

[01:30:26] But what if it’s someone in another org? So another org. There’s a person who is well liked within their org that is a major problem for your org. Then what do you do?

Ethan:

[01:30:39] So one of my managers had this problem and he. He literally put someone in a position he called a meat shield, where it’s like, this person is a huge pain for our org or this org. Your job is to keep them off of us. And honestly, it’s going to be a terrible job, but I’ll reward you later. So you need to, like, go be a meat shield till this project’s over. And so the imagery is just. Yeah, no, it’s like it was, you know, like, put your back into it.

[01:31:14] Let them beat you.

Ryan:

[01:31:16] Oh, my God.

Ethan:

[01:31:17] And of course, he had to. He was very honest with them of, like, that’s your role. And I know it’s a terrible role, but I’ll make it up to you. You know, get us through this project. Like, go take one for the team and I’ll pay you back. But what else can you do when someone’s in another org? The first thing, the most important thing, if you want to try and make a change, is understand why they’re valuable to their org.

[01:31:44] Because the reason they’re that other org, what they’re not doing is saying, I know we have this evil person, and as long as they’re pointed at other orgs, we’re okay with them being evil. That’s probably not what they’re saying. What they. They either aren’t seeing it or more likely that person. What they’re telling themselves is, well, they have a little bit of sharp elbows, but they get so much done that’s valuable.

[01:32:09] It’s okay. And if people’s feelings are hurt, well, they’re soft. It’s kind of how they’re. So you have to understand where they’re being seen as valuable. And then what you have to do is go to that leadership and have one of a few Discussions. Look, we’re willing to give you this value without all this pain. Is there a better way? That’s one way you can go about it. The other way you can go about it is, look, your guy is driving us nuts or is mistreating us.

[01:32:43] So that’s going to change. Either you’re going to give us a new interface, you’re going to. I can’t say you’re going to fire them, but you’re going to move them away from bothering us at least, or we’re going to stop cooperating. Right? Like if. If you’re going to inflict pain on us, we’re going to return the pain. Everything will be late, everything will be your fault. Everything will be escalated.

[01:33:07] Everything will require. Six document reviews we’ll put in place. I’ll spend the rest of the day asking Chat GPT, what is the most policy checkpoints I could put on our next release? Right, like you can express this.

Ryan:

[01:33:19] How do you message that with the right narrative without, you know.

Ethan:

[01:33:22] Well, I mean, if. Okay, so that’s. That’s my internal narrative, I guess. That’s my, like, all right, screw with me, I’ll screw with you back.

Ryan:

[01:33:29] Yeah.

Ethan:

[01:33:30] My external narrative would go something like, dear peer of mine, this person is really making things difficult. You have this person on your team. I know they’re a valuable member to you and your team, but unfortunately our team is really struggling with them and I’d like to make a change. Option one, you can find a new interface person for us. Option two is we’re going to have to adjust to distance ourselves for the sake of my team.

[01:34:08] And I think, unfortunately, that option is going to create a lot of friction between our orgs that really is going to take both of our time. And so I don’t want to take your time, but I have to to protect my people. Some of them are ready to leave. What would you do if people on your team were ready to leave? And so I’m gonna have to take some steps and that’s gonna increase bureaucracy and take a lot more of your time and my time.

[01:34:36] And I’m sure there’s a better way.

Ryan:

[01:34:39] That’s amazing because it’s the same thing. But you didn’t, you didn’t bluster or threaten it just by circumstance. If you don’t comply, your life will be harder. But I wish it wasn’t. But that’s just how it is.

Ethan:

[01:34:55] One thing people underestimate is most leaders at a certain level, they may not have expert social skills, but they’re pretty they got there for a reason. They understand what they’re being told, even if they don’t understand it consciously. They’re sensitive to unspoken messages, and they understand that I’m giving them a chance to collaborate, but that there’s a stick up my sleeve that I didn’t come to this with no other plan.

[01:35:34] And part of this is also how I project how you. If you look weak, like you’re begging, I don’t really care what you need. But, you know, I’m presenting it as like, wow, this is. This is really tragic because it’s gonna get hard for both of us. And what I’m conveying there is, look, you wanna trade punches, that’s okay. I play ice hockey, and sometimes someone on ice will hit me or, you know, be a little rough with me, and I’ll just look at them and say, you know, two can do this.

[01:36:09] This was your freebie. You decide. But I’m like, putting him on notice, like, hey, the next time I have two elbows also. Right.

01:36:19 — Handling weak managers

Ryan:

[01:36:19] Well, this conversation makes me think that let’s say you’re an engineer or someone who is reporting some. You want your, Your leader to. To be strong for your org. Because I can imagine if they’re not, let’s say they’re. They’re very nice and their org is getting, I don’t know, a lot of negative things are happening to it. And they just say, oh, that’s okay, you know, we’ll deal with it as it comes.

[01:36:46] And the. Everyone else is going to step on them and that’s going to affect you if your manager can’t stand up for the org.

Ethan:

[01:36:56] Absolutely. And I think, remember your manager. I’ve tried to showcase that your manager doesn’t have to be a mean person to be firm, to be professionally firm, to have backbone. They have to be skilled. A lot of people, this is important. A lot of people don’t know how to be socially skilled. And so they can only say hard things once. They let themselves get angry or upset. That’s their fuel is if I need to give you a hard performance message and say, ryan, you know, you’ve missed three dates in a row.

[01:37:32] I can only do that once. I get internally angry. And like, Ryan’s missed three dates. He let me down. And so now I’m coming at you. I’ve gotten my ability to speak to you and to say the truth because, see, I’m afraid you’re going to say, well, I haven’t. Or it was. My mom was sick or whatever, and you’re going to make it hard for me. So I have to be ready to fight. And some people can’t do that without first allowing themselves to be angry.

[01:37:58] You want a manager, ideally, who can stand their ground because they believe in it and because they are willing to have confrontation. And I’m no master at this, okay? I have plenty of friends will tell me where I’ve been a jerk. But I try to figure out how can I stick to what I believe without ever needing to resort to anger as a fuel. I can just stick to belief. To your point about what you want from managers, there’s a graphic image that says, look, all managers are either an umbrella or a funnel.

[01:38:38] And when the poo comes down, they’re either shielding their team with the umbrella or they’re a funnel dropping it onto their team. And you were discussing the funnel. The one who’s is like, yeah, we’ll deal with it. And what. What are they doing? They’re deciding that if you’re the engineer, they have a choice between push back on the boss above them, which feels scary and dangerous, and push back on you being upset that you’re being asked to do weekend maintenance for the fourth weekend.

[01:39:10] And you’re an easier target because I have rank, I have position, I control your performance review, I control your salary to some degree. So they’re making a choice, which is, it’s easier for me, safer for me. And what you have to do in that trick in that space is change that balance of safety. So you either have to help them feel safe pushing back on their boss or. Or help them feel not safe asking for your fourth weekend.

[01:39:43] Those are. Those are the only two choices, really, is you change the balance of safety.

Ryan:

[01:39:49] Well, that first one almost feels impossible to me. Like if I was. If I was reporting to someone and this manager just says yes to everything, has no backbone. I don’t know how I can give them a backbone, but I can definitely do the second option.

Ethan:

[01:40:07] Well, you give them a backbone exactly by the second option. But like this. You say, let’s say I’m your manager. You say, hey, Ethan,

Ryan:

[01:40:18] this is.

Ethan:

[01:40:19] There seems to be a pattern where we keep getting asked to do weekend work. I would love to help you push back on that. About how that’s damaging our team, distracting our team. I can give you some of the data, but I also have to be clear with you that my family is very important to me and I can’t keep working weekends. And so what I don’t want is for you to be in a position where you accept work that I’m going to have to decline.

[01:40:54] And then you’ll have the problem that you committed to it and it won’t happen. And so I don’t want to see you in that position. How can we plan to push back next time? That’s, that’s how you line it up in advance.

Ryan:

[01:41:12] And I’ve, I’ve heard this exact situation from a friend. Their manager was a yes man to manage their manager, very common. And so they were doing, committing to all these unreasonable deadlines and, you know, not protected at all. And it wasn’t a big deal because there’s turnover, but it’s fine, just hire new people and just keep saying yes to more and driving people as hard as you can. The narrative that you said there felt slightly more threatening than the, the other ones, because I can imagine, I can imagine that that more directly feels like you’re threatening not to do the work.

Ethan:

[01:41:52] So, number one, you’re right, that wasn’t as polished as some of the other lines I’ve given you. But I think with five minutes, I could create a more polished version. And one of the keys that you’re getting at here is the wrong time to start setting this up is while the request is in hand, what you want to do. Like, if you’re getting asked to work every weekend, the right time to have this discussion with your boss is like Tuesday or Wednesday, after the weekend’s over and a little time has passed and before the next crisis arrives.

[01:42:26] And the second piece, though is it’s then how do you make clear that your tolerance is limited? And yes, one of the costs of that, this is important if you’re not willing for the manager to say, well, if you don’t like it, quit, you can’t say anything. You’ve got to have enough confidence in yourself that if it does come to that, like when I went to my manager and I said, I need to know how important my career is to Amazon, I was also looking for an outside job.

[01:43:05] Like, I had decided, I’m going to do everything I can diplomatically to win this over, but I’m also okay leaving.

Ryan:

[01:43:14] I think that’s the big difference between this situation and all the other ones, which is in all the other situations, you had leverage. You could leave and they would feel pain. Whereas in this case, if you’re the lowly person or engineer, maybe they don’t see your value as much. If you leave, you say, okay, great, goodbye.

Ethan:

[01:43:37] I mean, if you really are that discardable, then you’ve not created much value in Other words, absolutely, leverage is required. But good engineers are valuable and teams that are total turnover factories are not that effective. Now, if there’s a company that totally has a view of bring them in, burn them out, move them on, you’re probably in a wrong place, like that is, you know, if you’re at a place where you can’t create any sort of value, the right answer is you will have to get out.

[01:44:15] And I think people get trapped believing there isn’t a better option. There’s always more jobs and I’m not belittling long searches or the difficulty or all the problems, you know, with maybe in engineering today of AI making it hard to get on entry level, but. But there are jobs if you do enough work. So one of the things you have to ask yourself is this being at the bottom of the funnel of bad work being rained down upon you, how bad is that compared to, say, doing the extra work to network and find something else or start your own thing?

[01:44:59] Don’t be a victim. You have agency. If you prefer, you know, you can look at ownership. The book Extreme Ownership by Jocko Willink, whatever. There’s many ways to look at it, but in the end you have to be willing to take control of your life. Even if that does mean, by the way, that you weather the bad manager while you complete the training program or find the job. What a lot of people like to do is complain about how unhappy they are, but not do anything else.

[01:45:33] I get it because it seems exhausting, like, oh, I just worked all day and now I worked all weekend. When am I going to look for a job? One of my beliefs is to work less, you often have to work more first, right? In other words, to get to the place where you can coast or have a good situation, you often have to do some ugly digging first. And this, the situation you’re setting up, is one of those.

[01:45:57] You also hit on another point that’s really important, create leverage. If you create that leverage, you’re always in the winning situation. And then thing is, if your stick is long enough, if your lever is long enough, it doesn’t take very much force, which is why I’m able to be polite about it. I can be very soft in my hints about what I like because the lever is so strong, people are like, oh, we want to make this work right away.

[01:46:27] And that’s what you’re trying to do, is have a big lever, I guess.

01:46:30 — Backchanneling

Ryan:

[01:46:30] On another topic, there’s this skill of back channeling, which is not in public forums, but one on One in different places, driving your message to the right people so that you get the right aligned outcome. So how do you effectively back channel?

Ethan:

[01:46:53] So I think of back channeling in two ways. There’s good back channeling and manipulative back channeling. Manipulative back channeling is where I’m trying to do something that I wouldn’t be willing to say in public. So that’s where you’re really undermining or painting the other, other proposal or the other leader in a negative light. I don’t think that’s usually necessary, and I’d like to say never necessary, but you’ll lose some because maybe they’ll do that.

[01:47:28] What positive back channeling is, is giving people a chance to ask questions and find out what their concerns are, where they don’t have to bring them out in public. So maybe, for example, I’m the CFO or I’m the financial person and I’m really concerned about the budget, but I don’t want to just ask a bunch of budget questions in the meeting or I don’t want to put you on the spot. Or maybe I don’t realize that you’re mostly concerned about the budget.

[01:48:02] If we have a back channel conversation, I can ask, here’s my idea. Is there anything you don’t understand? Do you have any concerns? And what I’m doing there is. I’m giving you a chance to feel, remember feelings and narrative. I’m giving you a chance to feel understood, to feel consulted, to feel listened to. Back channeling doesn’t just work because of information exchange. You do get data and that is good.

[01:48:31] But it also works because there’s all this subconscious I made you feel important, I made you feel listened to, I made you feel heard. And that emotional buy in, particularly if the other project leader doesn’t do that, gives you a huge leg up because you’re building trust. You’re building trust that, oh, you take input, oh, I can work with you. And so even if the two proposals are equal, or even if the other one’s a little better, remember part of my bet if I’m in executive funding proposals is who can I count on when the chips are down?

[01:49:08] Who can I count on to get this across the line? Who can I count on if there’s a problem? This other leader I don’t know as well, they didn’t consult me. Ryan came to me and asked me my input and showed understanding for my needs. Totally. I’m funding that because even though the payoff’s a little lower, I’M sure Ryan’s going to come through or I can work with him and so just realize we so underestimate emotions.

[01:49:40] As an executive making a decision on which project to fund, I’m wondering how it’s going to make me look. I’m wondering what happens if it goes wrong. I’m wondering what happens if there’s conflict between you and me later. Are you going to resolve it reasonably? I have all these fears and worries, and the back channel lets me quiet those down as well as address factual concerns. And so that’s why it’s so effective.

Ryan:

[01:50:05] I think if you were to negatively back channel, other people would see that you are a snake and then you would lose some rapport.

Ethan:

[01:50:17] I guess people do negatively back channel and they have to be careful about how they get caught, right? Or don’t get caught. You know, this is what you would call a backroom deal. You know, the classic backroom deal is like the kickback where you support this and I’ll make sure you’re taken care of. It’s kind of like, you know, a mafia movie or something, right? It’s that. It’s the I’m coming to you with a proposal that amounts to you should do the wrong thing because we’re both going to benefit and I just want to make sure we’re on the same page.

[01:50:51] And you know that even though I’m actually a snake, I’m not going to bite you. We’re snakes together. That’s the essence of that offer. And some people are up for that because some people are selfish and ambition runs wild.

Ryan:

[01:51:18] well,

Ethan:

[01:51:20] I had to pay my dues and I got screwed over when I was in the weak position. And so now that I’m in the strong position, it’s my right to take all I didn’t get. It’s not really I’m screwing that. See, they always tell themselves a story that makes themselves look better. It’s not really I’m screwing those people. It’s that I’m getting what I’m owed. Like, this is only fair. And that’s another thing that’s really important is just when you’re dealing with someone who seems to be vicious, do at least realize that their story to themselves is not that they’re evil.

[01:51:52] Their story to themselves is something else. And if you can figure out their story, you have a better chance to redirect that story.

01:52:04 — Influence without authority

Ryan:

[01:52:04] We talked a lot about soft power and influence, and it. It sounds to me like that’s not necessarily directly proportional to how many recursive reports you have. Have you seen examples where there was a person within an org that had a disproportionate amount of influence?

Ethan:

[01:52:22] Yeah, well, the first thing I would say is I’ve seen soft power come from ideas a few times. So Amazon for a while built a product. They called it Cloud Drive ultimately, but it was a storage system, kind of like Box or Dropbox, either one. And that proposal came from an entry level new college graduate engineer. He had an idea for it and he did two things to get it heard. The first was Amazon had an invite only engineering conference where only a tiny percentage of engineers went off site to this conference, but you could get an invite by making a proposal.

[01:53:05] So he fought his way into this conference. Then he knew Jeff Bezos came and reviewed an idea poster session. And so he emailed Jeff and said, I have this idea and I’m going to be at the poster session. Do you think you could stop by my poster which is a very small ask of the CEO. Well then he got an audience with Jeff and he pitched his idea and what Jeff did was like it and email his triple skip.

[01:53:37] So my boss, so there was Jeff, then this vp, then me, then a manager, then this new engineer. And my VP didn’t like the answer the idea at all. But Jeff had said look into it. And so, you know, and the project got funded, it got built. So outside his influence. And I can, I can tell another story about this. Whilst different people have different origin stories for it to some degree I worked with a slightly more senior person who created the Amazon Fire TV or helped put forward the proposal for it.

[01:54:19] And he also was not a super senior person, but he did a lot of legwork and had the knowledge and had the connections and managed to get that created. So what works here? How you get that outsize influence? I’ve seen it many times. You have to have an idea. And again, it’s that appealing storytelling. You’re telling a vision. These guys were telling visions. Wouldn’t it be great if we had a cloud storage system?

[01:54:50] Wouldn’t it be great if we had a low cost living room device where people could watch prime video? They were telling an understandable sound bite story that everyone agreed with, at least on the surface. Because wouldn’t it be great if we had a cloud storage system? Yeah. Wouldn’t it be great if we had, you know, a living room device? A low cost living room device? Yeah, sounds great. Oh well look, here’s, here’s a way we could do it.

[01:55:18] So I don’t know if you have a follow up question on that, but influence can come from anywhere. It usually begins with a good story based in some real fact. It can’t be usually total nonsense and then told with enthusiasm and told repeatedly and specifically told the people who have the power to do it.

Ryan:

[01:55:44] What about, what are the the least discussed reasons that people get fired that you wouldn’t see in the performance notes, but they are there. So kind of that second thing that’s actually getting someone fired when there’s an upfront story told in the perf notes,

Ethan:

[01:56:04] there’s a lot of rules about when you can and can’t fire someone. And so almost always the public story is some sort of performance story because that’s the one thing that’s allowed. So the hidden reasons are often if they’re not evil or the person isn’t just doing wrong work, it’s usually incompatibility. So the main reason I see people getting let go that isn’t discussed is their style and the leader’s style are very different.

[01:56:41] And that style can just be one person is very detail oriented, one person is very high level and this can be on either end. That’s one type of incompatibility. Another one can be with engineers. It’s often one person wants to be very tech oriented and what’s the best technology. And the other person wants to be very business outcome oriented of look what makes the money, I don’t care if it’s a hack.

[01:57:11] And where the firing occurs is where that becomes a religious debate and the two are just sort of at each other’s throats. And remember, what starts as a small thing often builds up where over time these become a label. That person is technically clueless, that person’s only about money and they start labeling each other and then it becomes personal. And once it’s personal, I’ve decided you’re a bad person.

[01:57:43] And I see all these things that make you bad. And you’ve decided I’m a bad manager. And once you have this lens, you’re a bad manager. It really doesn’t matter what I do. You’ll see everything I do is bad. Because so many things are ambiguous and open to interpretation. That’s where people get fired. But of course it’s always labeled somehow performance.

01:58:04 — Sexual harassment

Ryan:

[01:58:04] Well, you mentioned it could be labels perform. But you mentioned that you could get fired for performance. But there’s also those other HR reasons like sexual harassment and those types of things. Right?

Ethan:

[01:58:15] I’ve seen one of the Most blatant cases, in my opinion of sexual harassment, I have to be careful even now what I share. And that will make the point. There was an executive hired, very high level position, a chief level position, who immediately started harassing women. And he was so blatant that it ultimately, when we asked one of the HR representatives on his team about had she gotten any complaints, she started crying because he had harassed her too.

[01:58:49] So this is someone with the gall to harass a woman in human resources. But he disappeared. The leader terminated him right away as soon as this came out. But people ask me, how did such a person get in here? Didn’t we check references? And the thing I want people to understand is we didn’t have legal proof of this harassment. We had a lot of people who said it happened to them, we believed them. But we never took it through a court and got a judgment that said, oh, he’s a harasser.

[01:59:28] So even though that’s true, if I name this person and say, don’t ever hire Fred Smith, you know, he harasses women, Fred can sue me because I have no proof of that. I have no legal proof. I know it’s true. He knows it’s true. It. So that’s why companies don’t say it. So when someone calls up and asks about Fred, I refer them to HR and HR says, oh, Fred was employed here from this date to that date.

[01:59:57] And you know, they, that’s how they keep getting hired. I will say I’ve looked this person up on LinkedIn since then and the, the fact he ever worked at Amazon isn’t even on his resume. We caught it quick enough. He just deleted it. No awkward questions there? No. Why were you only there a month? He just deleted it, moved on. And that’s how they survive. So yeah, I’ve, I’ve seen and it’s hard to stop.

Ryan:

[02:00:27] That was the next place that that person worked more than one month or

Ethan:

[02:00:32] I guess, yeah, maybe he was more careful, you know, maybe because we caught it so quick. I don’t know, I, I don’t believe that what he did at our workplace was a one off. I don’t believe that. But you know, of course I don’t know that.

02:00:49 — Skip overruling firing

Ryan:

[02:00:49] Last time we met you mentioned that you as a manager could fire any one engineer that you wanted. Is there ever a case where you want to fire someone? Or let’s say a manager wants to fire someone but their skip says you can’t because the skip sees it and goes, that’s clearly there’s an ulterior motive here. That person’s fine. Like, has that. Have you ever seen that? Overruling.

Ethan:

[02:01:13] I have. I’ve definitely seen a case where a manager floats the idea. I’m not sure Ryan’s the right person for us. And the manager knows your work and says, why don’t I move Ryan over here? Right. Why don’t I? I think Ryan. And again, if I want to make you feel good about it as a manager, what I would say is, I’d say, you know, I see that you’re struggling with Ryan. I’ve seen him do well in other environments.

[02:01:44] Let me move him to this other place, he’ll be out of your hair. And I think he’ll be fine there. I don’t tell them that I think you’re going to be great there because that’s kind of hinting that they’re bad. I just make it socially easy for them to say, sure, move him away. That’s just as good as fired. From my viewpoint, this relies on your skip, of course, knowing or having a sense of things. And again, sometimes it is just incompatibility, your style.

[02:02:14] I can see that you and your manager are cats and dogs in a bag. It doesn’t mean that you can’t get along well. It doesn’t necessarily mean the manager is bad. It does mean they lack enough adaptability. So I can work on that. But I may choose to save you in that time. Because what gets forgotten is people have personalities and preferences and not everyone can work for every other boss. I certainly had managers that some people at Amazon love and they drove me batty.

[02:02:53] And I also know that for those managers, they knew I was a high performer, but they didn’t like my way of doing it. And that’s just style does matter. You know, you have music you like more than others. Some music you’re like, get this out of my ears. And some you’re like, I could listen to this all day. There is some of that element in work.

Ryan:

[02:03:17] I mean, the more that we talk about the behind the scenes of how managers keep orgs together and retain people, feels like the orgs are, they’re like duct taped together with a bunch of like, you know, well worded phrases. This person goes here, oh, I’m going to get you what you want next half and I’m going to move people. And it’s all kind of back of the envelope accounting in the leader’s mind. Is that an accurate assessment?

Ethan:

[02:03:45] You probably saw me smile when you said that. I certainly feel that way. What I mean is Managers start out like you would think. It’s just like I would compare it to an engineer with an architecture. When the project begins, you’ve got your nice little lines and arrows and boxes on a page of the system you envision. And when it ships, there is some resemblance to what you drew, but there’s all these special cases you didn’t think about and all these things you forgot that got like, well, what service can we put that in?

[02:04:21] Because we’re out of time and the thing is shipped, bears a resemblance to the thing you designed, but it is not this sort of pure, beautiful ivory tower. Well, I think the same is true of orgs where managers set out of course to manage only by performance and to have great managers and everyone happy and to develop everyone’s career. And then reality hits of schedules and difficult people and locations and office changes and people out on leave.

[02:04:54] And by the time it’s all done, you’re just really happy that it’s working mostly. And you know, I think that’s. I happen to enjoy military history. There’s a phrase for military history or military strategy that applies here that says, you know, with your battle plan, the enemy gets a vote. Like, it’s all great, you have a plan, but then someone starts shooting at you. Maybe if you heard it another way, the boxer Mike Tyson said everyone has a plan till they get punched in the face.

[02:05:27] Well, I think your manager has a plan for a beautiful org right until like deadlines and outages and whatever, punch them in the face and then that plan starts to become whatever gets them through the fight.

02:05:39 — How to fire managers

Ryan:

[02:05:39] We talked about individual firings for, for engineers. I think there’s a common sentiment that it’s harder to fire a bad manager than it is to fire a bad engineer. When you had managers reporting to you that let’s say they were low performing managers, how do you manage them out?

Ethan:

[02:05:57] So I think it is harder to fire managers. There are three reasons. Reason number one is they usually are somewhat more experienced. So they don’t have the obvious bad performance. Meaning the first time I ever had to let someone go, it’s because they stopped coming to work pretty easy. Like, well, it’s Wednesday and you haven’t been here yet. I’m calling you to let you know, don’t bother to come in tomorrow.

[02:06:26] Like pretty cut and dry. Managers don’t usually make mistakes at that level. So they’re more subtle though their weaknesses are more subtle because they got made a manager for some reason. They were technically good, they were politically good, they were organizationally Good. So they have some kind of strength that got them there. That’s reason one, Reason two is a manager’s work is hard to measure.

[02:06:51] How do I exactly measure a manager’s work? Well, is it that they hired? Is it that they did their performance reviews? Is it that the feature shipped on time? Is it bug count? The manager’s job? It’s very hard to put a lot of metrics, which means when it comes time to fire or to consider performance management, we’re in this he said, she said opinion problem. And I think the third reason it’s tough to fire managers is managers are part of the system.

[02:07:28] And so they know what it looks like. They know, like, oh, my boss isn’t that happy. How do I start positioning myself to be hard to get rid of? Like, they have a quicker Spidey sense, if that’s what they want. You know, if they want to stay, they have a quicker sense of how do I. How do I make myself difficult to pin down? And of course, it’s still possible. Like, I terminated a large, you know, not a huge number, but a large number of managers.

[02:08:02] It’s just. It’s trickier.

Ryan:

[02:08:04] Yeah, it sounds really hard to terminate. How do you do it then, in those cases?

Ethan:

[02:08:09] So my ideal way of doing it, which I wasn’t always able to do, is to talk to them and surface the idea socially that maybe this isn’t working out and would they like to collaborate on finding a better place? So I’m phrasing it. As I notice, I have the impression that we’re struggling. I would say something like, I don’t think we’ve gotten off to the best start here, and I’m wondering how you feel about it.

[02:08:42] Would you like to work on that or, you know, do you also see that problem and you’d like my help finding some other option? Very nebulous. Smart managers get the message. I’m your boss. I’m not happy. I’m offering you a door where I give you some time to find a job or maybe a severance package. Do you want to do this the easy way? Many managers will be like, yeah, I do. You know, and of course, they’ll never say, oh, of course you’re right, I’m failing here.

[02:09:16] I want a door. What they will say, if they’re equally smooth, is they’ll say, I’m so sorry that you’re not satisfied. Of course I’m doing my utmost. That’s me telling you, remember, I can make this hard for you. So you’re going to Pay well for my exit. That’s. That’s the counter negotiation. But ultimately, I want you to be happy and satisfied with my work. And if you’re not, of course, I’d like to discuss what changes we could make, which is my way of saying, I’m open.

[02:09:51] I’m open to making it, you know, so I’ve never used the word fired or performance because that creates something legally called constructive termination. Where if I’ve said, you know, we might have to fire you, do you want to quit first? That is a legal problem called constructive termination, where I. Anything I do from that point is then seen as I’d already decided to fire you and now I’m making up reasons.

[02:10:18] So I very carefully avoided that. They very carefully avoided ever saying, acknowledging that there’s an actual problem. What they say is, I’m sorry you feel that way. Right? Oh, I’m sorry you feel that way. That’s your problem. You feel weird things, but I want you to be happy. So if you’re not, of course I discuss alternatives. And that’s their way of saying, yeah, okay, I get that it’s not working.

[02:10:45] What are you going to give me? So that’s one way you quote, unquote, fire. The other way is classic performance management, in which case you have to try and write down clear goals and objectives. You have to start documenting. You have to give. You know, there’s a legal standard that. There’s a legal standard and there’s what HR wants, but you basically have to document a bunch of expectations and then document that they didn’t hit them.

[02:11:16] And then you have a performance case. Almost all managers will quit before that. What they’ll do is once you start the process, they’ll be like, oh, okay, this is going to take him 90 days. I have 90 days to fil. Find a new job. And they go find a new job.

02:11:31 — Leverage when people are getting fired

Ryan:

[02:11:31] You mentioned that when the manager, I guess, knows the system is getting managed out or this conversation starting, they can sometimes switch to, how can I extract more value for myself? And I think a lot of people who are less experienced or they don’t know how the system works too well is when they’re starting to get fired. They don’t. They feel they have no leverage. And the fact that severance exists makes me think that they do have some leverage, although I don’t know if I fully understand it.

[02:12:07] What, what is the leverage that, let’s say, an engineer has as they’re getting fired that, that a company would want to pay for? Because severance, I mean, well, there’s cost, there’s the optics of it. Like, it looks good, of course. But I’m after this conversation, I’m a little more cynical. I feel like they’re paying for something for that report. So that report must have some leverage in that conversation when they’re getting.

Ethan:

[02:12:34] Well, there’s several pieces of leverage. The first is this process. If you want to go through it, is going to take a lot of your time and you have to document things. You have to follow a process. You have to file a bunch of paperwork. HR is going to audit your paperwork. If you’re a low level manager, you know they’re going to write it. There are things I can do to string things out as a tactic.

[02:13:00] And so the first offer I’m making you is, hey, I’ll save you all that time if you will give me some money so that I have money in my pocket to go look for another job. I will save you this time. And frustration. That’s the first level of offer. The second level of offer is the story I’ll tell the team. Right. Is, look, I can either leave talking about what a terrible experience this was and what a monster you are.

[02:13:31] I don’t say those exact words to you, but I can show you how to say that, or I can leave saying, you know, this just wasn’t a fit for me and I’m so excited to be moving on to Acme Inc. And I can’t speak highly enough about how Ryan treated me through this whole process. Ryan’s amazing. So which story do you want me telling my co workers? And so those are a couple of the pieces of leverage. And again, it’s this.

[02:14:00] What’s that phrase of mine you like so much? The plausible story or plausible fiction?

Ryan:

[02:14:05] Polite fiction.

Ethan:

[02:14:05] The polite fiction. If I need to say this to you, I say, well, you know, ah, let’s say I’m working for you, Ryan. So far you’ve been nothing but supportive. And if I were to move on to a new job someday, I would, of course, explain how supportive you’ve been. But if I, if, if we’re discussing my moving on, you know, really, I would hope you’d want to support my transition financially so that I can tell that story.

[02:14:44] And that’s how I’d be like, you know, and even that wasn’t maybe as smooth as I would do it if I had to. And by the way, let’s say you, you’re starting to give me performance feedback and I need time to be in the right headspace to have this Conversation with you. I would just say that. I would just say, you know, Ryan, I appreciate you’re starting to give me some performance feedback. I know we need to have this conversation.

[02:15:15] Honestly, this moment. I can’t have it productively. Let’s have it tomorrow morning. Or can we have it tomorrow morning? And most managers won’t be like, no, I need to tell you how rotten you are right now because they’re already on edge a little bit. And it’s. So if you’re putting up this flag that says, I’m about to lose my shit on you, but tomorrow I’ll be a good child, they’ll be like, sure, come back tomorrow.

[02:15:40] Because they’re sort of trained to. And I’ve put you in a really difficult situation. This is important where if I’ve asked for space and you say no, we have to finish this right now. Now I have a beef with hr. He was so mean to me. I asked for space. I asked if we could just talk tomorrow morning. And he said no, I’ve got to ridicule you right now. Very empathetic to hr. And now you’re the one in the hot seat.

[02:16:07] Why couldn’t you give her until tomorrow morning? Why couldn’t you give him until tomorrow afternoon? Again, I hope. I would rather coach people on how to be in great high performing situations where they’re having different conversations. But you want to talk about the dark side, and that’s okay. I get that people want to know, don’t be afraid to buy time. You do have leverage and you may not get anything.

[02:16:38] Remember, if your boss is starting to line you up to terminate you, you may not get anything. But if you don’t ask, you certainly won’t. If you don’t open the door and say, well, you know what? How could we make an amicable deal? And here I’ll give you your last piece of your leverage. Remember, your manager isn’t sitting across the table wanting to feel bad about themselves. They want to feel good about themselves.

[02:17:08] And if they have a chance to spend a little bit of the company’s money, which isn’t really their money, so that they can look in the mirror and say, well, it didn’t work out with Ryan, but I took care of him, I gave him a cushion, they want to do that. And so they’re saving them hassle. There’s the public story you’re going to tell and there’s. They want to feel good about themselves. Most managers don’t like.

[02:17:37] They hate firing. They avoid it. You know, as much as you might think, meaning you, the listener, you. Someone might think managers are just sitting around going, who can I. Who can I pitch next? They’re not. They actually hate it when this is important. One reason people don’t survive firings is by the time the manager actually decides, oh, I just. I’ve. I’ve tried to coach them. You may not feel this way, but in their mind, they’re like, I’ve tried to coach them.

[02:18:06] I’ve tried to pair them with people. I’ve tried to give them all these chances, and they’re still not getting it. I don’t have any choice but to fire them. Once they’ve gotten to where they’re going to act, they’ve told themselves they’re out of options, which is why people getting back off the brink is very unlikely. You know, interestingly, right before I came here to talk to you today, I had lunch and I met a new executive.

[02:18:30] He owns a company that does half a billion dollars of business a year. So it’s a big company. He’s built it over his whole life. And he was telling me how he’s going to have to hold someone in his organization who isn’t performing these. Kept on his staff for two years, he’s finally at the point where he’s going to have to hold them accountable. And what I’m thinking is he’s put off that hard message.

[02:18:57] When he finally gets to where he’s going to give it, he’s so decided that they don’t meet the bar. They can’t get out of that. And it’s not because he’s evil. It’s just people delay. And so I coach people. People come to me all the time because I’m online. They say, ethan, I’ve just been put on a performance improvement plan. Can you help me survive it? Can you help me recover? And I tell them, honestly, no, you’re most likely going to be fired and there’s nothing I can do for you.

[02:19:29] But here’s why. It’s by the time someone gets around to giving you a form of formal performance plan, they’ve tried everything else they could think of, and at this point, they just want you gone as quietly as possible. And it happens again all the time. People write me back and they’re like, you don’t understand. I’m going to. Three months later, I get an email from this, as you were right, I was fired.

[02:19:55] Of course, there are exceptions to this, but it’s so common that I see this pattern of, how can I survive this? I’M gonna. I’m a hard worker. I’m a good person. I’m gonna show them. Three months later, you were right. I got fired.

Ryan:

[02:20:09] Even if people had the awareness, because I guess they’re gonna listen to this conversation, I think they still lack that. A lot of people lack that skill of creating the polite fiction. You know, how do you and I, we talked about. We don’t need to go deeper in it. But I do think that even if I. Someone came to me and they said, I’m in this situation, and I said, here’s your leverage. Good luck. I think they would have trouble delivering that message in an effective way of saying, for instance, especially the team health one, because the team health one is more of a threat.

[02:20:43] The saving time one is, hey, I can do something good for you. I’ll leave quietly. Just help me get some severance. The team health one is, I can make this hard for you unless you get me sufferance. I think delivering that kind of message as you’re leaving, especially when they have such a negative bias on you already, that’s going to take some incredible people skills. Yeah.

Ethan:

[02:21:06] Like, yes, you’re right. Okay. This is part of the message, which is engineers in particular want to believe this is, in fact, maybe the most important message, even if it’s not the most exciting, which is, as an engineer, you want to believe it’s your technical expertise that controls the fate of your career. And it’s not only that, it’s your people skills. It’s well known. Right? Satya has talked about this.

[02:21:32] Other leaders have talked about iq without eq, without emotional intelligence is a waste of iq. And so having strong interpersonal skills is critical. You know, it’s funny, there’s a famous book, I’m sure you know it, by Cal Newport, called so Good They Can’t Ignore youe. And what I find ironic about this is it’s all this stuff about how to be such a good worker, you can’t be ignored. Except this is a guy who has the social skills to both teach at Princeton and write books.

[02:22:04] He’s not without the human skills to present his incredible technical accomplishments delicately and well. So I think it’s a huge area of learning. Now to exemplify this, how would I, if I, on the spot or here, you know, live, had to construct a better way to say that? I would say something gentler, like, boy, this is such a hard message. And I have so many friends on this team. I just. I don’t want them to think badly of me.

[02:22:44] I don’t want them to think this is an unfair environment. Can you work with me so that this is a graceful exit?

Ryan:

[02:22:54] And that’s good, that works. You know, I don’t feel like started at all.

Ethan:

[02:23:00] And you know, later if I need to, I said, well, for it to feel good, I mean of course, you know, like I rough if it, you know, for it to feel good. I need enough security to get to my next thing. And I know you want that, I know that you don’t want anyone to suffer. And I tie into the other thing. But it is possible. And that is why I said this is how you get the time to go figure out how you’re going to deliver these messages.

[02:23:30] Messages. Something people overrate or I guess underrate is recognizing that a lot of hard messages you don’t have to deliver cold. I mentioned this before, but you, you can buy the time or make the time to pre plan. Maybe not every word. It isn’t memorizing a speech, it’s more memorizing a few phrases and how you’re going to, you know, say what you need to in a polite way.

02:24:01 — How to grow past senior eng

Ryan:

[02:24:01] I want to pivot to promos and talking about promotions because I know there’s a lot of stuff going on behind the scenes on alignment between managers and upper leadership on who’s going to get promoted. You mentioned the queue earlier and I think a lot of people, they get, they get stuck at this staff engineer, which is equivalent to the frontline manager level of how to get the promo to go further into those higher levels.

[02:24:28] Do you have any advice on how to break out of that, that common level that people get stuck at?

Ethan:

[02:24:34] Yeah, I mean, this is, this is my bread and butter, what I teach every day and kind of if I have a signature expertise, it’s how to move up further. What generally gets people stuck is they’re, they’re not learning the new skill of the next level. Instead they’re doing the current skill harder and harder. And so what gets you promoted early in life is hard, effective work. It is hard work. It is delivery.

[02:25:03] It is shipping a lot or building a lot. But as you get further, you need different skills. You need to be able to delegate, you need to be able to communicate. You also need to be able to let go of your hard skills because if you’re trying to micromanage everyone and tell them, write it this way, write it that way, build it this way, you’re driving your engineers nuts. And they, you’re going to drive away your good people.

[02:25:26] And so the way to get to the next level is to figure out what do I need to let go of, step away from, and what new thing do I need to master. And there’s a classic book on this. It’s by a guy named Marshall Goldsmith. It’s what got you here won’t get you there. It’s literally shows a ladder with a gap in it. Like, what got you this far isn’t going to get you up there. That’s kind of the classic book.

[02:25:54] But if you need specific advice to your role, it’s absolutely talking to people a level or two above who’ve been through this and asking them, what did you have to let go of? What did you have to learn? How do I do that? What do I need to let go of? What do you see holding me back? I’m a huge proponent of getting advice. Getting advice is even better than getting feedback. If I ask for feedback, I’m basically asking you to do something that you may struggle to do, which is criticize me.

[02:26:26] If I ask you for advice. Everyone has advice to share, and they’ll give you the same information, but in a more positive way. So to break through to that next level, it’s identify what’s different about that level and get good at that as opposed to just work harder. I see so many people working brutally hard, but ineffectively.

Ryan:

[02:26:51] When it comes to the most senior promotions, I think there’s this idea of readiness, which I feel like is what you just talked about, which is developing the skills and being in the position where your manager would say, yes, this person fills out the template, I guess, for. For their promotion. But then there’s also the case of limited number of spots and kind of positioning yourself so that you are put in one of those spots.

[02:27:16] And so imagine there’s a case where I’m a senior leader and I’m maybe I’m going for the director position, or maybe I’m going the. The principal engineer position. And there’s two of us both going for at the same time, but we both report to you. How would I position myself so that I get the promotion first?

Ethan:

[02:27:36] Absolutely. Let’s assume you’re both equally qualified for a moment. So you’re not going to win on skill. It’s there in relationship and within relationship trust. So trust. Trust is about, do I have to look over my shoulder and wonder, are you doing things not the way I would do them exactly, but in a way that won’t make me look bad? That will, if you prefer, make us both successful? Do I trust what you’re doing?

[02:28:02] Or do I? Because as a senior leader, I don’t have any time to micromanage you. I’m so busy myself, I need to be able to give you something and assume it will get handled. And trust. I often tell people, interestingly, trust is earned more in bad times than good times. My favorite analogy for this, which is a little bit violent, is when two people have been in a war together and I had to put my life on the line for you or you for me.

[02:28:30] We now have a lot of trust, because if I trusted you, like, if you came through when the bullets were flying, I know you’ll come through the rest of the time. Well, what’s the closest thing at work? This is usually a crisis, a big outage, a big embarrassment, a lost account. If you jump on that and somehow fail, fix it, or show all the effort and it hurts you or it costs you, you cancel a vacation.

[02:28:55] Not that I’m recommending doing that a lot, but if you make a decision that shows that you’re dedicated, I will remember when the chips were down, when we lost the Acme account, Ryan came through. He flew back from Hawaii early and made it. And that’s trust. Second thing is relationship. Relationship. In this case, I’m honestly thinking of, do I like you? And the point is, if I have two relatively equally qualified people and I’m going to suddenly spend more time working with one than the other, why would I not choose the person I enjoy?

[02:29:31] And again, I know some engineers are just. If they were dead, they’d be spinning in their graves. They hate that idea. Look, one person’s a little bit more difficult to work with or even just a little bit less fun, and the other one’s more fun. I have to spend eight hours a day with this person, or six, because we’re not in every meeting the same. I’m going to be drawn towards the person that’s fun.

[02:29:58] And I know that seems unfair. And it doesn’t mean you have to kiss up to everyone. It does, though, mean it’s a good idea to be friendly and happy, right? Like, be, be pleasant. Be, be. Be that helpful, smiling person, because I’d rather spend time with you. I heard it expressed this way by Amazon leadership, by Bezos, I think, which is, when you come into the room, do you suck all the oxygen out of it?

[02:30:27] Like, do you bring the room down where the room’s like, if you’re that person, you’re not going far, you’re not going any further. Whereas if, when you come in, they’re like, Right. Like, oh, Ryan’s here. This just got better. They kind of want you in that room. And, you know, it’s. It’s a. So if you want a simple rubric, do people want you in the room? If they kind of don’t, you’re in trouble. And if they kind of do, you’re doing great.

Ryan:

[02:30:55] I do agree. I think a lot of engineers are gonna. This is gonna piss them off, but that’s okay.

Ethan:

[02:31:00] See, it’s true. What I would say to the engineers I piss off is I am trying to help you. I’m not actually here because you’ll like me. I’m here to get you where you want to go. And that requires, you know, you asked me, could I be transparent? Part of the transparency is sometimes transparency is telling you things that I know you’re not going to like but that are holding you back. And you’re letting the fact you don’t like it let you get your backup and be like, I’m never going to do that.

[02:31:27] Well, okay, but then you’re never going to move up. And I’m not here to apologize for the way the world works. The way the world works sucks, but it does work that way. And I want people to know that.

Ryan:

[02:31:40] When I think of the most senior level promotions, baked into the promotion packets is this idea that you need champions, or sometimes known as allies, which is people at that level or higher that say that person is doing work that fits my. My perception of that level. So because of that, I’ve noticed there’s this artificial mechanism, like, let’s say I’m. Let’s use management as an example. Let’s say I’m a senior manager.

[02:32:09] I want to become a director all of a sudden. If I want to get promoted, I need to make sure that that VP over there knows my work and will vouch for me in the promo committee and that other VP over there. And there are cases where naturally I might need to be visible in front of those people, but there are also a lot of cases where it’s just, I need to fill out the promo packet in the way that we need so that the promo will go through.

[02:32:36] How would you go about that? Let’s say you are a very senior engineer or a very senior manager in tech, and you’re trying to get those allies, but there’s no. At least as far as you could see, there’s no natural collaboration with those people. How do you get them as a champion on your promo packet?

Ethan:

[02:32:55] So I’m going to tell you my favorite way and then I’ll give you other ways. My favorite way is you have your boss on your side, you’ve got that team with your manager where your manager wants to get you promoted. And what I did in those cases is I would email my peers or my skips and I would say, you know, Ryan’s going to be up for promotion in six months and because your group is adjacent to ours, I’m going to need to ask you for feedback.

[02:33:26] So I’m letting you know that you’ve got six months to watch Ryan because I’m going to be coming to you. And number one, me asking doesn’t feel self serving because I’m talking about you. And number two, six months later, when that vp, I’m sure wants to say, well, I only saw Ryan twice. I’m not sure if I can write anything. I have, they know that I’m going to say like, hey, I told you, like I put you on notice.

[02:33:56] Now sometimes I’ll just say I’m sorry it didn’t come together. I don’t know, I can’t provide feedback. But that’s, that’s the best way. And I did that when, when I got good at getting seen. This is important as a manager. I got good at getting my people promoted and that allowed me to attract more talent because, for example, I had a guy who didn’t get promoted and I said, look, you know, I’m good at this.

[02:34:22] I, I will get this done in six months, but you’ve got to give me till the next cycle. But you know, I come through so you have to trust. And he later told me, he’s like, yeah, that sucks. But I, I decided I’d give you six months and we got it done. So that’s the best way. Now let’s assume your manager’s not that on board or they lack that skill or you can’t sell them on it. By the way, if you have good relationship, I’d tell them like, would you consider putting them on notice that you’re going to need feedback?

[02:34:50] If you can’t, the next best thing is for you to go talk to them and say, hey, I’m up for promotion. I’m pretty sure my manager is going to ask you for feedback. You’re a logical feedback provider. What can we do over the next six months so that you’re in a natural position to give me, you know, to give feedback or, you know, how could I keep you informed of my work? What would, and one of the keys here is if you need, you know, whatever Four champions.

[02:35:25] You’re gonna have to ask six people or seven because one of them is going to leave the company and one’s going to get reorged and one’s going to say no. So you have to like over submission, subscribe. It’s like, you know, over capacity, just like anything else you design. And it is awkward, you know, to go to a, like a, another group skip. But the thing is you can frame it to them as hey, you know, this is our process.

[02:35:51] Like, you know, I’m sure you get requests all the time for feedback and you probably get a lot of late last minute requests. I’m trying to do it right and talk to you six months in advance. And so what you’re saying there behind the lines is see, I’m already good. I’m like not causing you to get hit with no notice. I’m here, you know, setting this up in advance, making it easy for you.

Ryan:

[02:36:18] Well my immediate thought, if I was that VP and someone came to me with that self serving request, I wouldn’t necessarily think that’s bad or anything. I think okay, that’s the process. But my immediate thought is this is not a high pry for me because I’m a vp, I got a billion things on my plate and this random person from another org is coming to me to serve themselves. Not necessarily something I care about, but just shows that it’s low on my priorities.

[02:36:47] How do you get them to prioritize that?

Ethan:

[02:36:49] Well, remember there is some truth that’s low on their priority list but they have people they need to get promoted where they need other VPs. So you have a VP too. And that VP this, the VP you went to knows that they’re going to need your VP’s help with some promos. And so the leverage you have without ever bringing it up to them is they do know it’s a system and it’s kind of part of their job. So is it a high priority for them?

Ryan:

[02:37:19] No.

Ethan:

[02:37:21] The only way you can make it a high priority is if you’re able to explain to them some relevance of your work. You’re going to have to give them what’s called wifm. What’s in it for me? You’re going to have to find some way to connect what you’re doing to something they care about at least a little bit. How are you going to make their life easier, faster, less stressful? When these systems work as designed, those VPs do work with your area somewhat and so they should have you.

[02:38:00] You shouldn’t be asking someone who like has no relation your work whatsoever ever. It’s just they don’t really need your level of detail. But you, you can try to find some way to give them a little bit of nugget. I do think these systems work mostly though because vice presidents do know that this is how the system works and that they kind of need to do it. And if vice presidents are good at nothing else, they’re usually good at understanding this is the grease that keeps the thing flowing.

[02:38:34] And by the way, I can write some mild approving verbiage of you very easily. It may look maybe because it was like I cut and pasted and replaced John with Ryan but you know, and replace you know, whatever the ops Org with the, the you know, the customer Org but I’ve written a million of these and I can do it pretty quickly.

Ryan:

[02:38:57] Well, I think a lot of those, those champion. The champion support is a little bit contrived and it’s hard for me without my manager’s support to get a. You said what’s in it for me? Like a legitimate case of what’s in it for that vp because they’re so far, they’re so high up. I mean I know some people who, they’re in this spot, they’re very senior ic their manager is supportive but they lack maybe that proactivity that, that you have.

[02:39:26] And obviously the, the engineer has the proactivity for their own case. So they, they debate, you know, should they go directly to that person? If so how. Because it doesn’t. They don’t have a great case.

Ethan:

[02:39:39] Yeah. And I think, I think that coaching your own manager works pretty well in this sense. Your manager often knows what they need to do or at least is open to it and it just hasn’t made their priority list. Because you obviously care more about your career than your manager does usually. Right. Because it like that’s just human nature. And so the manager means well and wants to help but isn’t. Isn’t getting around to it.

[02:40:04] And so if you just poke them a little bit. The other thing that is a middle ground is you can ask your manager would you send a note to that VP like I know I need to go get them to support me, but I don’t really know them. Can you send them a quick intro that says hey, I have a guy who’s coming up for promotion. He wants to come talk to you. You know, would you take a 15 minute meeting because we’re going to need your feedback.

[02:40:33] It still gives you, you still are doing most of the Work, they’re just sending an email. But that’s a middle ground. You’ve opened a topic though. I really want to point out to all the engineers. You asked me had I ever seen a case where a skip level saved an engineer from being fired. And I said yeah, I’ve seen people moved around. What I never really seen is an engineer or any other anybody else get promoted over their manager’s objection.

[02:41:03] You have to have your manager’s support because I can’t if I’m the big boss. I can’t promote you if the manager doesn’t agree because I’m just creating a mess. I’m telling the manager they’re wrong and now I’m telling them that they have to work with you more senior, you know, even though they didn’t agree, I’m just creating a complete mess. So if the manager isn’t supportive, I think engineers envision like I’m going to do such incredible work and I’m so technically astute that I will be recognized.

[02:41:37] Even though this manager is a bozo, it’s not going to happen. It just, it can’t. And so you either have to get your manager on board or get out from under them and find somebody who is on board. Now a plug for people like me. Go find the managers who. I made a profession of getting people moved up, not undeservedly, but fast. And hell, that became a huge selling point, right? Go to Ethan’s org, he’ll make deals and get you promoted.

[02:42:08] That was my recruiting tool. Come work for me and I’ll make it happen. Because it was known that I would back people and get the job done. I could attract talent and I could keep talent. I had one guy, my biggest star employee began as a mid level engineer, what Amazon calls a level five. He went to a senior engineer, he moved to manager from manager at an equal sort of lateral promo. He went to senior manager, then to director and by that point he caught up to me.

[02:42:41] I was a director too. So we were out of space and he went to found his own company. Now he was a top performer, but three promotions in eight years. We worked together eight years. You know, we’re still close friends of course, but he powered my career and I powered his. But that example also brought other people like, oh, look at, look at what happens in this.

02:43:01 — How to avoid politics

Ryan:

[02:43:01] Or we’ve been talking about politics for almost three hours now. And I can feel many, at many points people listening are probably thinking, I hate every single moment of this. I hate this. Some people are probably listening because they feel like they have to, but they wish they didn’t because have to know how the system works. And I know it’s inevitable a little bit that, you know you can’t avoid politics completely.

[02:43:31] But let’s just say for someone who really just despises politics for all it is, what could you do to avoid politics if you were working at a company?

Ethan:

[02:43:42] I think number one, it’s about choosing your leader. So if you can find a leader who shares your disdain of politics, hopefully that maybe comes from a similar tech background and wants to reward they believe as much as you do. It should be all about that work. Find that leader. That’s one thing you can do. The second thing is go work on the hardest area you can get into where expertise is really going to matter and you can stand out the most.

Ryan:

[02:44:12] good,

Ethan:

[02:44:12] solid engineering work but that lots of people can do, it’s harder to stand out. I remember Andy Jassy. He was CEO of AWS at this time, but in the past he had been, I’m sorry, before he was CEO of all of Amazon and he was talking about principal engineers and he said principal engineers are normally expected to like advise others and mentor and do architecture. But these guys, these few guys working on aws, we just need them solving the hardest problems.

[02:44:45] And so I absolve them of all of that other stuff. Well, for those engineers who just wanted to be. I don’t want to review architecture, I don’t want to give feedback, I don’t promo. I He was the perfect boss because he felt there they were working on such hard problems, he couldn’t spare them to invest in the rest of the org and he didn’t care. So you’re looking for that situation where your work is so essential because.

[02:45:14] Absolutely. Look, you’ve asked about difficult people and I’ve given you all these rules about it’s great that your boss, it’s important your boss like you. If you have a skill that no one else has and that skill is valuable, you will get exceptions. I remember Andy again when he ran aws. I remember him talking about a high level employee who a couple people criticized him and said, that guy’s not working that hard.

[02:45:47] You know, he’s not really working at an Amazon level. And Andy said, yeah, I think you’re right. But he’s in a role where we need him and I think he knows we need him and so we’re just going to have to live with it like he does. The one thing we really need. And you could kind of tell Andy was like, Andy’s one of the hardest managers I ever worked for. Just very demanding. But in this case he’s like, yeah, he’s got the leverage is what it is, you know.

[02:46:18] And he was very like good on him, was kind of his, you know, like, man, he’s kind of got us. We’re just gonna have to live with it. That’s what you have to do if you don’t want politics is you’ve got to be, you’ve got to have that essential skill again. You have to have some kind of leverage. So you have to literally live up to Cal Newport and so good they can’t ignore you.

Ryan:

[02:46:44] Earlier in our conversation you talked about how when everything’s growing, it’s positive. Some when everything is kind of stagnant in an organization, it’s a little more zero sum where people are battling. So I can also imagine a growing company is much less likely or will have less politics. I think some amount is unavoidable.

Ethan:

[02:47:04] Yeah, I like to give people an extreme to think about. And my extreme, I often point out, is the post office. Like what’s going on, good at the post office. And the only kind of roles are carry the mail or be the postmaster and there’s one postmaster. And that until that person leaves or dies, you don’t get that job. And so it just feels like growth changes so much of that because it’s not 25 people wanting one promotion.

[02:47:33] It’s, you know, lots of opportunity. And I point out during the 15 years I was at Amazon, the staff size of the company I joined, it was 14,000 people. I left, it was 1. 4 million. So it grew a hundredfold, which is insane. As a result, I was able. My first team there was six people and nine years later I had 800. That kind of growth opens up doors that in a more normal company that’s maybe growing 5% a year or 10% a year, it’s just never going to happen.

[02:48:12] So I strongly believe find high growth.

02:48:15 — Advice for younger self

Ryan:

[02:48:15] Last question is if you could go back to yourself at the beginning of your career and give yourself some advice specifically focused on this, I guess navigating corporate politics. Is there something that, that stands out to you that you’d say, yeah, I

Ethan:

[02:48:31] God, I wish I could go back to early me and share the idea that it’s not just being right, it’s, it’s not just being right, it’s not just being technically astute. There’s all this other stuff going on and wake up to it. I think I was blind or deprioritized it and thought, well, that’s. That’s management stuff. And even as an individual contributor, understanding, well, management stuff does control the world.

[02:49:02] I wish I had been more aware. So I guess the advice to get it down to a single sentence is take it seriously that your relationships are what actually control your progress.

Ryan:

[02:49:15] Awesome. All right, well, thank you so much for your time, Ethan. I really appreciate it.

Ethan:

[02:49:19] Thank you so much for the chance to do this a second time. It’s been so much fun and the time has flown by.

Ryan:

[02:49:24] Awesome. And do you want to plug your stuff real quick?

Ethan:

[02:49:27] Well, I think if you’ve gotten something from this discussion and you want to know how to navigate your career, that’s what I do all the time. I teach classes on it. I have a newsletter on it. So if you want help. My goal is to help people succeed rather than bang their head against walls. If I’ve done some of that in this podcast, I can do more of it with more time. All right. Three and a half hours in the chair, baby.

Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.

Timestamps

00:00:37 - How he convinced Google leaders

00:09:26 - Building the MVP

00:11:43 - How he made time for Kubernetes

00:25:28 - Technical details on building Kubernetes

00:38:46 - Rallying the open source community

00:50:01 - Scaling Kubernetes up for AI training workloads

00:55:31 - Reflections on getting a PhD

01:00:22 - The inevitable trajectory of software is death

01:04:16 - Top book recommendations

01:05:22 - Advice for his younger self

Transcript

00:00:37 — How he convinced Google leaders

Ryan:

[00:00:37] Let’s start with Kubernetes because that’s super interesting. I don’t fully understand the business motivation. Like let’s say I was your director or something like that and you, you came to me at this and you said hey, let’s do this for everyone. I don’t fully understand what would be in that strategy doc or that, what would you say in that meeting that would say here’s the impact for Google if we invest in building this for the industry?

Brendan:

[00:01:09] Yeah, it’s funny because the hardest part actually of the project I would say in those early days was actually articulating that. And I think it was really clear in our heads. But figuring out how to convince people was tricky. And I think there were a variety of different ways that we articulated why it was important. One of them was related to the MapReduce white paper. So MapReduce at the time, especially Hadoop and Big Data were a big deal.

[00:01:45] I think that other things have kind of replaced them at this point. But like MapReduce was a big deal and the big data revolution or whatever they called it. And you know, Google had written the original white paper, but Hadoop was an open source project that Google had nothing to do with and got no credit for. Right. And they just read the white paper and they re implemented it and it’s not the same.

[00:02:12] It’s similar but it’s not the same. And so part of the argument was like, look, we have this cloud and we want to be influencing the technological landscape. If all we do is kick out white papers, we’re not, if it doesn’t run, if it’s not something that people can run, we’re not going to be in the driver’s seat. And so that was one of the arguments. I think the Other couple of arguments were like, why Containers?

[00:02:43] Why not why, why people are using VMs, why containers? And a lot of that was talking about, look, the demands of writing software. We know internally, we know from doing this internally that the demands of writing reliable software necessitate having systems that are sort of like autopilots for your application. And we know that this is something that, as software becomes more and more critical to more and more businesses, this is something that they’re just going to have to have.

[00:03:14] And so that’s sort of the why Containers part. And then I think that the third part was the like, why Open Source, right? And in some states that’s like the most interesting conversation because people are like, wow, this would be, you know, you’ve convinced me, right? Like, you’ve convinced me, we should build it, we should make it available to the world. It’s something that they’re going to find useful.

[00:03:34] But wouldn’t it be so much better if they could only use it on our platform? And you’re like, yeah, that’s absolutely the case, but you can’t win if you make it only on your platform.

Ryan:

[00:03:48] Why is that?

Brendan:

[00:03:49] Well, because there’s other platforms out there, right? And so if you make it an exclusive, then the people who, for other reasons are on other clouds or on, on premise, they’re shut out and so they’re going to just go build an alternative, right? Like the open source. It’s sort of like the Linux. You know the reason Linux won, right? It’s because Linux could go anywhere, right? The whole reason that open tech and open ecosystems win is because the majority of people are going to be not on your platform.

[00:04:25] Like, if you’re, if you’re not the leader and you know, GCP was not the leader, then the majority of people are not going to be on your platform. And so if you make it such that the majority of people can’t use your thing, they’re just going to ignore you and then they’re going to go build their own, right? Whereas if you go and build it and you build it for everybody, but you make sure that it’s awesome on your platform, then you have a chance of attracting more people to your platform than otherwise.

[00:04:51] And then I mean, also in some ways it’s just like the aesthetic of the time also, right? It’s like if everybody’s using Linux and everybody’s using Docker and everybody’s using these programming languages that are all open. Like, if everything else is open source, you don’t want to be the thing that’s not right. And there’s only a few places where, like, where that hasn’t been true historically in technology where you could be different and still succeed.

[00:05:15] And you have to be so differentiated that. And I don’t think that we were that differentiated. You have to be so differentiated such that people are like, oh, actually I want that thing so bad, I don’t care that it’s closed.

Ryan:

[00:05:27] So at the time, just thinking what was the competitive landscape? I guess if I, if I remember it was, I mean, AWS was dominant, they were there first and doing very well. GCP at that time, probably an up and coming company or I guess offering. And so my understanding then that the idea is let’s pull market share away by giving kind of open source distributing Kubernetes to more and more developers and then they’ll be more open to kind of migrating or using GCP because you’ll make a.

Brendan:

[00:06:00] They’ll pay attention, right? They’ll pay attention. And also like you change the dialogue too, right? Like tail light chasing is hard, right? Like if, if someone else has built VMs and everybody’s using VMs and all you’re doing is saying like, well, we’re building the same thing that they have, only maybe it’s a little bit better because we do something else over here or you know, whatever. Like that’s a hard market strategy to articulate.

[00:06:22] But if you create a brand new playing field where you are the thought leader, suddenly people are listening to you. Even if they’re not using it on your platform, they’re listening to you. And that gives you way more voice. You get a lot more voice in the market. It changes the narrative, it changes who people are listening to. And so that control of the story is an important aspect of I think, how you break through that dynamic or you try to break through that dynamic.

[00:06:51] Obviously they’re still in third at this point. So didn’t work out, but I mean, I wouldn’t say it didn’t work out. I think it worked out in general, but it’s still hard. Overcoming those kinds of market dynamics is hard. And I think the other thing that happened is everybody in the cloud consolidated around it. And so now Kubernetes is just sort of a utility everywhere.

Ryan:

[00:07:12] That perception benefit of being the dominant offering, which, I mean if you, if you look at what happened in hindsight, it makes a lot of sense. I mean that is what happened and it’s, it’s a wonderful benefit. But I guess when you were looking forward and you Were talking to leadership. Were you cognizant of those benefits and saying we need to do this because we’re going to kind of become the dominant offering and it’s going to have all of these optics benefits?

Brendan:

[00:07:44] I mean, I think we absolutely wanted to make sure that we were front and center in terms of thought leadership. And we definitely articulated that. Being in a thought leadership position is valuable.

Ryan:

[00:07:56] It’s interesting because I feel like it’s hard to quantify. I mean, if I was in that meeting and we’re trying to make a call, how much is that worth?

Brendan:

[00:08:06] Yeah, well, I think you have to also realize though, at the time it was pretty cheap, right? It was like eight or nine engineers and we kind of. And this is in some sense, I mean, it’s both a blessing and a curse. Like, part of the reason why we articulated and argued for having such a distinct brand, where the Kubernetes brand was separated from the Google brand, was that it kind of gave us freedom to fail also.

[00:08:29] It was like, hey, if these eight people go off and do something and it turns out to be stupid, like we’ll just kill it off. And it won’t have, like, it won’t, you know, it won’t, it won’t damage the broader perception of the cloud. And so I think there’s that benefit of the open source part of it too. It helped with adoption, right? Like, it helped us and especially as we went to like the Linux foundation and things like that and truly established like an independent entity, it helped ensure that, you know, people like Red Hat or Azure or AWS could take a bet on Kubernetes and feel confident in that bet.

[00:09:05] But it also was an insurance policy against failure. And also, to be honest, it simplified a lot of things too, because we were competing against startups at the time. Docker is a startup. They can be way more agile than a big company. And so by virtue of being a separate entity, we could be a little bit more agile also.

00:09:26 — Building the MVP

Ryan:

[00:09:26] The earliest conception of this project was you and two others kind of hacking something together.

Brendan:

[00:09:32] Yeah, it was a demo. I mean, it was sort of a demo almost. It was like, look at what we can do if we just smash a bunch of existing open source tech together.

Ryan:

[00:09:39] What did that demo do?

Brendan:

[00:09:42] I mean, it was basically like a basic cube control. It was like, hey, here’s a container I built. I mean, at the time you had to explain Docker to people. You were like, hey, here’s Docker. I used it to like build this Container image and then you could run it and deploy it and see that it had gotten distributed across a bunch of machines and that you could load balance to it because you’d hit a single endpoint and it would go, I’m replica one.

[00:10:06] And then you’d hit reload and go, I’m replica three. So it showed that it was replicated and then basic health checking so if you killed it, it would come back. And a V1 to V2 upgrade, that was about it.

Ryan:

[00:10:21] How long did that initial MVP take to build?

Brendan: