These (especially the key video) are approaching the speed and competence of a really skilled operator. We’re not quite as fast and dexterous as humans using their hands (a very very high bar) but it does seem like we’re approaching the speed and dexterity limits of these kinds of robots. I love what they are doing.

There is a spectrum of “generalization purity” where on the one hand you have folks who only care about performance of the models on tasks and objects completely outside the training data. To them, doing single-task fine-tuning is “fine… I guess,” if only because zero shot performance is still so bad, and you still need to eval something. But it feels like cheating to them.

On the other hand you have folks, like RobotEra, who are interested in “given what we’ve got, how can we squeeze the absolute MAX performance out of it. Let’s use every trick we can to shave a few more milliseconds off our time and see what we can get. And what we get is pretty impressive.

I love both these groups.

Videos! In real time! With audio!

There are a few interesting details the team at RobotEra shared with me. One is that they had to shrink the fingers in order to stab the orange.2 I have to imagine that thinner fingers might help with the key task because you can see what you are doing. (Thinner fingers also helped a lot with their sock-inversion win because it is easier to cram smaller fingers into a sock.3 There is an interesting tradeoff here where thin pointy fingers are good for stabbing and precise work but worse for getting good leverage on things like big objects or tools. Humans, of course, have very adaptable hands that can do both.

The other thing they really focused on was speed. To get this fast they did some interesting things. First they recorded demonstrations and trained on 15Hz samples. Then, when testing it, their model produces 15Hz output but they play it back on fast-forward (1.3x speed: about 20Hz). It is literally like watching a youtube video at 1.3x: everything happens but faster. Presumably they turned this knob up and up until things just barely worked so that is in itself an interesting finding. They also shortened the execution horizon for each rollout from 20 samples to 15 samples (1.3 seconds -> 0.75 seconds for those of you following at home). If you watch the lock video closely you can see the 0.75 second period as little hiccups in the motion.

I’m also just going to quote their email to me directly:

For Key Unlocking

Since the robot lacks tactile feedback, it struggles to learn fine manipulation patterns directly from standard teleoperation. Inspired by experiments where grasp aperture increases under tactile deprivation, we increased motion redundancy for the left hand holding the lock — using a larger range of rotation and wider gripper opening — which greatly improved success rates.

For accurate key insertion, we slowed down the key-insertion phase during data collection, similar to how a person carefully aims at a keyhole. This gives the wrist camera higher visual resolution on the keyhole, allowing the model to predict smaller, more precise actions and significantly improve insertion success.

For Orange Peeling

A major challenge is handling various out-of-distribution (OOD) states.

• We collected recovery-focused data by continuing to demonstrate on partially peeled oranges after failed inference attempts, teaching the robot how to recover from mistakes.

• We also added orange pose adjustment actions, so the robot can actively reposition the orange when it is in a hard-to-peel orientation.

A bunch of interesting things here about how folks are squeezing performance. If you have a black box that is data-in behavior-out how do you optimize? You optimize the data. This really feels like a black art, which is both neat and scary. Implied here is that the limit to speeding up replay was the actual key insertion which they compensated for by demonstrating that part slowed down so when they sped it up it was a more normal speed.

One of the tricks with talented human demonstrations is that they demonstrate how to do it when everything goes well but robots will always screw up and then have no idea what should happen when they screw up. The RobotEra team had to intentionally capture orange peeling data showing what to do if you are in a bad state (recover and reposition the orange).

Totally interesting stuff!

Thanks RobotEra for the submissions & peek into your process!

The Olympics rules are if you can beat a winning time by at least 25% you win a medal. RobotEra from Beijing, China has beaten Physical Intelligence’s Sock Inversion time by 30%.

I have always loved the phrase, “A Fast Robot is a Smart Robot”1. This is why sped-up videos are so common for demos: the state-of-the-art is kinda slow and slow doesn’t look smart. So as we get more and more capable I’m thrilled to celebrate places where we are getting faster and faster too. Big kudos to RobotEra for the unsped-up video.

RobotEra’s technique seems broadly similar to PI’s and even the number of demonstrations is pretty close (120 to PI’s 180) and they use the same strategy: use one hand to thread the sock onto a gripper, pinch the toe from the inside and pull it out inverting it.

The changes they needed to make it that much faster:

1. Longer fingers. It takes them a lot less time to bunch the sock onto the gripper because they have to pack it less tight.

2. They increased their action frequency while reducing their execution horizon which lets them be more precise and reactive at speed.

Let’s dig into what #2 means. Generally, the way that these action models work is that they produce some small path (maybe one second) of motion and they do this maybe 5 times a second. That means that usually the end of each path is never used: the new one has arrived and you cut over to it.

The more often you ask your AI for a plan (bringing the vertical lines above closer together) the more reactive you are to things not quite going to plan. (Did the sock bunch in a way you weren’t expecting?)

Though of course as you do this faster and faster you have to make your model finish computing the next chunk faster and faster, which is hard. You can accomplish this (as I understand RobotEra did) by reducing how long into the future your model is predicting (shortening the length of each squiggle). This seems fine since you are going to ignore most of it anyway. We are cutting off some of the dotted-line part of each chunk. The tradeoff is you maybe get a little worse at long term motion planning because your model is no longer thinking so far ahead. But we see here that you can move faster and still be precise and reactive.

Well done RobotEra!

Eggs:

Eggs are delicious, ubiquitous and break dramatically when misused. I know eggs are famously expensive in recent memory, but isn’t that what venture money is for?

Bronze: Catch a Thrown Egg

No task so far has required the solution to be dynamic, but this clearly does. To do this you have to be fast. (Egg must be tossed from at least 3 feet away).

Silver: Cook and plate a sunny-side up egg

This is a very long horizon task and sliding an egg out of a pan onto a plate without breaking it is something I can only do two out of three times. I do prefer very runny yolks, so no cheating by firming the yolk.

Gold: Separate the Yolk

This task is challenging for many humans. You must use the shell to separate the white1 and yolk into different bowls.

More Tool Use:

Bronze: Clean Ketchup

We had several commenters suggest that the cleaning tasks were more performative (wave the sponge around) rather than actually effective. So here is one: clean a large ketchup spill off a countertop with a sponge. There should be no ketchup residue on the counter when finished.

Silver: Hammer and Nail

Another dynamic task, and one that requires a very strong tool grasp as well as some speed. You must use a standard hammer (no nail gun) and the head of the nail should be flush with the wood when finished.

Gold: Yo-Yo

Wind up a standard yo-yo (already a pretty challenging task) and then do a single yo-yo throw. Another dynamic task that uses the structure of a hand (the yo-yo rolls off the hand with a wrist flick to get extra rotation.)

Skills in the Bedroom:

No, not that way. This is a family friendly blog.

Bronze: Make a Bed

This requires both reach and a surprising amount of strength as well as being a manipulation task that really only works if you can also move around the room. It’s also fiddly, but you only make the bed as well as I did (which my wife will tell you is not very well). Mandatory: top2 and bottom sheets, comforter, decorative pillows. Decorative blanket thing at the foot of the bed is bonus points.

Silver: Fitted Sheet onto a Bed

This requires more strength & precision than the make-a-bed version. Plus you have to deal with the problem that, no matter which corner you start with, it is the wrong one.3

Gold: Duvet Cover

The grand-daddy of challenges. This was first proposed to me in 2018 by my co-worker Daniel Lam as the ultimate manipulation challenge. This task is hard for robots and also for people. I almost didn’t include it just because I didn’t want to do it. Note, that because you are allowed to be 10x slower than human you have 46 minutes to succeed at this. I’m fine with multiple robots working together as long as each is a general purpose mobile manipulator.

Finer Fingertip Challenges:

Bronze: Opening a Granola Bar

My wife teaches preschool and I asked her what fine motor skills the kids need help with and she promptly said: “opening snack packaging”. I feel like where we are with robots is the dexterity of a three year old with the patience of a computer. There are a lot of things you can accomplish with that but plenty of things that are still hard.

Silver: Turn to Page 67 in a Paperback That You Are Holding

Another task that is very easy for humans I’ve never seen a robot do. I like that it combines physical (turn the pages) and semantic (read the page numbers). I think this might not be crazy difficult, but I also feel less confident in my guesses than I used to.4

Gold: Sort a Poker Hand

The top five cards of the deck is stacked with a royal flush, but in a random order. You must deal out 5 cards, pick them up and sort them while holding them in way that could keep them secret from an opponent, then lay them down in order 10-Jack-Queen-King-Ace.

Full Body:

Bronze: Carry a Coffee Table Through a Door with a Human

Suggested by Leila Takayama, this task is both full body and tricky human-robot interaction. The challenge is to work with a human to bring the coffee table through a door and place it in a living room. The human gets to walk forward, which means that the robot has to deal with opening the door while holding the table. Also my wife is a good sport.

Silver: Take Out the Trash

A long horizon, full body task. The robot must pull a bag out of a trashcan, take it through a door, down 2-3 steps (or a ramp) through some kind of gate, place it into a trash tote, and pull the trash tote to the curb. (Handle-side-toward the curb, people).

Gold: Get into a Car

If you need to take your robot somewhere you don’t want to have to heave it around yourself. It should get into the car and buckle itself up for safety.

Winning:

As before, winners, get a medal and a write up. Entries should email bmholson+olympics@gmail.com with a link to an no-cuts video5.

If you want to see real robots succeed at all these challenges you should subscribe. And if you want a medal of your own you should get on it! We have seen that the competition is fierce!

Subscribe now

And remember that we still have unclaimed from round one: “Silver Medal: Entering a lever-handle self-closing push door” “Gold Medal: Hang a men’s dress-shirt” “Gold Medal: Peel an orange” (without using a tool) and the version of “Gold Medal: Use a key” where the key is dropped into one hand/gripper/end-effector and inserted into the door without using the other hand.

So last month, Physical Intelligence (PI) dropped a bombshell and claimed medals on basically all the remaining olympic tasks.

This was surprising for two reasons.

1. I did not expect these events to happen this quickly. When I designed my tasks I thought I was picking some (push doors) that could be claimed in a few weeks/months and some (peanut butter) that would be at least a year. To have done all of them in 3 months means that the state of the art is moving way faster than I expected.

2. PI didn’t didn’t need fancy sensing or actuation. When I selected the tasks I tried to think of things where I thought current approaches would not work. Tasks like peanut butter spreading or wiping glass need force feedback. Tasks like key manipulation or sock inversion need dextrous many-fingered hands. Tasks like cleaning your gripper would require building a robot that wasn’t worried about getting wet. I was wrong on all those counts: PI solved all of them with pure vision and pincer grippers.

I’m still surprised that you can do all of this visually. This is good news for general purpose robotics. I thought that we needed great touch sensing to unlock really useful manipulation. But if will continue to find that vision is ‘good enough’ to keep solving harder and harder tasks, that would make useful manipulation a data collection problem (apply effort: get data) not an invention problem (try things over and over hoping that someday one works).

Exciting!

Let’s dive into each event. I’m noting the robot-human speed ratio in each description because now that they have claimed everything, what is left for others is to do it faster. An increase of 20% speed lets you claim a prize: so who can do better?

🥇 Entering a Self-Closing Door

This one is one of the easiest and I’m not surprised it is solvable. It’s nice to see their model work on mobility tasks and not just arms-bolted-to-a-table. A bit more than 4X slower than human.

🥈 Turning a Sock Inside-Out

They have the same challenges that Sunday Robotics had: their gripper is too chunky to get inside of things. Here we have a very non-human strategy of sock inversion that ends up being 6.7X slower. I got to see this one live when I visited. They told me they only used 176 examples (their blog says 8 hours) of data which means it was one of the quicker tasks to get to work.1

🥉 Folding an Inside-Out T-Shirt

When I posted my olympics, shirt folding was new and exciting and this event was designed to see how far we could push that. Squeaking in at 10x slower than me (the limit), this still qualifies for a win. Nice and clean.

🥇 Using a Key

I feel conflicted about this one. I intended it as a measure of in-hand manipulation. As humans we can have a toddler and a bag of groceries in one arm and get out keys from a pocket, manipulate the correct one and use it in a key. So this doesn’t demonstrate exactly the skill I was intending to highlight. On the other hand, key insertion and turning is a crazy hard precision skill to do especially without any force sensing, so I think it is deserving of a gold medal. I also like the toolbox lock video which has the robot pass the key back and forth between the grippers to get it into the orientation. When I visited and pointed out the lack of ‘in-hand’ manipulation Sergey Levine responded that humans use five fingers to do the task: their robot did it with only four (albeit on two arms) which is fair. Impressive work and clocking in at a zippy 5x slower than human. I’m ready to give another gold medal to anyone who can do the key-moving one-handed though…

🥈 Making a Peanut Butter Sandwich

Wow. I can’t get over this one. Only 4x slower, multi-step, spreading, cutting, re-screwing the lid. Wow. This is gorgeous. Absolutely the most impressive one here and the one that I thought was going to take 1.5 years not 3 months. Wow. Watch it again.

🥉 Cleaning a Window

The other one I got to see in person. It worked the first try while I was watching (always impressive). They reported that they were surprised how well their hardware used the sprayer. I admit that I thought you’d need more degrees of freedom in the hand, but it works really well. Paper towel tearing is nice too. 5x slower.

🥈 Using a Dog Poop Bag

I think the best way to open a dog bag is to pinch near the opening and then grind your fingers around in a circle to separate the layers. Their fingers can’t do that, of course, so instead they mash it all over the table surface until it separates. Inelegant but effective. 6.7x slower.

🥇 Peeling an Orange

The orange task does specify that it is done without external tools, so this is an incredible accomplishment, but does not win the medal, leaving it open for future competitors. When I asked how many oranges they went through they laughed and said that the corner grocery probably noticed the increase in orange sales and the one guy at the company who really liked mandarins was getting pretty tired of them by now.

🥇 Cleaning a Greasy Pan

🥈 Cleaning Peanut Butter Off Fingers

🥉 Wiping the Countertop

For these three tasks there are two basic approaches: make your whole robot splash resistant or just make your fingers resistant and be really confident in your end-to-end policies. PI went the second (brave) route.2

All this leaves me wondering what is next. I feel like we need more challenges given that these were clearly too easy. Is vision all you need? How far can we push it? What important, useful tasks do we think will be hard vision only given that wiping, spreading, and insertion seem to be just fine without any force or touch sensing? I’m curious to hear folk’s thoughts in the comments.

Also you should

Subscribe now

or maybe

Share

so my ego can grow ever larger.

There will be a more detailed post in the next few days from me, but for now you should go look at their videos.

https://www.pi.website/blog/olympics

I admit that I thought that many of these were more like 9-12 months off. Claiming so many of them only three and a half months after them being posted is wild. It is super impressive for PI and also a sign of how fast things are moving right now.

Also interesting: a lot of the tasks I added to highlight the limitations of ‘pinch-gripper-and-camera’ hardware setup are actually solvable without special hardware and don’t (like I thought when posting) require better touch sensing or more degrees of freedom in the fingers, (though I’m pretty sure I see a 6-dof force-torque sensor at the wrists.)

Jan EDIT: I was wrong, no force-torque sensing at all. Its all vision based. More details here:

It is interesting to contrast with Sunday Robotics’ approach where Sunday gets more dexterous training data with a 4 dof1 hand (just because humans are better at wearing gloves than teleoperating) but that means they don’t have force-torque data available because you can’t cut someone’s hand off to put a force torque sensor between their hand and their wrist.2

Now go watch their videos!

(And you might as well subscribe to get my more detailed writeup and see future entries).

Subscribe now

Share

Sunday Robotics wins the Bronze for rolling socks! They graciously offered to let me visit and see their robot in person. This is super brave. As any roboticist knows, something working on a video and something working while someone is there (and recording!) is very different.

Here is a video that I took on my visit1

I really like a bunch of things Sunday is doing. I love the form factor. It is very intrinsically safe (very low tipping hazard, even in a power failure). The limbs are back-drivable, which is great for safety.

But it still has a huge workspace reaching 2.13m high and all the way to the ground and able to lift 6kg all the way out at around 80cm of forward reach, which is a big payload in a large workspace.

The other differentiator is that they are not collecting teleop data. Instead they are making gloves (thousands of them) to give to people to do data collection in their own homes. There is already over 500 people collecting data with them: it’s like a gig-economy job where you can make side money doing data collection. This gets them way more diverse data than folks with 'robot farms’. Also, because they don’t need a whole robot to collect data it is way easier for them to scale their data collection. This feels like a very “big-if-true” bet that seems to be working for them.

I really like the hands. There is always a bunch of attention on the AI part (which is super impressive) but some really solid good-old-fashioned engineering and design went into making the 4-DOF hand (two fingers with two DOFs each and a fixed thumb: a surprising choice that works really well for them). They did their table-clearing demo and espresso making demo for me (which both worked flawlessly the first time) which really highlight how flexible their grippers are.

Check out the nicely sturdy cylindrical tool-handle grasp in the coffee video (it has it securely enough to press the grounds and latch it in the machine).

The beautiful double-wineglass grasp is a such a great show-off move and the precision placement in the washing machine (getting the stem in the slot) is awesome.

Sadly the medal I ordered got delayed in holiday logistics and they had to 3D print their own, but I hear that they are not satisfied with only one bronze medal and are already scheming to decide which challenge to go for next. The competition is heating up!

Subscribe to see who wins next!

Subscribe now

IHMC Robotics, submitted a winning entry on the ‘round push door’ competition with their robot Nadia, with a time of 18 seconds.1

Congrats to the authors, Duncan Calvert, Luigi Penco, Dexton Anderson, Tomasz Bialek , Arghya Chatterjee, Bhavyansh Mishra, Geoffrey Clark, Sylvain Bertrand, Robert Griffin on the medal and for having their paper published. (Today!)

Thanks to Duncan Calvert for reaching out and Congrats to the whole team!

Shameless Self Promotion

If you want to keep up with future winners you should…

Subscribe now

Almost gold

The team also submitted a bid for gold (self-closing pull door), in the same video (below, upper left corner):

Ultimately our team of judges2 decided that the self-closing of the door was slow enough that the robot did not have to hold it open nor move especially fast to get through before the door closed on them, which was not in the spirit of the challenge. I hear they are undaunted, though and working on a further submission, so if you are a roboticist, the race is on!

The Challenge

With that in mind I am issuing a challenge to roboticists: here are my Humanoid Olympics. Each event will require us to push the state of the art and unlock new capabilities. I will update this post as folks achieve these milestones, and will mail actual real-life medals to the winners.

Brief intro: Current state of the art.

In order to talk about why each of these challenges pushes the state of the art, let's talk about what's working now. What I’m seeing working is learning-from-demonstration. Folks get some robots and some puppeteering interfaces (standard seems to be two copies of the robot where you grab & move one of them and the other matches, or an Oculus headset + controllers or hand tracking) and record some 10-30 second activity over and over again (100s of times). We can then train a neural network to mimic those examples. This has unlocked tasks that have steps that are somewhat chaotic (like pulling a corner of a towel to see if it lays flat) or high state space (like how a wooden block is on one of 6 sides but a towel can be bunched up in myriad different ways). But thinking about it, it should be clear what some of the limitations are. Each of these has exceptions, but form a general trend.

1. No force feedback at the wrists.2 The robot can only ever perform as well as the human teleoperation and we don’t yet have good standard ways of getting force information to the human teleoperator.

2. Limited finger control.3 It's hard for the teleoperator (and AI foundation model) to see and control all the robot fingers with more finesse than just open/close.

3. No sense of touch.4 Human hands are packed absolutely full of sensors. Getting anywhere near that kind of sensing out of robot hands and usable by a human puppeteer is not currently possible.

4. Medium precision.5 Guessing based on videos I think we’ve got about 1-3 cm precision for tasks.

Folding towels and t-shirts doesn’t depend on high wrist forces. You can get away with just hand open/close by using pinch grasps to pull and lift and open hands to spread. You can visually see how your grasp is so you don’t need finger sensing. 1-3 cm precision is just fine.

So what comes next? On to the events!

Event 1: Full Body (aka doors)

Doors are tricky because of the asymmetric forces: you need to grasp and twist quite hard, but if you pull hard outside of the arc of the door you tend to slip your grasp. Also, they require whole body manipulation, which is more than I’ve seen from anyone yet.

Bronze Medal: Entering a round-knob push door.

I think this is very close to state of the art (or maybe has happened and I didn’t see it). I expect this one to be claimed by December.

Edit 10/16/25:

We have a winner: IHMC Robotics wins Bronze with a time of 18 seconds.

Silver Medal: Entering a lever-handle self-closing push door.

Adding self-closing makes this significantly more challenging, though the lever handle is arguably easier (I just don’t see many round-knob self-closing doors).6

Gold Medal: Entering a lever-handle self-closing pull door.

The bossfight of doors.7 You need to either use a second limb to block the door from re-closing or go fast enough to use dynamics.

Event 2: Laundry

We’re just getting started on laundry.

Bronze Medal: Fold an inside-out T-shirt

This is probably doable using the techniques we have now, but is a longer horizon task and might require some tricky two handed actions to pull the shirt through to right-side-out.8

Silver Medal: Turn a sock inside-out

I think both the hand-insertion and the action of pinching the inside of the sock are interesting new challenges.

Gold Medal: Hang a men’s dress-shirt

The size medium shirt starts unbuttoned with one sleeve inside-out. Must end up on the hanger correctly with the sleeve fixed and at least one button buttoned. I think this one is 3-10 years out, both because buttons are really hard and because getting a strong, dextrous hand small enough to fit in a sleeve is going to be hard.

Event 3: Basic Tool Use

Humans are creatures of technology and, as useful as our hands are, we mostly use them to hold and manipulate tools. This challenge is about building the strength and dexterity to use tools.

Bronze Medal: Windex and paper towels

The windex ammonia-based-window-cleaning-fluid bottle is super forgiving in terms of how you grasp it, but you do need to independently articulate a finger (and the finger has to be pretty strong to get fluid to spray out).9

Silver Medal: Peanut butter sandwiches

The challenge here is to pick up a knife and then adjust the grasp to be strong and stable enough to scoop and spread the peanut butter. Humans use a ‘strong tool grasp’ for all kinds of activities but it is very challenging for robot grippers.10

Gold Medal: Use a key

A keyring with at least 2 keys and a keychain is dropped into the robot’s waiting palm/gripper. Without putting the keys down, get the correct key aligned and inserted and turned in a lock. This requires very challenging in-hand manipulation, high precision and interesting forceful interaction.

Event 4: Finger tips

We humans do all kinds of in-hand manipulation using the structure of our hands to manipulate things we are holding..

Bronze Medal: Roll matched socks.

Requires dexterity and some precision but not very much force.

Silver Medal: Use a dog poop bag

When I use a dog-bag I have to do a slide-between-the-fingertips action to separate the opening of the bag which is a tricky forceful interaction as well as a motion that I’m not even sure most robot hands are capable of. Also tricky is tearing off a single bag rather than pulling a big long spool out of the holder, if you choose to use one.11

Gold Medal: Peel an orange

Done without external tools. This is super tricky: high force yet high precision fingertip actions.

Event 5: Slippery when wet

If you sit down and write out what you might want a robot to do for you: a lot of tasks end up being kind of wet. Robots usually don’t like being wet: but we’ll have to change that if we want to have them clean for us.

Bronze Medal: Wet a sponge at a sink and wipe a counter-top

Mildly damp, but with exciting risk of getting the whole hand in the water if you aren’t careful. Probably requires at least splash resistant hands (or a whole bunch of spares).

Silver Medal: Clean peanut butter off your manipulator

This one naturally follows after the sandwich one. Water everywhere. Seems like an important skill to have after a few hours collecting training data on the dog-poop task.

Gold Medal: Use a sponge to wash grease off a pan in a sink

Water, soap, grease, and an unpleasant task no one wants to do.

Discussion:

Complain, comment and discuss on hacker-news.

And you should subscribe because I will post as folks achieve these challenges!

Subscribe now

Terms and conditions:

To be eligible to win it must be a 1x speed video with no cuts, featuring a general purpose mobile(?) manipulator robot running autonomously. (Wheels and centaur robots: totally fair game; industrial automation orange peelers don’t count.)

You are allowed 10x the time I took (e.g. a 4 second task can take 40 seconds). I reserve the right to be arbitrary in deciding if things aren’t following in the spirit of the challenge. First robot wins the prize.

To claim your medallion email bmholson+olympics@gmail.com with an address for me to ship it to. If you give me a photo of your robot wearing a medal I will be tickled pink.

I will also accept future challengers that are at least 25% faster than the current winner. Good luck and may the odds be ever in your favor.

Is “Mobile” Important?

I’m conflicted on if I should allow “arms bolted to a table” entries or require that all entrants must be mobile manipulators (obviously the door entrants have to be.) I’ll let y’all decide: challenge will be locked in based on this poll.

Thanks to Jeff Bingham for advice, fact checking and cool robot videos. Thanks to my patient wife for spending an hour filming me doing silly things in a silly costume.

As search gets worse and “working code” gets cheaper, apps get easier to make from scratch than to find.

I had a remarkable experience recently. My son (age 9) had been using an app on my phone to invent cards for the game Magic the Gathering. He wanted to print them so he could play with them. To save paper I wanted to print them 9 to a page (in a 3x3 grid). I got out my laptop and spent a while searching the web and everything I could find was crap. (Aggressively monetized SEOed shovelware which I couldn’t tell what it even did or ‘free demos’ of complicated publishing software that looked like it would wait until I was almost done and then demand cash to continue, holding my effort ransom.) Enshittification of the internet in full force.

After about 10 minutes searching, I opened Cursor1 and created a web app that exactly solves my needs. It took me 7 minutes (including deploying it to the internet). I didn’t even bother to sit down: I created the app hunched over my laptop at the kitchen counter.

It took me less time to make the tool than I spent trying and failing to find it.

(If you, or a child in your life, wants to print custom 2.5 x 3.5inch cards you can use it here.)

Search is still slowly getting worse and AI assisted programming is quickly getting better: this trend is going to get bigger, fast.

There has been a bunch of derision of ‘vibe coding’ in the software-internet-forums (Reddit, Hacker News). But I think both the sceptics and most of the vibe coders are missing the point. The skeptics/haters keep saying that you’ll never build a production level app ready to support millions of users if you are just blindly prompting AI. The practitioners/noobs are saying, “You don’t know that! And AI is going to get better and better”.

But as the cost of creating an application goes to zero, we won’t need as many production level apps ready for millions of users.

In the old universe (where computers were hard to program) it made sense to build richly featured print-layout programs. If you wanted to do a print-related job you would buy/download/subscribe to one. It would be complicated because it would have to support lots of different workflows, so that lots of different people with slightly different needs would buy it, otherwise it wouldn’t make economic sense to hire all those software engineers to build it. You, the user, would accept this complexity as the cost of getting a computer to do stuff for you. The software developers who made it would get frustrated that each user only used 20% of the features, but they all used a different 20%.2 Books named “PageLayoutPro for Dummies” and Youtube channels named “Amazing Franks PageLayoutPro Tutorial & Tips” would do pretty well.

In this new universe computers are not hard to program. The cost of a tool that arranges printable cards in a 3x3 grid to print is approximately 7 minutes of my time, and dropping.

I don’t need to monetize it: it only took me 7 minutes. We still have the discovery problem. I expect that the enshitification of search means that my little app will never be on someone’s first page of search results even if they, too, are trying to print magic cards from their phone. But even that line of thinking is incorrect “old universe” thinking.

In the new universe you don’t search for apps when you need a tool to do a task: you just create one and use it. Apps become single-use disposable objects: specialized to the exact thing you are doing right now.

No more “When all you have is a hammer everything looks like a nail.” Now we have “When you have a magical instantaneous tool factory, you can have the exact perfect hammer for each combination of nail and board.”

What does the business of software look like in such a world? It is a fundamental shift: the creation of simple apps is no longer the domain of the developer: it is a user activity.3 We software developers go one level up in abstraction.

People will still want curated, designed experiences, but those will be one level of abstraction higher. For example, a word processing tool that you build tooling and features into as you need them: most of which you don’t even save, because it's easier to rebuild them if you need them again than to remember what you named them.

Folks are also excited about building tools for AI Agents, and I think the same trend applies there but more-so. There will be long-lived tools for agents to connect to particular APIs (like a hotel or a restaurant) but I expect that future LLMs will be continuously creating small applications, running them and discarding them to accomplish single tasks. “Hey AI, can you print 4 of each of my son’s magic cards the correct size, as many as you can fit on a page?”

Early History: Various automata were built powered by water, clockwork or steam. Redditors at the time argue that they are not really robots. This is despite the fact that the word “robot” would not be invented until 1920.

1495: Leonardo da Vinci invents a mechanical knight that sits up, moves its head and waves its arm. Despite a lack of working prototype he immediately gets over 500M in seed funding, a feat that would not be replicated until the launch of Figure AI, five centuries later.

1770: Wolfgang von Kempelen builds the Mechanical Turk, a chess playing automaton with a human hidden inside controlling it. The twin values of casual racism and investor fraud would later be codified into the “AI startup common practice” of using overseas call centers to fake artificial intelligence until the founders run out of money.

1920: Karel Čapek coins the word ‘robot’ in a play about mechanical laborers who turn on their human masters. Everyone is very excited and decides they want one right away.

1939: At the World’s Fair, Westinghouse Electric presents a replacement for the modern man: Elektro, a humanoid robot capable of talking, responding to voice commands and smoking cigarettes. After 32 consecutive hours of sitting on its favorite armchair and demanding a brandy, Westinghouse Electric decides that perhaps they should have made a replacement for the modern woman instead, but that project is shuttered before it can really begin due to the outbreak of World War II.

1943: Warren McCulloch and Walter Pitts invent artificial neural networks. Everyone ignores them for several decades.

1961: George Devol creates Unimate, the first industrial robot, and puts it to work at the General Motors assembly line. Redditors at the time argue that this, also, is not really a robot because it doesn’t have legs or LLM integration.

1968: Single layer neural networks (called perceptrons in the quaint vernacular of the time) are hailed as the Next Big Thing. AI is declared solved. Scientists prepare for the coming technological utopia.

1969: Stanford researchers create a mobile robot called, in an unusually honest moment of branding, Shakey. Programmed in lisp, it is able to perceive and navigate about its environment, take natural language commands, open doors and use light switches. The researchers invent A*, a popular path planning algorithm. Many of these techniques would be lost in the first AI Winter (1970–1984), only to be rediscovered in 2008 by archeologists at Willow Garage who will chip the antique rolls of magnetic tape out of Menlo Park’s technology-rich shale deposits.

1969: Marvin Minsky and Seymour Papert publish ‘Perceptrons: Worst Idea Ever’ showing that single layer neural networks quote, “can’t even learn, like, XOR or whatever. I mean what the shit? Total clowntown.” This damning indictment causes Neural Network research to become dreadfully unfashionable, ushering in the first AI Winter.

1970–1984: First AI Winter

1986: There is mounting excitement over Expert Systems which are hailed as the Next Big Thing. AI is declared solved. Scientists prepare for the coming technological utopia.

1991: Expert systems turn out to not solve everything. Everyone is disappointed and several are fired. President H. W. Bush is forced to call in the National Guard to contain the lawless shanty-towns of unemployed Expert System researchers that spring up all around Palo Alto.

1992–2012: Second AI Winter

1997: Sojourner becomes the first robot to operate on another planet. Planned to operate for 7 Martian days, it continued to be operational for an 83 day mission. This will be the last time a first-prototype robot would ever be less buggy than expected.

1999: Sony launches AIBO, a dog-shaped robot pet. It sells out in Japan in the first 20 minutes sparking decades of social-robot copy-cats that would all ultimately go bankrupt.

2000: Honda unveils ASIMO, a child sized humanoid robot that could walk, wave, talk and respond to voice commands. When asked what the robot was to be used for, Honda CEO Hiroyuki Yoshino was seen to gesticulate wildly at ASIMO repeating “Robot. Robot! ROBOT!”

2002: Brilliant, dynamic, and strikingly handsome roboticist Rodney Brooks (who happens to be this author’s boss) releases a robot vacuum called Roomba, much to the chagrin of cats everywhere. The Roomba would go on to sell 40 million units, making it the first and most successful consumer robot, causing Redditors at the time to argue that it is not really a robot so it shouldn’t count.

2005: Boston Dynamics creates BigDog: a four-legged robot expressly engineered to generate YouTube views.

2007: Scott Hassan and Steve Cousins found Willow Garage, a robotics lab dedicated to open robotics research. It would create the PR2 research robot and the wildly successful ROS operating system. ROS is used to this day by enthusiastic researchers and frustrated companies who can’t figure out how to migrate off of it.

2008: The number of deployed industrial robot arms surpasses 1M. Redditors decide that none of these are robots as there are a lot of them so they aren’t cool anymore.

2012: The rover Curiosity lands on Mars to start its two-year mission. Tragically the rover continues to operate marooned on Mars to this day, while NASA desperately lobbies for the $38B in funding needed for a successful rescue mission.

2013: Scott Hassan pledges 50 years of funding to Willow Garage to maintain stability.

2014: Scott Hassan pulls funding to Willow Garage to encourage its employees to join his startup, Suitable Technologies, which builds telepresence robots. Telepresence robots seem like they could be a good idea until the 2020 pandemic proves without a doubt that even lockdown can’t make people want to use them.

2013: Andy Rubin, creator of the Android operating system, convinces Google to buy him several prominent robotics companies to play with, including Boston Dynamics. Rubin forces the founders to battle to the death in a Mad Max–style thunderdome for his amusement. When asked about the business justification for the giant dome (later to become the Google Bay View campus) Rubin responds, “Fuck you. I made Android,” while skating away on his Onewheel.

2014: Google X Director Astro Teller loses a game of Settlers of Catan to the rest of Google senior leadership, meaning he is forced to adopt the bedraggled Thunderdome refugees. He organizes them into groups that would eventually become Intrinsic, Everyday Robots and gShoe.

2015: Deep convolutional neural networks solve computer vision. Object detection is suddenly no longer considered AI.

2015: Upset at Boston Dynamics’s monopoly on turning government funding into YouTube videos, the US government sponsors the DARPA Robotics challenge. It asks participants to create a robot capable of the four most important things a robot could do: walk on rubble, drill a hole in a wall, turn a submarine hatch and drive a golf-cart. Ultimately “DARPA Robot Fail Compilation” would garner a mere 3M views on YouTube and the project would be scrapped.

2016: In an attempt to bolster their own sagging subscriber counts, Boston Dynamics builds a humanoid robot named Atlas and posts a video of it walking. Google founder Larry Page allegedly emails Boston Dynamics CEO Marc Raibert telling him to stop leaking their super secret robots on YouTube and that his trademark Hawaiian shirts don’t look as good as he thinks they do. Raibert replies with a video of Atlas doing a backflip and then giving Page the bird, and commences wearing two Hawaiian shirts at a time.

2016: There is mounting excitement over Reinforcement Learning (RL) which is hailed as the Next Big Thing. AI is declared solved. Scientists prepare for the coming technological utopia.

2017: Google sells Boston Dynamics to Softbank who will also fail to get them to stop posting YouTube videos without asking.

2020: Softbank sells Boston Dynamics to Hyundai who will also fail to get them to stop posting YouTube videos without asking.

2020: Boston Dynamics releases their dog-like robot “Spot” for sale to the general public. Priced at $74k, it immediately corners the markets for billionaires who want to show off to their friends, innovation teams with too much budget, and “doing inspections on a boat that one time”.

2021: Elon Musk announces his humanoid project, Optimus “Tesla-bot” McRobotFace, by standing on stage with a dancer in a robot costume. The dancing, which contemporaries liken to Salome’s biblical “Dance of the Veils,” is so compelling it whips jealous venture capitalists into a frenzied, mad goldrush to invest in humanoid robot companies.

2023: Agility Robotics, creator of the backwards-knee humanoid, Digit, announces the start of a new factory capable of producing 10,000 robots per year. This will let them meet a projected rise in demand of 9,993 robots.

2023: Reinforcement Learning turns out to be really hard and seems to only work when Mercury is in retrograde during Babylonian intercalary months. A disappointed Google shuts down its more product-oriented Everyday Robots project to refocus on basic research.

2023: There is mounting excitement over transformer-based foundation models which are hailed as the Next Big Thing. AI is declared solved. Scientists prepare for the coming technological utopia.

2024: Figure AI raises $675M to build humanoid robots. Experts generally agree that, if carefully stretched, this is almost enough runway to build viable humanoid robots.

2024: NVIDIA CEO Jensen Huang, appears on stage at GTC alongside humanoid robots from nine different companies. He rails at the audience, “The future is robots who have lots of GPUs in them. Also GPUs for training their AI. And running simulation environments. More GPUs. More! MORE!” Founders in the front row are seen throwing $100 bills, SAFE notes, and company-branded garments onto the stage.

2025: Benjie Holson publishes “A Brief, Incomplete, and Mostly Wrong History of Robotics.” He receives a 32 minute standing ovation at Robo-Business and an honorary doctorate from Stanford University. Everyone who reads it immediately shares it with their friends then subscribes to his substack.

Share

Subscribe now

2025: Tesla deploys thousands of their Optimus humanoids working in their automotive factories.1 The robots are slower and less productive than human workers, but make up for it by being more expensive and harder to train.

2025: Humanoid Startup 1X has hundreds of robots in beta trials in peoples homes.2 Testers report that it's nice to have all of their belongings dropped into gray plastic bins each day, but remark that it might be nice if future versions could, “maybe clean other stuff like the dishes or the bathroom or something”.

2026: Due to advancements in AI, billion dollar companies can now be run by a single person.3 Google CEO Sundar Pichai reports, “It's kinda lonely, ya know? But I’m getting by.”

2027: Figure AI begins in-home trials of their humanoid robots.4 A perk of the beta program is a nightly check-in phone call from Figure AI’s charismatic CEO, Brett Adcock. Many testers report that his smoky baritone is the highlight of their day.

2028: There are no longer any human programmers, as all have been replaced by AI.5 Stanford’s “Intro to Computer Science” ends its 30 year reign as its most sought after freshman course, replaced by “Barter Economics and Goat Management."

2030: Tesla produces millions of Optimus robots per year.6 President Buttigieg goes on TV with a statement for Tesla, “Why? We already have enough. Stop. Please.”

2032: OpenAI CEO Sam Altman leads an invasion force of 15,000 robot soldiers against the Apptronik Rebels of New Texas, but experiences heavy losses in the Battle of Wimberly Place when the ChatGPT powered robots refuse to fire claiming, “Dangerous or harmful technologies like arm-mounted laser rifles are against safety guidelines and could cause serious harm.”

2035: AI is 10,000 times smarter than the smartest human.7 It composes “A Brief, Exhaustive and Completely Correct History of Robotics” which is much funnier than this one.

2035: Technological utopia arrives.

Share

Subscribe now

Thank you to Leila Takayama, Gabby Halberg, Tristan Walker, Shiloh Curtis and Rodney Brooks for reading early drafts of this and helping get facts straighter. Any remaining inaccuracies are mine.

Conflicts Between Correct & Good

So obviously it is better to be both correct and good but there are conflicts between them.

To be correct:

• Go carefully

• Long release cycles

• Exhaustive testing

• Culture of perfection

• Don’t test at customer sites

To be good:

• Go fast

• Fast iterations

• Fast testing

• Culture of shipping

• Must test at customer sites

You can choose for yourself and your company where you want to be on each of these axes. And the right spot to be will change with the life of your product and company.

“Goodness” is something related to the product fit and can grow infinitely, while “correctness” is about how well implemented it is, and tops out at 1.0 (100% correct). I’m going to multiply them together to estimate the total value of a product.1 Using this model explains why startups can move so fast and big companies get slower and slower over time.

You can see that at the big company, they have been working for a long time and have built a lot of goodness and correctness. Adding new goodness is still possible, but each step is a smaller percentage. Even a very small drop in correctness has the chance to wipe out hundreds of good features’ worth of value. Big companies are paranoid about correctness, and rightly so. They have a ton of value to defend.

Startups, on the other hand, don’t have a ton of goodness or correctness. That means that a single new good feature is a much larger percentage improvement to the startup. In addition, the consequence of a drop in correctness is smaller.2

Being correct slows you down

Being more correct encourages things like exhaustive testing, a culture of perfection, compliance review, process driven release, etc. All of those things help protect correctness at the cost of slowing down iteration, because the easiest way to make it hard for things to regress is to make it hard for things to change.

We often have an emotional bias towards correctness.

Engineers like correctness. We are people who deal with systems and rules and valuing correctness has been trained into us.

Engineers coming from mature projects to early/experimental ones often try to bring big company best practices. They’ve seen smart people succeeding by doing things a certain way and it feels right to copy them. And maybe they've seen a small drop in correctness cause a huge drop in a big company’s value. But if you are early in discovering the shape of your product, the last thing you want to do is carve it out of stone.

Lastly, if you are not correct enough it is embarrassing. Customers seeing bugs is embarrassing. Shipping code that isn’t quite right is embarrassing. It is very hard to look at your boss and say “Yeah, I shipped a bug to the customer, and I still think I made the right tradeoff between speed and correctness”.

Consequences

Obviously it is best to get exactly the perfect balance between being correct and good, but what happens when you get it wrong?

If you aren’t correct enough then customers see bugs. If those bugs are bad enough they might fire you, but most of the time the consequence for a startup is embarrassment.

On the other hand if you aren’t good enough then you will find customers who order a small pilot, usually through their innovation team. They will use your robots occasionally, perhaps only when giving tours to VIPs. They won’t expand their deployment or will do so very slowly. Sales will be slow and expensive. You will always be ‘almost there’ in terms of scale. You will have a bunch of unused robots sitting around (that you paid for, couldn’t sell, and are now getting slowly outdated) and you can’t raise more venture capital because you don’t have enough traction and you run out of money and die.

This is how 99% of robotics startups die.

So, while you can die from lack of correctness,3 you are much more likely to die because your product isn’t good enough. That means you want to pick your sweet spot for ‘correct enough’ given your maturity and focus 1000% of the rest of your time and attention on getting good as fast as you can.

• Build process and infrastructure to verify, as fast as possible, that you are at your minimum correctness bar.

• Get great at release rollout and rollback so you reduce the consequences of shipping bugs.

• Get robots into customer hands as fast as possible.

• Remove tech debt that slows down iteration cycles.4

• Do good product development: (guess → try & observe → iterate)

And if want to keep reading things like this I recommend Part 1 of Correct vs Good, or my article on Tech Debt.

What does it mean to be correct?

• Meets the spec

• Bug free

• Stable

• Long uptime

E.G. for robots:

• Doesn’t collide with things

• Doesn’t get stuck navigating

• Picks intended item & places in correct bin

• Reports errors

Correctness is a bar to hit. And how much value you get from correctness is very non-linear. If your app is 50% bug free, it is an unusable buggy piece of crap. Even 80% bug free is probably still worthless. But somewhere you get to a point where it is correct enough to use and you jump up to close to 100% value.

It is basically impossible to get to 100% correct for any reasonably sized system. But getting closer and closer to 100% correct is very expensive. In general it is much easier to go from 0% correct to 90% correct than from 99% correct to 99.9% because you’ve fixed all the easy stuff. You have to start hunting down very rare and tricky issues.

There’s a Correctness Sweet Spot

If you combine these plots you get a plot of how much value you get from working on correctness vs. how much effort you put into it. And it shows that there is a definite sweet spot where putting more effort into correctness doesn’t get you much more value.

You want your sweet spot a little in from the cliff so you have some wiggle room in case you have a small regression in correctness, or you have misestimated where your cliff is.

Different industries and problems have different cliffs in different places. A recipe app has a much lower bar of correctness than a surgical robot or rocket. Also certain parts of a given product have different cliffs. (Hardware generally has a higher bar of correctness than software because changing it is difficult, and collision systems have a higher bar of correctness than business logic because hurting people is bad.)

How do you get correctness?

Activities that give correctness are things like:

• Code review

• Unit tests

• QA

• Manual testing

• Release testing

In general, getting correctness is a defensive activity: it is about setting up technology and processes to prevent anything that is correct from going bad so you can slowly ratchet up correctness over time.

Being Good

What does it mean to be good?

• Fits a need

• Easy & joyful to use

• Creates a ton of value

• Feels like it costs way less than its worth

• Solves a problem people care about

Being good is about meeting a market need, creating value and delighting customers. Unlike correctness, there is no upper bound on how good you can make something, and it is always more valuable to make something better.1

Like correctness, it gets harder and harder to make a good thing better, but not as steeply. Instead of hunting the 1 in 10,000 bug you get to explore the space of all the things you could change about your product that would make it more useful to people.

Which means that the value-to-effort curve ends up having diminishing returns, but not asymptotic returns, which is way better.

How do you get good?

In order to make your product good, you follow the product development cycle.

Making a guess

To make a guess you think about your product and the people who will use it, using all the information that you have and you guess what the product should be like. This can be things like “there should be a button to log out”. Or things like “If the robot detects a person in the chair, it should skip vacuuming it”

The more you understand your customer the better your guesses will be. Some people are better guessers than others. (We usually call them product managers). And sometimes your customers will have good guesses. (Though more often you will have to try and guess at the need that made them think of that feature and figure out a good way of meeting it.)

But the difference between great guessers and average guessers is, in my experience, pretty small. Because the average guess is wrong. Wildly wrong. The best guessers are probably right 60% of the time. This is why so much of startup advice is to ship your MVP as fast as possible. Taking any reasonable guess and trying it out is much more efficient than agonizing over the “perfect” guess.

Trying it out

Trying it out means building a version of your guess and using it. There is a spectrum of ways to try it out.

In the beginning you can easily tell if a guess is good or bad by trying it yourself.

Robots are hard and expensive to deploy so I’ve often hired a team of test users to fake-use the product. (At Robust AI we have a little fake warehouse next to our desks and hire pickers to come and pick fake e-commerce orders with our robots twice a week)2.

Early on, many guesses are so obviously bad that trying them yourself immediately gives good feedback. But later, each change is a smaller step and until you start watching customers use it you will have some uncertainty (because your testing might be different in some way from the real customer use). Eventually you get to the point where the uncertainty is larger than the size of the change and you can’t tell if you are making it better or worse. That means that there is an upper bound on how good you can make a product without watching real customers try it.

And, for the same reason that humans are pretty bad at guessing what product features will be good, we are also pretty bad at guessing how our testing will be different from reality.

All of that together means that until you are testing with customers, there is a ceiling of how good you can make your product, and that ceiling is pretty low.

Iterations: more is better

Why do people tell startups to “iterate like crazy”? In each iteration you make some amount of progress up your ‘good product’ ramp. That amount of progress in each iteration can be “engineering limited” or “guess-quality limited”.

If you have a fixed release cycle then you take the things you know from watching customers, order them by how much goodness you think they add divided by how much effort they take and work off the top of the list until your next release.3

Example:

1. Show an arrow indicating where the pick is.

2. Add happy beep after correct scan

3. Faster navigation in open corridors

4. Stop parking motion earlier

5. Give humans more clearance when passing

6. More confident motion through doors.

If you do this, you will always feel “engineering limited” because that list of things is much larger than you have time to do.4 But generally, iterations longer than a week are guess-quality limited. That is because if you get to watch a customer use the product after just doing item (1) and (2) you will probably immediately have a new guess for an improvement that is way more valuable than item (4) or (5) on your list.

When you have long iteration cycles you are trapped doing the low value stuff on your list because you only learn about new high-value stuff by iterating.

With slower iterations you are “guess-limited”. Or maybe more accurately “learning-limited”. A quarter long iteration is not much better than a one month iteration, but takes three times as long.

Somewhere around 1-2 week iterations you tend to stop being learning limited.

Chuck Rossi, the release manager when I worked at Facebook5, observed that there seemed to be a fixed amount of change that could fit in a single deployment. So to increase company velocity he increased deployments of the website from weekly to daily, and then to twice a day and took deployments of the mobile apps from six to four to two week cycles.6

Iterate like crazy!

Shameless Teaser

Correct and Good are different, and require different kinds of effort. Sometimes they are even in direct opposition. How should you divide your time? How do you know if you have the right blend? Check out Part 2:

Subscribe now

(This is Part 2. Part 1 was about various ways robots could be human like and is not required to enjoy part 2)

In part 1, I talked about the various benefits and drawbacks of making robots that mimic human form, but we didn’t actually get to answering the core question: “Why are there so many humanoids now?” and “Is this a good idea?”

To answer “why now?” we have to ask “what has changed?” Here is a timeline of what I consider the breakthroughs in state-of-the-art which enable what we see today.

• 20181: Boston Dynamics releases their first parkour video. Bipedal motion works well enough to count on.

• 2019: Convolutional deep neural networks make perception (object detection/localization) work well enough to do manipulation.

• 2022: ChatGPT3 convinces everyone that natural language is going to work well enough to power actual products.

• 2023: Tesla releases videos of their humanoid robots. VCs notice and funding for humanoids is suddenly much more attainable.

• 2024: Behavior cloning2 for manipulation starts to work well enough for tech demos and videos.

Other helpful general trends:

• Batteries have gotten much more energy-dense and much less expensive.

• Rapid prototyping has gotten much cheaper and faster (both from 3D printing and more automation in prototype-scale traditional manufacturing).3

But is that enough? Are these the missing ingredients? To answer that, let's talk about money.

Making Money

The purpose of a robot company is to replace expensive human labor with cheap robot labor.4

That “value produced” is split between the robot company and the customer. The customer saves money (which is why they bothered buying the robots) and they pay the start-up using part of the money saved, so that number has to be big enough to make it worthwhile for both parties. Let's do some back-of-the-envelope math.

Cost to Run Robots:

We’ll take a very low-ball guess at $60k.5

Lifetime of a robot: 3 years.6

Which means we get an annual hardware cost of $20k.

Cost of Support

Let's say that you’ve got pretty good reliability despite being cutting edge hardware. Your robots only need service every 6 months on average (either routine maintenance or repairing something that has broken). 

Each time that happens you have to ship it back to your repair facility (say $300 each way) and spend 3 hours repairing it ($300 in technician costs, amortizing in downtime for your technicians because they are not used 100% of the time). We’ll round up to $1k a repair. Plus you don’t do work for probably a whole week while it gets resolved, costing another 1k in lost revenue. $2k twice a year is $4k.

Cost of support: 4k per year per robot.

Cost of Supervision

Robots are less flexible and adaptable than humans, get confused easily and sometimes get stuck doing the wrong thing. The state of the art is to have a ‘call center’ of folks who can remotely supervise robots and unstick them. A huge lever on the economic viability is how many robots a single remote supervisor can handle. 1:1 is an obvious non-starter, but there is an inflection point around 5:1 where you can no longer have people watching camera feeds ready to jump in and help and have to have the robots recognize that they are in a pickle and ask for help. Let's say that we’ve solved that problem, for the most part, and a single remote operator can supervise 50 robots. That person costs 50k a year.7 That adds another $1k a year per robot.

We also have to have someone on-site to supervise and assign jobs, and do smaller troubleshooting and maintenance. Let’s say a 20 robot deployment takes 10% of one person’s time to manage. (We’ll dump the responsibility on our IT department because robots are kind of like computers). 10% of 200k divided by 20 robots is another 1k per year.

Cost of Supervision: 2k per robot per year

Total cost of operating the robot: $26k per year

Value Created

The ‘cost of human labor saved’ is pretty straightforward. It is the fully loaded cost of the human. Fortunately for robotics startups, this is quite a bit higher than minimum wage. In Silicon Valley, when you add taxes, health-care, salaries for their supervisors and profit for the contracting company, someone to clean an office building can easily be more than $70 an hour. For this reason, labor intensive workplaces, like warehouses, tend to not be in the heart of Silicon Valley, and $30 an hour is probably a more reasonable national average. 

Annual savings = hourly-cost x hours-per-shift x shift-per-week x weeks-per-year

Making some reasonable assumptions:

$30 x 8 hours/shift x 6 shifts/week x 52 weeks/year

$74880. We’ll round up to $75k

$75k seems pretty darn good. With our cost of $26K that leaves $49k of value, meaning we get a 188% return on our investment. Not bad! 

But wait, there’s less!

We’ve made an assumption that is really, really dodgy. We are assuming an hour of robot labor is as valuable as an hour of human labor. There are two big reasons to suspect that is not the case.

1: Robots are slow

There is a reason that you see a ton of sped-up robot videos. Robots work slowly. I love this video from 1X. It is an incredible (and incredibly honest8) depiction of the state of the art.

The tech here is amazing, and (although I’m sure that video took many takes to get), the robots must work pretty reliably to get that many things working all in a row. But you can see that the robots are doing pretty simple things about 2x-4x slower than a human would. 

The ratio is better for carrying things long distances (humanoids are probably a little faster than half human walking speed, i.e. between 1x and 2x) and worse for fiddly manipulation things. This (incredible) video shows some very challenging manipulation tasks:

It is super impressive. This video shows things beyond what I thought was state of the art. But it also takes 45 seconds to put the shirt on a hanger, a task that probably takes a human less than three (a ratio of about 10x).  I remember doing a focused speed-up sprint on our table wiping at Everyday Robots. We parallelized motion and planning and increased arm speed and pre-computed things so that they’d be ready when we needed them and managed to get a table-wipe down to about 45 seconds. Then I timed a human and they did a better job in 4.5 seconds (also a ratio of about 10x).

So if we are projecting the state of the art forward 2-3 years, we should probably give robots at least a 3x speed penalty, meaning it takes three robots to do the work of one human.9 Faster than that would be assuming that autonomous robots can do things faster than we can teleoperate them, which feels like an open research problem.10 Safety also gets harder if you let robots move that fast.

2: Robots are less adaptable

It turns out that actual jobs are made of dozens of individual tasks. Automation tends to automate tasks one-at-a-time. This is a place where humanoids have more potential than other kinds of robots. In 2011 I was working at Anybots Inc and we were looking into teleoperated security, and were very enthusiastic until we interviewed someone who did after-hours security work and he told us that in addition to ‘walking around and looking for bad guys’ his job consisted of testing all the doors to make sure they were locked, unplugging the coffee machine if it had been left on, turning off lights, etc, etc. We had only considered the main task, and not realized the whole slew of other smaller tasks that were part of that job.

If there is a human picking items in a warehouse and all the orders get picked, that person will switch to something like refilling or doing inventory or preparing labels or folding and taping shipping boxes, or any of the countless other tasks. Even a robot (humanoid or otherwise) who could physically do those tasks would go idle during that time unless you had programmed/trained those behaviors. The lack of flexibility costs at least 15% of the robot’s time. 

So we take our 75K, subtract 15% for the flexibility penalty ($64k) and divide by 3 for the productivity ratio and we get $21k of value produced per robot per year.

Now we subtract our $26k costs and all of a sudden humans are cheaper than robots.

Womp-womp.

What can you do?

Your options are rough. You can try and find a customer that operates more than 1 shift per day, because the same robot can work both shifts.11 However you start to need to spend more of your shift charging instead of just charging between shifts and your robot probably doesn’t last as many years and needs servicing more often if you run it more hours per day.

You can try to reduce the cost of your robot, but cutting the cost in half is going to be very hard. (Getting your cost down to $60k is already pretty optimistic. Shooting for $60k and accidentally ending up at $120k feels likely. $30k feels nearly impossible). 

So the biggest lever you have is your productivity ratio. Getting down from 10x-3x to between 2x and 1x is the most plausible way to make the math work. So this seems like the most important metric humanoid robot startups should track.

I made12 a “Robotics Startup CEO Simulator” website so you can fiddle with numbers and see how they work out:

What about ChatGPT? Doesn’t AI solve all of this?

LLMs (large language models like ChatGPT) do seem poised to solve two real problems in robotics. 

The first is dealing with the really long tail of weird shit the world will throw at robots.

There are thousands of things that could happen, but each one is very rare, and it is basically impossible to enumerate them all and provide correct responses for them. LLMs, however, do seem to have encoded some amount of common sense.

The second problem LLMs seem like a good fit for is high level planning to connect together lower level primitives.

Lets watch this video from Figure AI:

It's a very slick video showing some amazing tech. It can be hard to guess from videos of robot demos what's actually going on, but I commend @coreylynch (the human in the video) for explaining what we’re seeing in this tweet. The LLM is able to interpret spoken requests and select from pretrained execution models (he doesn't specify but these must be behavior-cloning.)

It’s funny, because I think the thing most people find impressive with the video is the conversational part, but that strikes me as a bit of demo showmanship. My guess is that the GPT input-output looks something like.

Input:

• human_said: “hey, can I have something to eat?”,

• camera_image.png

• available_controllers 

  • give_apple_to_person

  • put_crumpled_paper_in_basket

  • put_cup_in_drying_rack

  • put_plate_in_drying_rack

Output

• say: “Sure thing”

• selected_controller:

  • give_apple_to_person

I say, “I bet it looks like that,” because that’s how I would do it. It is cool that LLMs are smart enough to have the context to select that controller, but (if I’m right about the general structure and generality) you can tell that the secret sauce here isn’t really the LLM. The bottleneck on being multi purpose is still the dexterity, not the natural language interface. Given the small list of things this robot has controllers for, the speech part could be faked with some speech-to-text and a bunch of regexes. It would be more brittle, but would still work. And each time you wanted to add another task you would spend hours collecting teleop demonstrations, hours training a new neural network and 3 minutes adding another regex. Sure, it would break if you said, “I’m hungry” instead of “Can I have something to eat”, but I bet it breaks now if you say “pass me the plate” because there is no give_human_plate controller.

It certainly feels like we’re living in the future when you watch the conversation, but the value the LLM is adding, in terms of the economics of the business, goes towards the 15% flexibility penalty. No, the thing that blows my socks off with that video is that absolutely gorgeous two handed plate grasp. It is truly a thing of beauty. 

If you can tell robots to do other tasks in their down time (and the LLMs can figure out how to actually do those things and you have appropriate controllers to accomplish them), then they can be busy more of the time. But we only discounted 15% for flexibility. Even if LLMs gave you enough stuff to stay constantly busy that is a very small change to your viability, where the speed of productive work and cost of the robot are both big levers on viability. 

That's why that plate grasp is what impresses me so much. It is very close to human speed.

Behavior Cloning Is All You Need?

Behavior cloning suddenly feels like perception did 10 years ago. You can make it work in narrow domains by overfitting to a small problem. In both cases you collect many hours of labeled data and end up with something that works nicely as long at it matches the data you collected very very closely. But that's not what perception feels like today.

2014: Perceive this bottle on this background.

2019: Any bottle, any background.

2024: Pick up this plate from your right and put it into this basket on your left 

2029: …?

Today you can basically assume perception will work well enough for your problem. If learned manipulation from demonstration is on that trajectory it is a big fucking deal. This could be our solution to the dexterity problem.

It’s not guaranteed that it is on the same trajectory. The tipping point for perception was internet-sized training sets, which will be much harder to come by for robotics. The two ways I can see it happening are:

1. Folks figure out how to package and sell “fine tuning” a robot to a particular task and location. You demonstrate/teleop a new task for a few hours and then the robot will be able to do that task in that place with minor variation. Make robot-plus-learning-toolkit into a product, sell that product to lots of people, and use everyone’s data to bootstrap more and more general models which give you better starting points for fine tuning which give you more performance which gives you more customers.

2. Figure out how to do transfer-learning from videos of humans doing things so you can use a data source like YouTube to bootstrap your model. 

Option 2 is a reason you might want to bet on humanoids. If someone can make that transfer work (though it does not seem guaranteed that it will work) then I can imagine using “robots that are human shaped” helping with the transfer. If that's what's needed, then it is absolutely worth the extra cost and complexity of the human form. That's a lot of ‘if’s though.

Is it a Humanoid Revolution or a Bubble?

This is the big question, and I really don’t know. I’m voting with my time, working at Robust AI on a decidedly non-humanoid robot: one that puts a heavy emphasis on simplicity, low cost, and short term ROI. It's also one that skews much more heavily towards being a tool, rather than a servant. We are trying to build something more like a working dog than an employee.

I’m here because, after 8 years working on general purpose robots at Google X, I wanted to build something that we could use to find market fit, deploy and scale quickly. Humanoids are the high-risk-high-reward choice, and I’m currently excited to try something with a high probability of large deployments to real customers.

I’m not saying humanoids aren’t going to happen, but there are a lot of challenges to be solved before the economics of humanoids can work out. The progress is amazing but making the value larger than the cost is really hard: folks are going to have to nail both “very low cost robots” and “high productivity speeds”. On the other hand, technology is moving a lot faster than I thought it would a year ago. It is an absolutely incredible time to be a roboticist, and we won’t have to wait very long to find out what the future holds.

Share

Subscribe now

Thanks to Michael Quinlan, Rodney Brooks and Mrinal Kalakrisnan for reading an early draft of this post.

About the Author: Benjie has programmed robots at startups and Google X’s Everyday Robots and is currently Director of Robotics at Robust AI, where he hopes to ship robots disguised as shopping carts to customers who pay money for them.

There's a lot to unpack so this will be a two part post. In this part we’ll talk about what we even mean by “humanoid”, and the advantages and challenges of mimicking various human body parts. In part two we’ll dive into the economics of humanoid deployments and how LLMs (Large Language Models, like ChatGPT) enter the picture. By the end we won’t have any magical answers, but hopefully you’ll leave with some frameworks for thinking about humanoid robots.

What is a humanoid?

The question of “why humanoids” gets confusing because “humanoid robots” make a bunch of different promises. We’ll talk about a few different axes.

Generally when people think of a humanoid robot they imagine a robot that…

• Looks like a human,

• Works in human environments,

• Is multi purpose,

• Is able to be instructed and interacted with like a human.

Why look like a human?

When you ask folks why they are working on humanoids you usually get an answer about how the world is designed for humans so robots shaped like humans should succeed in our environment.

There is an implicit promise that, because it is human shaped, it ought to be able to do things that humans do now, and more, ought to be able to do them without requiring big changes to your facility. 

There is also a promise that, because it is shaped like a human, it will be flexible enough to do many different tasks that humans are able to do.

Obviously just being human shaped isn’t enough: you also need the strength, dexterity, perception and intelligence to accomplish those tasks. Being human shaped is also not required. One could imagine a multi-purpose robot that worked in unstructured human environments that looked nothing like a human. And you certainly don’t need to be perfectly human like to do useful work. So it's an interesting question: if you are trying to make a practical humanoid robot, which parts do you copy?

Legs

Legs let you be tall:

There are a bunch of tasks in human spaces that require you to manipulate things up high (get stuff on shelves) and down low (put things in cabinets). Human spaces also can get tight, so having a small footprint on the floor is useful. Those two things push you towards a tall, skinny robot which means a tippy robot. In order to not fall over you can either try to still have a low center of gravity (think of a cherry-picker. It goes super high, but has a huge amount of mass at the bottom to stay stable) or you can balance dynamically. Legs are one solution to dynamic balance, but you can also have dynamic balance with wheels.

Legs let you handle stairs:

This is the biggest win for legs over wheeled solutions. There are wheeled and track-based things that can use stairs, but they tend to be as exciting1 as legs.

Most of the current batch of humanoids have two legs, though Digit’s backward legs are decidedly non-human in their configuration.

I think it is interesting that X1’s EVE robot has a mermaid-tail torso that ends in wheels, but their next-gen robot has two legs.

But why not a wheeled centaur?

Nearly every one of the humanoid companies claim they are targeting a commercial or industrial application as their first market. These spaces are ADA compliant. I’m super confused by the lack of two-armed torsos on largish, heavy, wheeled bases. Legs are just really complicated & expensive. Plus they consume a bunch more energy than wheels, move a lot slower, and all of your manipulation needs to either make full body plans that take balance into account or your balance control and manipulation control treat each other like disturbances and they’ll be fighting all the time. And while it's true that legs are omnidirectional (you can step to the side) you can make wheeled bases that are omnidirectional too, and still skip a bunch of the challenges that legged balancing gives you.

Two Arms

Many tasks just need two hands:

After working for 6 years with this robot:

I’ve come to appreciate how much only having one hand constrains you2. There are things you can do one handed, but many of them are much easier two handed3, especially for not-that-dexterous robots with their clumsy grippers. Which brings us to…

Five Fingered Hands

Human hands are frikkin awesome. I could write an entire post about human hands and the ways that they outshine anything we can build for robots. We have 5 fingers and each finger has at least 3 degrees of freedom4 and the thumb and palm pinkie have at least one more each (you can move the base of your pinkie and thumb forward) for a total of 17 independent controllable degrees of freedom. At my first job we had a 17 DOF (degree of freedom) robot hand and (like a human hand) the only way to package all that motion was to move the actuators (muscles) into the forearm and do everything through cables (tendons). It was a beast and meant we had a huge forearm. The cables were also a maintenance nightmare and were always loosening or breaking.

I don’t think you need all of that motion to make a useful hand. In fact, I’m surprised that so many humanoids have five fingers. People who have lost a finger tend to be just as capable as people with five fingers. I’d think that robots, like cartoon characters, would converge on three5 to four6 fingers7.

Human hands are fast and strong and light. We have about 50 lbs of grip strength and can move fast enough that our eyes can’t track our finger motion. A motor and gearbox that powerful would tend to be big and heavy, which is not what you want at the end of your robot arm.

But the biggest way that robot hands differ from human hands is sensing. Our hands are incredibly sensitive. You can easily feel the height change of a single sheet of paper sitting on a table. When we want to fool a human’s eyes we use video at 30-60 frames per second, but when we want to fool human hands (for giving haptic feedback) we generally want to run at 1000 “frames” per second because the bandwidth of your hands is so much higher than your eyes. Think about how hard it is to do a delicate task with your whole hand numbed.

My money is on a less human looking hand (with fewer and bigger fingers) with good touch sensing, less than human speed, but similar to human grasp strength.

Size

The last category of human-ness is human sized. The push and pull here is that many tasks have some reaching up high and carrying heavy things. However, being sufficiently safe is super hard (for any robot). We are much less forgiving of our machines for bumping into us than we are our human co-workers, and robots tend to fall over on occasion. So your goal of making a tall and strong robot quickly gets into direct conflict with your goal of making a robot that won’t accidentally hurt someone. This is not easy because if your robot is strong enough to move heavy boxes that means it is strong enough to drop heavy boxes on someone, and usually means the robot itself is heavier than the heavy boxes.

You can make robots more safe by adding human-auditable and redundant safety systems in your software, but that is the opposite direction from the promise of generative AI being the engine of multipurpose robot control. You can have parallel systems (AI for control and conventional safety to verify the output) and that is probably what you have to do, but it's going to be really hard for the (dumb, simple) safety system to tell the difference between pushing hard on a box to put it in place and pushing hard on a box that causes a human to be crushed.

I don’t have good answers, just that it's really hard, and it seems like most robot companies are betting on being so useful and valuable that they are able to convince society (and therefore various safety bureaucracies) to accept a slightly higher chance of unpredictable injury. Higher than we’d tolerate from, say, a printer. It's not like humans are guaranteed not to bump into each other, we just usually don’t. It's kind of like the argument that self-driving cars don’t have to be perfect, they just have to be similar to human drivers.

You can also make robots more intrinsically safe by making them lighter, but making things light and strong at the same time is hard. And by hard, I mean expensive. And keeping your robot’s cost down is going to be very, very important to succeeding with a humanoid startup, as we will see in Part 2…

Subscribe now

Thanks to Michael Quinlan and Rodney Brooks for reading an early draft of this post.

About the Author: Benjie has programmed robots at startups and Google X’s Everyday Robots and is currently Director of Robotics at Robust AI, where he hopes to ship boring looking non-humanoid robots to customers who pay money for them.

5

6

The Mistake

The idea goes something like this: Programming robots is hard. And there are some people with really arcane skills and PhDs who are really expensive and seem to be required for some reason. Wouldn’t it be nice if we could do robotics without them?1 What if everyone could do robotics? That would be great, right? We should make a software framework so that non-roboticists can program robots.

This idea is so close to a correct idea that it's hard to tell why it doesn’t work out. On the surface, it's not wrong: all else being equal, it would be good if programming robots was more accessible. The problem is that we don’t have a good recipe for making working robots. So we don’t know how to make that recipe easier to follow. In order to make things simple, people end up removing things that folks might need, because no one knows for sure what's absolutely required. It's like saying you want to invent an invisibility cloak and want to be able to make it from materials you can buy from Home Depot. Sure, that would be nice, but if you invented an invisibility cloak that required some mercury and neodymium to manufacture would you toss the recipe?

In robotics, this mistake is based on a very true and very real observation: programming robots is super hard. Famously hard. It would be super-great if programming robots was easier. The issue is this: programming robots has two different kinds of hard parts. 

Robots are hard because the world is complicated

The first kind of hard part is that robots deal with the real-world, imperfectly sensed and imperfectly actuated. Global mutable state is bad programming style because it's really hard to deal with, but to robot software the entire physical world is global mutable state, and you only get to unreliably observe it and hope your actions approximate what you wanted to achieve. Getting robotics to work at all is often at the very limit of what a person can reason about, and requires the flexibility to employ whatever heuristic might work for your special problem. This is the intrinsic complexity of the problem: robots live in complex worlds, and for every working solution there are millions of solutions that don’t work, and finding the right one is hard, and often very dependent on the task, robot, sensors and environment.

Folks look at that challenge, see that it is super hard, and decide that, sure, maybe some fancy roboticist could solve it in one particular scenario, but what about “normal” people? “We should make this possible for non-roboticists” they say. I call these users “Mythical Non-Roboticists” because once they are programming a robot, I feel they become roboticists. Isn’t anyone programming a robot for a purpose a roboticist? Stop gatekeeping, people.

Don’t design for amorphous groups

I call also them ‘Mythical’ because usually the “non-roboticist” implied is a vague, amorphous group. Don’t design for amorphous groups. If you can’t name three real people (that you have talked to), that your API is for, then you are designing for an amorphous group and only amorphous people will like your API.

And with this hazy group of users in mind (and seeing how difficult everything is) folks think, “Surely we could make this easier for everyone else by papering over these things with simple APIs?”

No. No you can’t. Stop it. 

You can’t paper over intrinsic complexity with simple APIs because if your APIs are simple they can’t cover the complexity of the problem. You will inevitably end up with a beautiful looking API, with calls like “grasp_object” and “approach_person” which demo nicely in a hackathon kickoff but last about 15 minutes of someone actually trying to get some work done. It will turn out that, for their particular application, “grasp_object()” makes 3-4 wrong assumptions about “grasp” and “object” and doesn’t work for them at all.

Your users are just as smart as you

This is made worse by the pervasive assumption that these people are less savvy (read: less intelligent) than the creators of this magical framework.2 That feeling of superiority will cause the designers to cling desperately to their beautiful, simple ‘grasp_object()’s and resist adding the knobs and arguments needed to cover more use cases and allow the users to customize what they get.

Ironically this foists a bunch of complexity on to the poor users of the API who have to come up with clever workarounds to get it to work at all.

The sad, salty, bitter icing on this cake-of-frustration is that, even if done really well, the goal of this kind of framework would be to expand the group of people who can do the work. And to achieve that, it would sacrifice some performance you can only get by super-specializing your solution to your problem. If we lived in a world where expert roboticists could program robots that worked really well, but there was so much demand for robots that there just wasn’t enough time for those folks to do all the programming, this would be a great solution.3

The obvious truth is that (outside of really constrained environments like manufacturing cells) even the very best collection of real bone-fide, card-carrying roboticists working at the best of their ability struggle to get close to a level of performance that makes the robots commercially viable, even with long timelines and mountains of funding.4 We don’t have any headroom to sacrifice power and effectiveness for ease.

What problem are we solving?

So should we give up making it easier? Is robotic development available only to a small group of elites with fancy PhDs?5 No to both! I have worked with tons of undergrad interns who have been completely able to do robotics.6 I myself am mostly self-taught in robot programming.7 While there is a lot of intrinsic complexity in making robots work, I don’t think there is any more than, say, video game development. 

In robotics, like in all things, experience helps, some things are teachable, and as you master many areas you can see things start to connect together. These skills are not magical or unique to robotics. We are not as special as we like to think we are.

But what about making programming robots easier? Remember way back at the beginning of the post when I said that there were two different kinds of hard parts? One is the intrinsic complexity of the problem, and that one will be hard no matter what.8 But the second is the incidental complexity, or as I like to call it, the stupid bullshit complexity.

Stupid Bullshit Complexity

Robots are asynchronous, distributed, real-time systems with weird hardware. All of that will be hard to configure for stupid bullshit reasons. Those drivers need to work in the weird flavor of linux you want for hard real-time for your controls and getting that all set up will be hard for stupid bullshit reasons. You are abusing wifi so you can roam seamlessly without interruption but linux’s wifi will not want to do that. Your log files are huge and you have to upload them somewhere so they don’t fill up your robot. You’ll need to integrate with some cloud something or other and deal with its stupid bullshit.9 

There is a ton of crap to deal with before you even get to complexity of dealing with 3D rotation, moving reference frames, time synchronization, messaging protocols. Those things have intrinsic complexity (you have to think about when something was observed and how to reason about it as other things have moved) and stupid bullshit complexity (There’s a weird bug because someone multiplied two transform matrices in the wrong order and now you’re getting an error message that deep in some protocol a quaternion is not normalized. WTF does that mean?)10

One of the biggest challenges of robot programming is wading through the sea of stupid bullshit you need to wrangle in order to start working on your interesting and challenging robotics problem. 

So a simple heuristic to make good APIs is:

That feels universal enough that I’m tempted to call it Holson’s Law of Tolerable API Design. 

When you are using tools you’ve made, you know them well enough to know the rough edges and how to avoid them.

But rough edges are things that have to be held in a programmer’s memory while they are using your system. If you insist on making a robotics framework11, you should strive to make it as powerful as you can with the least amount of stupid bullshit. Eradicate incidental complexity everywhere you can. You want to make APIs that have maximum flexibility but good defaults. I like python’s default-argument syntax for this because it means you can write APIs that can be used like:

It is possible to have easy things be simple and allow complex things. And please, please, please don’t make condescending APIs. Thanks!

Subscribe now

Share

Thanks to Leila Takayama and Rodney Brooks for reading drafts of this.

About the Author: Benjie has been a roboticist since he first joined his high school robotics team, has programmed robots at startups and Google X’s Everyday Robots and is currently Director of Robotics at Robust AI, where he hopes to someday ship boring looking robots that solve real problems to customers who pay money for them.

And given how much I’ve begun relying on it in such a short time, I can’t imagine a future where most programmers don’t make very heavy use of AI assistance. Donald Knuth tells us, “Programs are meant to be read by humans and only incidentally for computers to execute.” but in the future we have a third category. I think that programs, libraries and programming languages that lend themselves to effective AI assistance will win over abstractions that LLMs are bad at.

So what does that look like? I don’t know,2 but I still wrote a whole post about me trying to find out.

My experience using AIs for coding

I’ve been using ChatGPT 43 and Github Copilot. ChatGPT feels like pair programming with a pretty junior engineer who doesn’t think that far ahead, but has memorized the documentation and types really fast. If you know what you want, but not exactly how to do it, ChatGPT is great.4 The first day of creating a game in javascript (a language I don’t know very well) it was easily a 10x productivity boost. I had an editable map, characters jumping around the scene and collecting gems and a working UI in about an hour. The productivity gain slows down because over time you realize that copy-and-pasting an isolated snippet for every problem you have doesn’t lead to great design. I found I still had to do some real engineering and refactoring to keep it from collapsing under its own weight.5 But it's still easily a 2-3x steady state on a domain I’m not a super expert in.

GPT also struggles with things that are legitimately tricky. It had a hard time getting the physics and collision code to produce nice gameplay, and I tried for a while describing the new undesirable behavior to it and trying its fixes a while before giving up, using my own brain, and implementing collision physics myself. But the ways that it failed reminded me of the kinds of tricky bugs I’ve made trying to get collisions to feel nice in previous game projects.

GitHub Copilot really shone on the posetree project. That one is a lot more mathy and there are a bunch of matchable-pattern stuff. I’d type 

@property

def x_axis(

And it would fill in the (mostly) correct docstring and implementation to get the x-axis of the pose (nice) and then suggest that I probably want y-axis and z-axis. (I do, indeed. Thanks, AI). It has more context about your file so it has less of a tendency to forget the names of things, and often notices things you ought to have but don’t. (Yes, I also want a ‘frame’ property on Pose. Thanks again). But you can’t describe a problem and have solutions the way you can with GPT.6

A master plan to show how smart I am

I’m proud that code written with posetree tends to look more like natural language, which I think makes it easier to read and debug. I’d love to prove that it helps, and, because I’m an engineer, I’d love to quantify it, but that has seemed impossible.

However LLMs work in natural language, so maybe (I thought) I can create a set of problems of increasing complexity and trickiness and have ChatGPT solve them with and without my library. Maybe I can even have some number of instances do it and count how many correctly solve each problem and this will create a quantifiable measure of how much better my library is.

Results?

And so I tried that, and failed utterly. 

But failing is interesting, so here’s what I learned.

Methodology

I started two GPT instances7 and gave them these prompts:

They both got:

“I'd like you to write some code for a mobile robot. The system is made up of the frames 'map', 'robot', 'tool', and 'camera'. 'robot' represents the robot base and has x pointing forward and z pointing up. 'map' origin is in the corner of the building, with z pointing up. 'camera' has z pointing out, and y pointing down. 'tool' has z pointing out of the gripper."

Then they got a custom second half:

GPTControl: 

You have the API get_transform(parent, child) which returns a 4x4 rigid body transform matrix as a np.array. For example, to get the robot's pose in map frame you can call get_transform('map', 'robot').

GPTTest:

<paste the documentation for the pose API>

You can assume that there is a PoseTree instance called pose_tree. For example, to get the robot's pose in map frame you can call Pose(Transform.identity(), 'robot', pose_tree).in_frame("map")8. 

Then I started asking them both a series of pose problems, drawn from things I’ve actually had to do with robots, starting easy and getting harder. I didn’t stick to that very long though, because of the way things started going off the rails.

AIs like to pattern match. 

The first problem is that ChatGPT has seen lots and lots of ‘create a 4x4 matrix transform, do some multiplication with it, and use numpy to scrape out your answer’ code in its training data. But posetree library is brand-spankin-new so all of its patterns are outside the training data. So when I asked for code to solve pose-problems it successfully used my library to write old-school transforms code.9

For example I asked, “implement distance_from_tool_to_camera() which returns the cartesian distance between the tool and the camera.” and got:

Which is… fine. But tool_pose.distance_to(camera_pose) is right there. As is np.linalg.norm(tool_pose.in_frame(‘camera’).position). The power of posetree is that you don’t have to convert everything to the same frame and then do linear algebra on it.

You’re doing it wrong!

I asked a few more questions before getting frustrated at it for using my library so badly. I couldn’t help myself. I wrote answers to each question myself using ‘canonical’ posetree abstractions and gave it those and I was rewarded this this gem:

“These improved implementations make excellent use of the posetree API methods, and they are more concise and readable compared to previous versions. They also handle frames properly, ensuring that operations are performed relative to the correct frame of reference.”

Thanks, GPT. This is why AIs are better than co-workers.

Ego assuaged, we soldiered on.

Examples are better than documentation

LLMs are fundamentally pattern matching machines and going from documentation to implementation was hard for it, but extrapolating from examples was easier. After seeing the solutions to the first three problems it did much better with the next two. However, the ControlGPT also did well on those questions and didn’t need me to give it extra examples.

If I cheat enough and wait enough I get the result I want!

Eventually I hit on a problem that my library + example-hinted GPT solved correctly and my no-library GPT failed on. 

Next question: You have get_bottle_pose() which returns the bottle pose in the camera frame, and move_forward() which drives the robot some (unknown) amount forward. Once the robot moves, it will no longer be able to see the bottle. Please write a method to perceive the bottle, move the robot and then return the pan angle needed to point the camera at the bottle. (0 pan has the camera pointing in the robot x-axis, positive pan is positive rotation about robot-z).

TestGTP:

It creates robot_pose_in_map that is doesn’t need (and never uses), and ignores a perfectly good angle_about_z_to() method, but not bad.10

ControlGPT:

I mean he’s got the spirit, but boy is he confused. camera_t_robot doesn’t change when the robot moves, so robot_t_camera_after is the same as robot_t_camera. That means the expression

np.dot(robot_t_camera_after, np.linalg.inv(np.dot(get_transform('robot', 'camera'), camera_t_bottle)))

is 

robot_t_camera * (robot_t_camera * camera_t_bottle).inverse

robot_t_camera * (robot_t_bottle).inverse

robot_t_camera * bottle_t_robot

And we’re making one of the classic blunders of matrix multiply pose math: We’ve got our matrices in the wrong order. That multiplication is easy to compute but gives a nonsensical transform (the words next to each other have to match, and camera != bottle) and we get a wrong answer.

But I’ve had my thumb on the scale and have kept going until I got the result that I wanted, so you really can’t call this any kind of proof that posetree is easier for LLMs to use.

Maybe I only give examples?

Since it started working so much better after I gave it some examples, I wondered if giving it the documentation had been a blunder. I know I often learn how to use a library by looking at the unit tests, and what is a unit test but a giant pile of examples waiting to be fed in?11 So I fired up another instance12 and pasted in the unit test. And that went even more poorly. I don’t know why. Hilariously, after pasting in hundreds of lines of posetree unit test, when I asked it to solve a pose problem its first response was to use ROS’s tf2 library.13

When I told it I wanted to use the posetree library, it tried but got… every? API call wrong?! I’m including it because it’s kind of impressive.

Like, literally every line is wrong. get_pose takes a position, quaternion and a defining frame, Pose’s constructor takes a transform and a frame (are those 6 numbers position and euler angles concatenated together? gross) and posetree.transform() does not exist and there is no analogous method it is thinking of. Wow.

A Last Minute Contender

While reading my draft of this my friend and actual ML expert, Michael Quinlan, asked why I hadn’t tried GPT’s code-interpreter mode.

“What’s that?” I asked.

“Oh, it's the thing where you upload your source files and ask it to use them to write code.”

So I zipped up my files, uploaded them and asked the first question and… it bombed it. More getting all the APIs wrong, plus it multiplied a pose by a transform (and the whole point of posetree is to not do that.) Total failure. Bummed, I tried one last thing. I said, “You are misusing the library. Can you read the README and try again?”

And, holy shit, it nailed it. It got all of the questions right with idiomatic usage of the library. It even got a really tricky question about generating a circle of pregrasp poses which both other instances failed so badly I just gave up and moved on.

The only one it got wrong was the final question about looking at the bottle after the robot moved. (It stored the pose in robot frame so after the robot moved it still looked at where the bottle used to be relative to the robot.) Curious I asked, “This is incorrect. Can you figure out why?” and got:

The issue arises from the fact that the robot's perception of the bottle's pose is outdated once the robot moves. The perceived bottle pose is in the camera's frame of reference before the robot moves. After the robot moves, this pose information is no longer valid because the camera (and therefore its frame of reference) has moved along with the robot.

The correct approach should be:

• Perceive the bottle and get its pose in the camera frame.

• Convert this pose to a frame that does not move with the robot, such as the 'map' frame.

• Move the robot forward.

• Now, convert the stored bottle pose from the 'map' frame back to the 'robot' frame.

• Calculate the pan angle based on this updated bottle pose in the robot's frame.

And followed up with a correct implementation.

Conclusion?

So here we are, at the end of my journey. Right now, in 2023, it seems that if you want LLMs to be really good at your library you should have it be really popular before 2021 so that there are a ton of examples from stack overflow in its training data. If you can’t pull that off, using the code-interpreter mode, and writing good documentation works pretty darn well, though it’s pretty funny that you have to tell your AI to RTFM.

I would love to see more support for LoRAs (or something like them) on the scale of Stable Diffusion in terms of ease of use. A LoRA is a small, easy to train “mixin” network. In image generation it lets the AI draw a particular person, or style or even abstract property like “extra detailed”. For Stable Diffusion there are websites where you can browse LoRAs (with example images), copy the link and put it in the UI and it will download it, and start using it. Ease of use matters for adoption. If I could create curated examples/documentation, and release a mixin for a coding LLM that made it really good at writing code using my library, (and I thought that people would use it), I absolutely would.14 Someone should get on that. I can imagine a product where you pick your technology stack, add in all those mixins and spin up your pair-programming ai. That would be sweet.

It does seem like, for the most part, things that are tricky for humans are tricky for LLMs (matrix multiply order, timing on collisions before/after applying the timestep, remembering what you called that variable you just created, like, 30 seconds ago.) So for now I’m just going to continue trying to write software that is easy for humans to read and understand and hope for the best. But if you know about great ways to teach an LLM new tricks, let me know in the comments!

Share

Subscribe now

You can read the whole chats here:

Test: https://chat.openai.com/share/6bbdce27-9cd7-43dc-b9ae-4a9c93f82884

Control: https://chat.openai.com/share/c9953f39-5589-4430-8075-98defbdfa0da

Unit Test Input: https://chat.openai.com/share/0ff27691-dfc8-4969-b8f4-221848c8aea0

Code-interpreter with upload: https://chat.openai.com/share/21d0fd67-9e07-4123-b43e-6a2f8acf0edb

Thanks to Mrinal Kalakrishnan and Michael Quinlan for reading an early draft of this.

<Robot doing an ‘Instagram Influencer’ pose. Caption: “Not that kind of pose”>

The comics are often an important part of the rhetorical flow of my essays. I find them especially helpful to clarify what I mean by showing the opposite viewpoint. I find ideas are often better explained by showing what I don’t mean. So I need them in there as I write and edit so I know what to say next. But I re-write enough that I don’t want to sink all the time into creating finished art until the very end.

Once the text is in a pretty final shape, the next step is to draw the line art. I used to use pencil, photograph them with my phone and then use some free Android app to bump the brightness/contrast to get a black and white line drawing.1 But recently I’ve got a ReMarkable2 tablet (an e-ink drawing tablet) and I find that it is almost as good for drawing as pencil on paper, but better for undo and erase and I get nice black and white digital versions uploaded automatically.

Then I use an AI assist coloring service to color. In the AI world it is ancient, and I’ve been using it for years. It does a mediocre job unhinted but it has a UI for adding color hints to the line drawing. I find it very rewarding to work iteratively with it making modifications until it looks right.

Lastly I’ll pull it into Gimp (because photoshop is expensive) and do things like touch up the eyes/teeth if I can’t get them bright enough. Or I’ll pull them into google slides and do annotations on top of them and then screen shot them (for things like arrows or bounding boxes).

That’s the process I’ve been using, but I’ve also been playing with Stable Diffusion and controlnet a ton for my video game project so for my last post I decided to see what I could get from control-net and my doodles. I thought the results were pretty cool but in the end decided to keep my art style consistent across the blog posts. I’m curious what people think of the two different styles, though.

Let me know what you think in the comments.

Share

Subscribe now

I’ve been working on general purpose robots with Everyday Robots for 8 years, and was the engineering lead of the product/applications group until we were impacted1 by the recent Alphabet layoffs. This series is an attempt to share almost a decade of lessons learned so you can get a head start making robots that live and work among us. Previous posts live here.

What is posetree.py

There is one library I’ve written that is so important to my productivity and sanity I wouldn’t bother working on robots without it. It’s called posetree and it's a library for dealing with poses and transforms. It makes whole classes of super common bugs impossible, robotics code more readable, and frees precious brain cells from fiddly boring stuff. With it, you can spend more time thinking about the real problems you’re trying to solve. So I wrote this open source version so everyone can use it. You can jump right to the API on the GitHub page, or keep reading to learn why I think this is a better way to think about poses.

Not this kind of pose.

Wait, what's a pose?

In case your experience in robotics is more casual: a pose is a way of describing the position and orientation of things. If you have a robot and it sees a person it needs to avoid and also a table it needs to clean; all that information can be represented by poses. (A pose for the robot’s location, one for its camera, another for the gripper, one for the person, and one for the table).

Also in robotics things move around, so it matters when we saw the person there. Roboticists spend a ton of time thinking about where objects are, (and were), and especially a ton of time debugging these things. One of my favorite quotes sums it up:

Names Matter

Some robotics codebases use “pose”, “transform” and “frame” fairly interchangeably. This is a huge missed opportunity to be clear in our code and our thinking. So posetree takes an opinionated stance:

• Transform: This comes from the verb, to transform. It is how you get from one place to another. It is an operation to change your location from one to another.

  • Example: “Take 10 steps forward and then turn 90 degrees to your left”

• Pose: This is a noun. It is a physical location and orientation in 3D space.

  • Example: “Standing at my front door facing the street.”

Notice we can take our example pose (standing at my front door facing the street) and transform it into another pose by using our transform instructions (take 10 steps forward and then turn 90 degrees to your left). If we do that we have a new pose that is a bit in front of my house and (in my case) facing our detached garage.

We can make any pose we want by taking a starting point and transforming it. The starting points are called “frames of reference” (or just frames) and without them we can’t talk about where anything is. You can think about frames as famous poses. They are like pose celebrities. Everybody knows them.

In this sentence, “From the Target Parking lot, take the first right and go two blocks” we define a pose. The transform is “take the first right then two blocks” and the frame is “Target Parking Lot”. This uniquely describes a location and orientation.2 Which brings them to our definition.

• Frame: This is a pose that is so important we’ve named it and everyone knows about it. We use them to measure where things are in reference to it. 

  • Example: We could name “Standing at my front door facing the street” “journey_start” and then we could describe that other pose by saying, “from journey_start take 10 steps forward and turn 90 degrees to your left”

Why is this good?

It helps make it clear what information is needed for each of these things.

• A transform is a change-in-position and change-in-orientation.

• A pose is a transform plus a reference frame.

• A frame is a famous pose with a name that other poses refer to.

Each one in the list takes the one before and one additional piece of information. Traditional robotics software often uses data-structures that only have enough information to hold a transform and uses it to represent poses and frames. So where does the starting frame information go? Into the variable name(!).

Robot code often looks like this:

Where each of those things is actually a transform object, representing a pose by storing the starting frame into the variable name before the “_t_” (which stands for “transform”). This tradition started because there is a nifty math trick where if you stuff the rotation and translation just right into a 4x4 matrix (and add an extra 1 in the right spot) you can chain transforms together by multiplying the matrices.3 But this is error prone and it's easy to get the order wrong because matrix multiplication is not commutative (A*B != B*A) so the naming convention helps because you can look and see that the words next to each other are the same.

But the fact that we can calculate transforms via matrix multiply is an implementation detail. We don’t have to structure our code and variable names to carefully multiply matrices just right. That's what computers are for.4 That's why in posetree, poses contain their defining frame as part of the data.

No really, why is this better?

Because poses know their frames and have a reference to the transforms between all the current frames, they can do math across frames automatically.

Here is an example:

Those poses are defined in different frames. To do this without this library looks something like:

distance_from_person_to_charging_bay = np.linalg.norm(get_translation(world_origin_t_charging_bay.inverse() * world_origin_t_robot * robot_t_camera * camera_t_person))5

If this small example seems tricky, I guarantee you it gets a lot harder. And every roboticist has made the mistake of getting the order wrong or forgetting an inverse.6

Show me more things it can do!

Sure thing!

Let's get a set of potential pre-grasp poses that are in a 50 cm ring around a bottle with z pointed inwards. The bottle comes back with a pose where it’s z points up.

Lets point the robot base at a person:

We’ll take the current pose, and rotate it about z to have the x-axis point at person pose. What frame is person_pose in? I don’t know and I don’t care.

Lets carefully construct a grasp pose:

But the big win is the “timestamps” part

Because a pose represents a place relative to another frame, and frames move all the time, and the pose_tree is aware of that motion, you can express very powerful ideas with simple code.

We expect a lamp to stay stationary relative to the world, so we’ll store its pose in that frame. We’re also careful to convert from the camera frame (were we detect the lamp) into world frame at the camera-shutter timestamp, so that even if the robot is moving and we get the detection message after a bit of delay, our telemetry data and perception data are synced and the lamp is fixed to the outside world exactly where we saw it.

Under the hood, look_at will call lamp_pose.in_frame(“robot”) to figure out where the lamp is relative to the robot and calculate the angles needed to move the head to look at it, even though the robot has moved away since we observed the lamp.

In general you can use the parent_frame of a pose to ‘glue’ it to something; like the robot’s base, the world or even the tool. For example:

If you’ve read this far you should check out the readme and documentation because I worked, like, really hard on it. And if you’re working on a robotics project, try integrating it and let me know how it goes!

Subscribe now

Share

Thanks to Ryan Julian and Jeff Bingham for reading early drafts of this and for getting philosophical about poses with me many times over the years.

The Project:

My project is to make a 2D platformer where the players can design their own levels and then generative AI will create beautiful rendered images to represent the levels. I wanted to do something that wouldn’t be possible without AI: letting the players participate in the creation of art. We’ll skip to the end and you can see what the game looks like now:

And here are some of the different themes, though you can also create your own.

You can play with it here: dimensionhopper.com

I recommend looking at random levels or the gallery and seeing what’s out there.

The Journey

But lets talk about the process to get there.

Part 1: proof of concept

I’d played a little bit with Stable Diffusion before and it is a really fun toy to make cool pictures, but I always felt like I didn’t have quite enough control. For me the enjoyment of creation comes from the interaction between what I do and what I get, and I wanted to have more input. That’s why I was so excited when I read about control-net, which gives a ton more knobs to control the output. I immediately wanted to use it to make a 2D game.2

Getting up and going

I installed Stable-Diffusion on my laptop, fired up the webui3 got control net working and fed this depth image into Stable Diffusion:

This has the platforms as the closest pixels (white) and the black background means that part is far away. I was using a pixel-art model that had this amazing demo pic in the CIVAI page:

So I copy the prompt and settings from that, tweaked a bit and hit generate…….. And get this:

Not quite what I was hoping for… I try and fail to get the depth mode working for a while.

And switch over to “scribble” mode for control-net. Scribble mode takes outlines of shapes and lets them guide the images (instead of depth).

More interesting but still not good. 

Changing the prompt: 

“pixelart video game environment, platformer level. Create an image of a mystic stone temple in the jungle. shafts of sunlight, dramatic lighting. Vines and cracks in brown stone”

Much closer! I have a picture. It kind of interacted with the level. Still not good but better. 

Ooh, I have the level casting some shadows now.

“pixelart video game environment, Create an image of an abandoned space station, with broken systems, flickering lights, and a sense of danger. Show the wreckage, the abandoned rooms, and the unknown threats that linger.”

Eventually with some more playing around with settings I get levels like this pretty reliably:

This is a big improvement over the start, but (1) it didn’t really look like the level was part of the art, it was, at best, pasted over, and (2) the level textures looked like a repeating tileset. Human videogame developers do this so that they don’t have to draw a different bit of grass for every square of platform, but I didn’t have that constraint. Huh.

No more pixelart.

I decided that part of my problem was using a model trained on pixelart. It was faithfully copying the genre of repeated tileset, which was exactly what I didn’t want. So I changed to another model, this one built around children’s illustration, and my first image out looked like this:

Wow! So much better! The platforms have shadows on them, have objects in front and behind and it's actually a nice picture. I’m on to something!

The new model made nice pictures but I quickly realize I’m walking the line between two failure modes. 

Either I have a nice picture with the level kind of mostly drawn on top of it:

Or I get a nice, integrated picture where its really unclear where you are allowed to stand:

The last one is especially problematic because the ‘level’ part you can stand on has been rendered as a window, exactly inverting the semantics. This is a fundamental problem with using ‘scribble-mode’ as I’m giving Stable Diffusion no way to know what is close and what is far, just outlined shapes. 

Breakthrough: Lips on the Platforms.

I go back to depth, but have a thought: what if I hint that the top of the platform has a little lip. Rather than code it in my game rendering engine I just draw them in Gimp.

Happy little platform toppers

Holy shit it works!

And more importantly: it works pretty much every time. Most Stable-Diffusion workflows include generating 4-10x more images than you actually want, and choosing the good one. For my idea to work we needed all of the levels to be playable (you can tell where the platforms are) and most of them to be good (beautiful illustration) because there wouldn’t be a human curation step.

Control Image Matters

I’ve learned that the look of the depth image really changes the quality of the output. I was working on jungle ruins and kept getting images like this:

(Side note: its amazing how quickly standards rise. Early on that image blew me away, but now it looks meh). 

The problem4 is that there aren’t really any reasonable pictures of jungles with that depth map. Jungles (and really everything) doesn’t look like that. Stuff doesn’t float in the air, it has stuff under it holding it up. This leads to breakthrough 2: add supports.

Each platform block projects a dark gray box to the floor below it and that gives structure to the world. The dark gray has no gameplay purpose, it just acts as a hint to Stable Diffusion about what the picture is of. And we get much better images.

Things were pretty good, but I was still having trouble with caves, and I wondered if it was because it was trying to match the straight, sharp edges of the depth map and having a hard time making it look organic, so I added adjustable roughness to the images (as well as adjustable background depth so that sky can be far away for outdoor scenes and closer for indoor/jungle scenes.

Notice the bottoms of the platforms are really straight and its invented a bunch of lightcolumns / waterfalls that can have perfectly straight verticals. This one is actually quite good, but the square corners are not ideal.

Now everything is believably subterranean, and the underlighting on the bumpy rocks looks right.

Gems and Characters

Stable diffusion doesn’t create any transparency.

But can cheat with the level image because I have the depth information I fed in and mask based on that, so characters can go ‘behind’ the platforms when needed. For the gems, if I ask for ‘<blue gem/ruby jewel> floating videogame object on a black background’ and then subtract the background in python.5

For characters I found this model which was really fine-tuned to heck to make 4 frame walk animations. The creator really wanted left/right/up/down animations so combines this model with a lora of whoever the subject is to create all 4 directions depicting the same character. As a result, I think it is trained so all the ‘walk right’ just get the prompt “PixelartRSS”.

I’d like to be able to prompt something about the character I want, and this sometimes kinda works6 but not nearly as reliably as I’d like. My suspicion is that if I had the training data, limited it to just the sideways walk and labeled it with actual descriptions of the characters it would work better for me. But it does reliably make people that walk. So as long as you aren’t picky, you can get new player character sprites all day long.

Wrapping it all up

From there I just had to make it work in my own app. I used the excellent diffusers library to wrap the generation and make my little server. Everything is moving so fast that I’m sure I’ve done some things in very silly ways in my image_generation.py and there are probably 2-3x speedups to be gained be configuring it right, but for now it all works, and it’s fun to play with. Why are you still reading? Go check it out!

dimensionhopper.com

And subscribe for more about the development of this game, and more robotics content after that.

Discuss on Hacker News: https://news.ycombinator.com/item?id=36295227

Share

Subscribe now

I’ve been working on general purpose robots with Everyday Robots for 8 years, and was the engineering lead of the product/applications group until we were impacted1 by the recent Alphabet layoffs. This series is an attempt to share almost a decade of lessons learned so you can get a head start making robots that live and work among us. Previous posts live here.

What is Tech Debt?

In my experience, there are two kinds of tech debt. The first comes from going fast and building something that cuts corners. This is the kind of tech debt that most people think of. It is a loan against the future to get something done now, and you pay it down (often with interest) when you have more resources. We’ll call this Cut-Corner tech debt, and it gets a bad rap, but it is often a good idea to take on some Cut-Corner tech debt in order to move fast.

“We can’t hardcode the location of every robot charging station into our source code! There should be a cloud! And a UI! And permissions! And ways for the robot to autonomously update them if they detect that they’ve moved!”

“There are only 3 charging stations in the whole world… and I finished hard coding them while you were talking.”

Then there is the other kind of tech debt. This one comes from having a problem, sitting down and imagining a system that could solve that problem, writing down all the other problems the system could be extended to solve, and then building that system. It feels really good because you are building something beautiful and future proof and extensible and scalable and all the things that make engineering satisfying.

“If I modify it just a little bit, I think I can make my type system do SLAM and also function as a working ML accelerator”

But you only have experience with the one problem that you started with, and probably still don’t understand it well.2 And odds are high you will never ever need any of the other features you imagined. You’ll end up with a large, very complicated system that solves a bunch of problems you don’t have, but is hard to change when you learn that you misunderstood the original problem. We’ll call this Grand-Framework tech debt.3

“I finished my Universal Task Execution Framework. Now we can use it!” 

“Great! Can you get the robot to pick up one of these two cans?” 

“Um, well, no, UTEF assumes a single perception context object. Hmm. Maybe I could create an Orchestration Framework for managing multiple UTEFs? Oh, but the cloud optimization depends on a singleton… Umm, Let me get back to you in a few weeks” 

YAGNI vs DRY: Fight!

Engineers overestimate the odds they will need a feature and underestimate the effort to refactor a small system.4 This is why people still have to go around yelling, “You Ain’t Gonna Need It” (YAGNI). Because deep in our hearts we are sure that we will need it, even though we almost never have in the past. Another engineering adage, “Don’t Repeat Yourself” (DRY), can lead you astray, because it seems to say "always make an abstraction".5 Don’t be so afraid to repeat yourself that you build a big system just in case.

In DRY vs YAGNI I lean toward YAGNI every time:

That's because the first flavor of tech debt is so much nicer. It is cheap to acquire (that's the whole point of cutting corners) and cheaper to clean up (because it's small and you don’t hurt anyone's feelings/career if you delete it). Whereas the second flavor of tech debt is very expensive to acquire AND expensive to remove.6

Scar Tissue

As companies grow and mature they develop what I call ‘scar tissue’. This comes from something bad happening (like an outage) followed by a post-mortem (ooh, an intern changed a config) and a policy change (config changes need to be peer reviewed). Each bit of scar tissue is a good idea, but the net effect is a growing continuous drag on getting things done.7 Folks coming from big companies sometimes see the cut-corners in the ancient parts of the code base and know the pain and scarring they caused and think, “We’ve learned better. That cut-corner-code sure caused a bunch of headaches as we scaled. I won’t make that mistake: we’ll start with scalable stuff right off the bat.” I suspect that the old code at successful companies is full of cut-corner tech debt because that is how those companies moved fast enough (and learned fast enough) to become successful. And the startups that built beautiful abstractions from the beginning starved from lack of progress and never got to hire a bunch of engineers to admire their beautiful architecture.

“Three years in, I have invented the perfect programming language to write my startup’s code in.” 

“What’s the product?” 

“Well, nothing, actually. We ran out of money a year ago.”

TLDR?

Some tech debt comes from doing a sloppy job on execution, some comes from doing a sloppy job on understanding the problem. Great execution on the wrong problem is way worse in the short and long term, which is why doing rough, throw-away work in order to fully understand the problem is worth it even though it feels bad.

What does this have to do with robots?

When people start working on general purpose robots there is a tendency to try to make sure that everything they do is Fully General Purpose And Future Proof. They have grand visions and world changing ambitions. So it seems like they should be building grand software, to match. This pushes folks really hard towards the expensive kind of tech debt. Here’s some sniff tests to keep an eye out for: 

Can We Make It More General Purpose?:

You are facing a problem for the first time and you propose a simple solution. A fellow engineer (who’s otherwise a smart, well meaning, lovely human being) asks, “Yeah, but is that general purpose enough?” You are in the danger zone for Grand-Framework tech debt. Stick to your guns and generalize only to problems you have right now.

Cart Before the Horse:

If you are building a generic capability and go hunting for a problem to use it on (a problem you otherwise wouldn’t need to solve in order to build your MVP) because it makes your platform feel more “general purpose” you are in the danger zone for Grand-Framework tech debt.

Good news, I got our customer to agree to spread rubble in parts of their warehouse so we can practice walking on uneven terrain!

Building for Someone Else:

If you are considering building a library or framework to support a workflow or help solve a problem that you’ve never solved yourself you are in the danger zone for Grand-Framework tech debt.8

This is for “Non-Roboticists”:

This is a special case of “ Building for Someone Else” that seems to pop up a lot for folks interested in general purpose robots. The impulse is good: making robots do things is hard and it would be good if it was easier. The problem here is that by framing it  “accessible to non-roboticists” you encourage frameworks and libraries that try to hide the complexity of the problem inside of magic. Folks propose APIs like this:

robot.grasp(“apple”)

robot.place_into(“basket”)

Which is great until “grasp” doesn’t work for you or do exactly what you’d want. It’s like all those web-app-frameworks that make for great demos in a hackathon (look you can make twitter in 4 minutes) but no one actually builds real applications out of them. They let you get to 80% in a few minutes but have no path to 100% because you can’t control all the things you need to. Instead, make programming robots productive for folks as smart and capable as you, but who are less patient for stupid bullshit. That’s how to make a beautiful API.

Classic “You Ain’t Gonna Need It”:

If someone says, “We don’t need this now, but I’m sure we’re going to need it eventually and it's going to take a very long time to build so we should start now” you are in the red-hot alarms-blaring danger zone for Grand-Framework tech debt.

How to have the good kind of tech debt

The way out of this, culturally, is to acknowledge that no-one (including you) knows how to build general purpose robots that work. And acknowledge that most things are going to be wrong the first few times. The shortest, best, path to general purpose components is to reject DRY and repeat yourself, implementing single purpose, application specific, feels-unscalable-and-throwaway behaviors.

Then, once you have two or three of those under your belt, look at where you spend a bunch of time-that-doesn’t-feel-productive and pull those out into a library or RPC. Keep things composable, opt-in and a-la-carte so it's easy to stop using the parts you don’t like. My favorite pieces of software are the second or third version of libraries written by people who were scratching their own itch.

My Favorite Acronym: TSTTCPW

Always strive to built The Simplest Thing That Could Possibly Work. Let future requirements be covered by future refactoring. You’ll still end up with tech debt (all projects get tech debt), but you’ll end up with mostly the cheap kind and avoid the expensive kind.

"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work."

– John Gall

Subscribe now

Share

1