A Challenge to Roboticists: My Humanoid Olympics

Summary

I was a little disappointed by China’s World Humanoid Robot Games.1 As fun as real-life Rock ‘Em Sock ‘Em Robots is, what people really care about is robots doing their chores. This is why robot laundry folding videos are so popular: we didn’t know how to do that even a few years ago. And it is certainly something that people want! But as this article so nicely articulates, basic laundry folding is in a sweet spot given the techniques we have now. It might feel like if our AI techniques can fold laundry, maybe they can do anything—but that isn’t true, and we’re going to have to invent new techniques to be really general purpose and useful.With that in mind I am issuing a challenge to roboticists: Here are my Humanoid Olympic events. Each event will require us to push the state of the art and unlock new capabilities in robotic manipulation. I will update my Substack as folks achieve these milestones, and will mail actual real-life medals to the winners.Current State of The ArtIn order to talk about why each of these challenges pushes the state of the art in robotic manipulation, let’s first talk about what’s working now. What I’m seeing working is learning from demonstration. Generally, folks are using puppeteering interfaces. Most common seems to be two copies of the robot so that a human can grab and move one of them while the other follows, or a virtual reality headset with controllers for hand tracking. They then record some 10-30 second activity hundreds of times over. Fromm that data, a neural network is trained to mimic those examples. This technique has unlocked tasks that have steps that are somewhat chaotic (like pulling a corner of a towel to get it to lay flat) or have a high state space (like how a towel can be bunched up in myriad different ways).Thinking about this method of training robots to do things, it should be clear what some of the limitations are. Each of these has exceptions, but together they form a general trend.No force feedback at the wri...