Optimizing our way through Metroid

Summary

← Blog Optimizing our way through Metroid People ask me: “why do you let your employees spend so much time playing Nintendo games?” People think we do it for the marketing. People think we do it to have cool demos. People think our blog series on learning autonomous testing concepts via how they come up in games is a pedagogical gimmick and nothing more. People are totally wrong. The honest truth, the underlying reality beneath the hype, is that this is actually how we figured this stuff out. None of us were fuzzing or PBT experts coming into this business, and if we were that wouldn’t have helped anyway, because our ambitions quickly went way beyond the state of the art in those fields. So we started asking questions like: “why can’t you beat The Legend of Zelda with a fuzzer,” and pixel by grueling pixel we learned enough to build the Antithesis platform. The lessons we learn in the Nintendo domain transfer very well to our core business of testing big, complicated distributed systems (if you don’t believe me, I gave a whole talk on that topic). Today’s war story is about one such lesson – the barrier we encountered in a Nintendo game, the new technique it inspired, and how that makes our platform smarter and better at testing everything. Let’s get started! The case of the red door Metroid (1986) broke all kinds of ground when it came out – featuring nonlinear, exploration-based gameplay, challenging tactical sequences, and cleverly-hidden secrets. To our knowledge, the game has never been successfully completed by an autonomous system. But Antithesis cruises through most of the game with ease: fighting enemies, gathering power-ups, and slowly filling in the entire map… …until it gets to this door: How do we know that that door is a problem? Well, while we’re testing your software, we stream observations of the system under test into our analytic database. In the case of Metroid, some important observations include Samus’s X and Y position on the game map. Then we...