World Emulation via DNN

Summary

I turned a forest trail near my apartment into a playable neural world. You can explore that world in your web browser by clicking right here: By "neural world", I mean that the entire thing is a neural network generating new images based on previous images + controls. There is no level geometry, no code for lighting or shadows, no scripted animation. Just a neural net in a loop. By "in your web browser" I mean this world runs locally, in your web browser. Once the world has loaded, you can continue exploring even in Airplane Mode. So, why bother creating a world this way? There are some interesting conceptual reasons (I'll get to them later), but my main goal was just to outdo a prior post. See, three years ago, I got a simple two-dimensional video game world to run in-browser by training a neural network to mimic gameplay videos from YouTube. Mimicking a 2D video game world was cute, but ultimately kind of pointless; existing video games already exist and we can already emulate them just fine. The wonderful, unique, exciting property of neural worlds is that they can be constructed from any video file, not just screen recordings of old video games. My previous post didn't really get this across. So for this post, to demonstrate what makes neural networks truly special, I wanted to train a neural network on gameplay videos of the actual world. Recording data To begin this project, I walked through a forest trail, recording videos with my phone, using a customized camera app which also recorded my phone's motion. I collected ~15 minutes of video and motion recordings. I've visualized motion as a "walking" control stick on the left and a "looking" control stick on the right. Back at home, I transferred the recordings to my laptop, and shuffled them into a list of (previous frame, control → next frame) pairs just like my previous game-emulation dataset. Now, all I needed to do was train a neural network to mimic the behavior of these input→output pairs. I already had ...