Runway releases its first world model, adds native audio to latest video model

Summary

The race to release world models is on as AI image and video generation company Runway joins an increasing number of startups and big tech companies by launching its first one. Dubbed GWM-1, the model works through frame-by-frame prediction, creating a simulation with an understanding of physics and how the world actually behaves over time, the company said. A world model is an AI system that learns an internal simulation of how the world works so it can reason, plan, and act without needing to be trained on every scenario possible in real life. Runway, which earlier this month launched its Gen 4.5 video model that surpassed both Google and OpenAI on the Video Arena leaderboard, said its GWM-1 world model is more “general” than Google’s Genie-3 and other competitors. The firm is pitching it as a model that can create simulations to train agents in different domains like robotics and life sciences. “To build a world model. We first needed to build a really great video model. We believe that the right path to building a world model is teaching models to predict pixels directly is the best way to achieve general-purpose simulation. At sufficient scale and with the right data, you can build a model that has sufficient understanding of how the world works,” the company’s CTO Anastasis Germanidis said during the live stream. Runway released specific slants or versions to the new world model called GWM-Worlds, GWM-Robotics, and GWM-Avatars. Image Credits: RunwayImage Credits:Runway GWM-Worlds is an app for the model that lets you create an interactive project. Users can set a scene through a prompt or an image reference, and as you explore the space, the model generates the world with an understanding of geometry, physics, and lighting. The company mentioned that the simulation runs at 24 fps and 720p resolution. Runway said that while Worlds could be useful for gaming, it’s also well-positioned to teach agents how to navigate and behave in the physical world. With GWM-Rob...