Playable Environments: Video Manipulation in Space and Time

Plotting Behind the Scenes: Towards Learnable Game Engines

“…Many learnable methods focus on reducing the need for manual labor and 3D assets in computer graphics [Holden et al 2017;Kuang et al 2022;Liu et al 2021;Starke et al 2019Starke et al , 2020, but only provide narrow video game functions. More related to our work, neural video game simulation methods show that annotated videos can be used to learn to generate videos interactively [Davtyan and Favaro 2022;Kim et al , 2020Menapace et al 2021] and build 3D environments where agents can be controlled through a set of discrete actions [Menapace et al 2022]. While bringing us closer to learnable game engines, when applied to complex or realworld environments, these works have several limitations: do not accurately model game logic, do not model physical interactions of objects in 3D space, do not learn fine-grained controls, do not allow for high-level goal-driven control of the game flow, and, finally, do not model game AI.…”

“…To overcome the limitations of [Davtyan and Favaro 2022;Kim et al , 2020Menapace et al 2021Menapace et al , 2022, not only we model the states of an environment, but we also consider detailed textual representations of the actions taking place in it. We argue that training on user commentaries describing detailed actions 1 Unreal and Unity engines are used to photorealistically render environments for film production.…”

“…1. Our synthesis model maintains a state for every object and agent included in the game and renders them in the image space using the compositional NeRF [Mildenhall et al 2020] of [Menapace et al 2022] followed by a learnable enhancer for superior rendering quality. To model the logic of games and game AI that determine the evolution of the environment states, we introduce an animation model.…”

“…In particular, we show that using text labels describing actions happening in a game is instrumental in learning such capabilities. While certain prior work [Kim et al , 2020Menapace et al 2021Menapace et al , 2022 explored maintaining and rendering states of games, we are not aware of any generative method that attempts enabling fine-grained control, modeling sophisticated goal-driven game logic, and learning game AI to the extent explored in this paper.…”

“…As far as we are aware, no existing work provides this set of capabilities under comparable data assumptions. • A synthesis model, based on a compositional NeRF, producing videos at the original frame rate and doubling the output resolution with respect to [Menapace et al 2022]. • An animation model, based on a text-conditioned diffusion model with a masked training procedure, which is key to support complex game logic, object interactions, game AI, and understanding fine-grained actions.…”