iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

Abstract

How would a static scene react to a local poke? What are the effects on other parts of an object if you could locally push it? There will be distinctive movement, despite evident variations caused by the stochastic nature of our world. These outcomes are governed by the characteristic kinematics of objects that dictate their overall motion caused by a local interaction. Conversely, the movement of an object provides crucial information about its underlying distinctive kinematics and the interdependencies between its parts. This two-way relation motivates learning a bijective mapping between object kinematics and plausible future image sequences. Therefore, we propose iPOKE -invertible Prediction of Object Kinematics -that, conditioned on an initial frame and a local poke, allows to sample object kinematics and establishes a one-to-one correspondence to the corresponding plausible videos, thereby providing a controlled stochastic video synthesis. In contrast to previous works, we do not generate arbitrary realistic videos, but provide efficient control of movements, while still capturing the stochastic nature of our environment and the diversity of plausible outcomes it entails. Moreover, our approach can transfer kinematics onto novel object instances and is not confined to particular object classes. Our project page is available at https://bit.ly/3dJN4Lf.

Controllable Animation of Fluid Elements in Still Images

“…These methods [4,12,15,15,16,31] generate a video from a single still image automatically and thus do not allow the user interaction to control the animation. Different from the above set of works, [1,2,5,6,10,11] allow the users to interact and control the movement in the animation to varying degrees, and hence are more closely related to the current work. Dorkenwald et al [5] propose a oneto-one mapping between image and video using a residual representation, that allows the user to provide a single direction of motion for video generation.…”

“…Dorkenwald et al [5] propose a oneto-one mapping between image and video using a residual representation, that allows the user to provide a single direction of motion for video generation. [1] and [2] propose methods that govern the animation of different parts in the image with a single poke at a particular location defined by the start and end location of the motion. However, these methods [1,2,5] are unsuitable for our problem that necessitates the use of a sparse set of input directions and speeds at arbitrary locations.…”

“…[1] and [2] propose methods that govern the animation of different parts in the image with a single poke at a particular location defined by the start and end location of the motion. However, these methods [1,2,5] are unsuitable for our problem that necessitates the use of a sparse set of input directions and speeds at arbitrary locations. The closest approach to our work is [11].…”

“…There has been a rich body of work [1, 3, 5, 7, 10-12, 17, 21, 26] that has focused on generating animations from still images. While [3,7,12,17,21,26] focus on uncontrollable image-to-video synthesis, attempts [1,2,5,10,11] have been made for controllable image-to-video synthesis with the user-provided direction of the motion of the objects in the images. While these methods provide some control to the user, they suffer from certain drawbacks.…”

“…While these methods provide some control to the user, they suffer from certain drawbacks. Specifically, [1,2,5] either allow the user to poke at just a single pixel location or provide a single user direction. Halperin et.…”