Learning Intermediate-Level Representations of Form and Motion from Natural Movies

Abstract

We present a model of intermediate-level visual representation that is based on learning invariances from movies of the natural environment.The model is composed of two stages of processing: an early feature representation layer and a second layer in which invariances are explicitly represented. Invariances are learned as the result of factoring apart the temporally stable and dynamic components embedded in the early feature representation. The structure contained in these components is made explicit in the activities of second-layer units that capture invariances in both form and motion. When trained on natural movies, the first layer produces a factorization, or separation, of image content into a temporally persistent part representing local edge structure and a dynamic part representing local motion structure, consistent with known response properties in early visual cortex (area V1). This factorization linearizes statistical dependencies among the first-layer units, making them learnable by the second layer. The second-layer units are split into two populations according to the factorization in the first layer. The form-selective units receive their input from the temporally persistent part (local edge structure) and after training result in a diverse set of higher-order shape features consisting of extended contours, multiscale edges, textures, and texture boundaries. The motion-selective units receive their input from the dynamic part (local motion structure) and after training result in a representation of image translation over different spatial scales and directions, in addition to more complex deformations. These representations provide a rich description of dynamic natural images and testable hypotheses regarding intermediate-level representation in visual cortex.

Event Segmentation and Biological Motion Perception in Watching Dance

“…While such ambiguity is fundamental to many visual processes (Koenderink and Van Doorn, 1997) it is acute in the case of silhouettes where local deformation of the silhouette depends upon motion and the curvature of the surface in complex ways. Our calculation of a Motion Index does not decompose changes of the silhouette into different factors of form and motion (Cadieu and Olshausen, 2012;Tomasi and Kanade, 1992). Thus, the brain activity related to our motion index might be due to motion features, form features or the process of factorizing the image motion into form and motion features.…”

A three-layer model of natural image statistics

“…One powerful class of models specifies the distribution in a top-down manner in terms of latent variables which explain the structure in the visual stimuli (Olshausen and Field, 1996;Hyvärinen et al, 2009;Karklin and Lewicki, 2009;Zoran and Weiss, 2009;Ranzato and Hinton, 2010;Cadieu and Olshausen, 2012). Another class of models corresponds to a bottom-up approach where the visual stimuli are processed in multiple layers of computation (Osindero et al, 2006;Köster and Hyvärinen, 2010;Gutmann and Hyvärinen, 2012b).…”

Dynamic Filtering of Time-Varying Sparse Signals via <inline-formula> <tex-math notation="LaTeX">$\ell _1$</tex-math> </inline-formula> Minimization

“…These results will report in-depth comparison of a single challenging video sequence (the Foreman sequence 2 ) as well as aggregate statistics from a batch of video from a BBC nature documentary (as used in [52]). The documentary footage is valuable as broad comparison because it contains many different types of motion, including static frames with localized changes and highly dynamic frames with moving subjects across large portions of the visual field.…”