Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders

Abstract

Recent neuroscience studies in awake and behaving animals demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from this video data. In this work we introduce a new semi-supervised framework that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this method, the Partitioned Subspace Variational Autoencoder (PS-VAE), on head-fixed mouse behavioral videos. In a close up video of a mouse face, where we track pupil location and size, our method extracts unsupervised outputs that correspond to the eyelid and whisker pad positions, with no additional user annotations required. We use this resulting interpretable behavioral representation to construct saccade and whisking detectors, and quantify the accuracy with which these signals can be decoded from neural activity in visual cortex. In a two-camera mouse video we show how our method separates movements of experimental equipment from animal behavior, and extracts unsupervised features like chest position, again with no additional user annotation needed. This allows us to construct paw and body movement detectors, and decode individual features of behavior from widefield calcium imaging data. Our results demonstrate how the interpretable partitioning of behavioral videos provided by the PS-VAE can facilitate downstream behavioral and neural analyses.

Semi-supervised sequence modeling for improved behavioral segmentation

“…The results above utilize a GRU architecture; to ensure these performance gains are not architecture-dependent, we perform the same hyperparameter search over λ h and λ p using two additional architectures: a temporal convolutional network [23, 22] and an MLP neural network with an initial 1D temporal convolutional layer [3, 46] (Fig. S2).…”

Manipulation-specific activity in motor and somatosensory cortex as mice handle food

“…Regarding the behavioral analyses, most of these focused on experimenter-chosen features (holding vs oromanual modes, transport-to-mouth, lowering-from mouth, regrips), which potentially introduces bias. The GLM analysis avoided this by focusing on whole kinematic and neural traces, but newer unsupervised methods (Wiltschko et al, 2015; Batty et al, 2019; Pereira et al, 2020; Hsu and Yttri, 2021; Whiteway et al, 2021) might reveal further features or structure of food-handling behavior overlooked here.…”

Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs

“…Ultimately, such descriptions will facilitate relating complex naturalistic behaviors to the underlying neural activity patterns that generate them [3, 11]. This will provide insight into how and why behavior may differ across environmental contexts [2, 3, 11, 12], under pharmacological manipulations [6] or across health and disease [13, 14].…”

“…Just like syllables form the building blocks of spoken language, behavioral syllables may be composed to perform complex sequences of behavior. A large focus of previous work has been to discover such behavioral syllables in an unsupervised manner, thereby obtaining a low-dimensional description of high-dimensional behavior [4, 5, 12, 16–19].…”