Semi-supervised sequence modeling for improved behavioral segmentation

Abstract

A popular approach to quantifying animal behavior from video data is through discrete behavioral segmentation, wherein video frames are labeled as containing one or more behavior classes such as walking or grooming. Sequence models learn to map behavioral features extracted from video frames to discrete behaviors, and both supervised and unsupervised methods are common. However, each approach has its drawbacks: supervised models require a time-consuming annotation step where humans must hand label the desired behaviors; unsupervised models may fail to accurately segment particular behaviors of interest. We introduce a semi-supervised approach that addresses these challenges by constructing a sequence model loss function with (1) a standard supervised loss that classifies a sparse set of hand labels; (2) a weakly supervised loss that classifies a set of easy-to-compute heuristic labels; and (3) a self-supervised loss that predicts the evolution of the behavioral features. With this approach, we show that a large number of unlabeled frames can improve supervised segmentation in the regime of sparse hand labels and also show that a small number of hand labeled frames can increase the precision of unsupervised segmentation.

Descending neuron population dynamics during odor-evoked and spontaneous limb-dependent behaviors

“…Multiview camera images ( Figure 1n ) were postprocessed using DeepFly3D ( Günel et al, 2019 ) to estimate 3D joint positions ( Lobato-Rios et al, 2022 ; Figure 1o ). These data were used to train a dilated temporal convolutional neural network (DTCN) ( Whiteway et al, 2021 ) that could accurately classify epochs of walking, resting, head (eye and antennal) grooming, front leg rubbing, and posterior movements (a grouping of rarely generated and difficult to distinguish hindleg and abdominal grooming movements) ( Figure 1p , Figure 1—figure supplement 1g ). Animals predominantly alternated between resting, walking, and head grooming with little time spent front leg rubbing or moving their posterior limbs and abdomen ( Figure 1—figure supplement 1h ).…”

“…Behaviors were classified based on limb joint angles using the approach described in Whiteway et al, 2021 . Briefly, a network was trained using 1 min of annotations for each fly and heuristic labels.…”

Descending neuron population dynamics during odor-evoked and spontaneous limb-dependent behaviors

“…Multi-view camera images (Figure 1n) were post-processed using DeepFly3D [33] to estimate 3D joint positions [40] (Figure 1o). These data were used to train a dilated temporal convolutional neural network (DTCN) [41] that could accurately classify epochs of walking, resting, head (eye and antennal) grooming, front leg rubbing, and posterior (hindleg and abdominal) movements (Figure 1p, Figure S1g). Animals predominantly alternated between resting, walking, and head grooming with little time spent front leg rubbing or moving their posterior limbs and abdomen (Figure S1h).…”

“…Behaviors were classified based on limb joint angles using the approach described in [41]. Briefly, a network was trained using 1 min of annotations for each fly and heuristic labels.…”

Coordination and persistence of aggressive visual communication in Siamese fighting fish

“…(1) no flaring; (2) partial flaring; (3) full flaring. The model is a dilated Temporal Convolutional Network (dTCN)53 , which has shown good performance on similar behavioral classification tasks28,54,55 (SFig 1F).dTCN architecture and loss: The dTCN model is composed of multiple "Dilation Blocks." Each Dilation block is composed of two 1D convolution layers (filter size of 9-time steps and 32 channels per layer), each of which is followed by a leaky ReLU activation function and weight dropout with probability p = 0.1.…”