ConstrastivePose: A contrastive learning approach for self-supervised feature engineering for pose estimation and behavorial classification of interacting animals

Abstract

In recent years, supervised machine learning models trained on videos of animals with pose estimation data and behavior labels have been used for automated behavior classification. Applications include, for example, automated detection of neurological diseases in animal models. However, there are two problems with these supervised learning models. First, such models require a large amount of labeled data but the labeling of behaviors frame by frame is a laborious manual process that is not easily scalable. Second, such methods rely on handcrafted features obtained from pose estimation data that are usually designed empirically. In this paper, we propose to overcome these two problems using contrastive learning for self-supervised feature engineering on pose estimation data. Our approach allows the use of unlabeled videos to learn feature representations and reduce the need for handcrafting of higher-level features from pose positions. We show that this approach to feature representation can achieve better classification performance compared to handcrafted features alone, and that the performance improvement is due to contrastive learning on unlabeled data rather than the neural network architecture.

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

“…This section provides a high-level overview of animal behavior classification frameworks for small laboratory animals. We outline a general taxonomy that organizes methods as supervised or unsupervised at a coarse level, and with varying degrees of supervision Taxonomy for Animal Behavior Classification Supervised Classification Hand-crafted Features, Behavior Labels [137], [33], [72], [20], [47], [79], [60], [101], [35], [45] Behavior Labels [91], [160], [63] Hand-crafted Features, Pose and Behavior Labels, PE [139], [145], [146], [4], [94], [121] Pose and Behavior Labels, PE [179] Optical Flow, Hand-crafted Features, Behavior Labels [161] Residual Learning, Optical Flow, Behavior Labels [14] Residual Learning, Pose and Behavior Labels, PE [178] Residual Learning, Optical Flow, Behavior Labels [105] Unsupervised Classification Hand-crafted Features, Pose Labels, PE [61] Fully Unsupervised [144], [11], [172], [8], [16], [73] Fig. 8.…”

“…-ConstrastivePose [179] uses contrastive learning to reduce differences in pose estimation features and its random augmented version, while increasing differences with other examples. These features have a similar structure as handcrafted features and perform comparably on semi-supervised behavior classification.…”