2022
DOI: 10.7554/elife.76218
|View full text |Cite
|
Sign up to set email alerts
|

Selfee, self-supervised features extraction of animal behaviors

Abstract: Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream application… Show more

Help me understand this report
View preprint versions

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1
1
1

Citation Types

0
16
0

Year Published

2022
2022
2024
2024

Publication Types

Select...
4
2
1
1

Relationship

0
8

Authors

Journals

citations
Cited by 17 publications
(19 citation statements)
references
References 86 publications
0
16
0
Order By: Relevance
“…The model can then be fine-tuned with small amounts of training data to be optimized for downstream tasks. Recently, contrastive learning has been applied for feature extraction from animal videos by Jia et al[25], by performing contrastive learning on the frame images directly in similar fashion to existing work in computer vision.…”
Section: Methodsmentioning
confidence: 99%
See 1 more Smart Citation
“…The model can then be fine-tuned with small amounts of training data to be optimized for downstream tasks. Recently, contrastive learning has been applied for feature extraction from animal videos by Jia et al[25], by performing contrastive learning on the frame images directly in similar fashion to existing work in computer vision.…”
Section: Methodsmentioning
confidence: 99%
“…The model can then be fine-tuned with small amounts of training data to be optimized for downstream tasks. Recently, contrastive learning has been applied for feature extraction from animal videos by Jia et al [25], by performing contrastive learning on the frame images directly in similar fashion to existing work in computer vision. Contrastive learning's goal is to learn representations of data such that similar datapoints are close to each other, while dissimilar ones are far apart, without the need for labels [24].…”
Section: Contrastive Learningmentioning
confidence: 99%
“…This section provides a high-level overview of animal behavior classification frameworks for small laboratory animals. We outline a general taxonomy that organizes methods as supervised or unsupervised at a coarse level, and with varying degrees of supervision Taxonomy for Animal Behavior Classification Supervised Classification Hand-crafted Features, Behavior Labels [137], [33], [72], [20], [47], [79], [60], [101], [35], [45] Behavior Labels [91], [160], [63] Hand-crafted Features, Pose and Behavior Labels, PE [139], [145], [146], [4], [94], [121] Pose and Behavior Labels, PE [179] Optical Flow, Hand-crafted Features, Behavior Labels [161] Residual Learning, Optical Flow, Behavior Labels [14] Residual Learning, Pose and Behavior Labels, PE [178] Residual Learning, Optical Flow, Behavior Labels [105] Unsupervised Classification Hand-crafted Features, Pose Labels, PE [61] Fully Unsupervised [144], [11], [172], [8], [16], [73] Fig. 8.…”
Section: Taxonomy For Animal Behavior Classificationmentioning
confidence: 99%
“…This approach has the benefit of not requiring the experimenter to specify rules for deriving output labels for training, at the cost of potentially making the process more sensitive to noise and occlusions (Hausmann et al, 2021 ; Luxem et al, 2022 ). Selfee (Jia et al, 2022 ) and BehaveNet (Batty et al, 2019 ), unlike VAME and DBM, operate directly on snippets of video rather than pose data from DLC/SLEAP, employing autoencoder-style training directly on video data for nonlinear dimensionality reduction. Selfee operates on short 3-frame snippets of raw video, has been used in mice to identify behaviors such as social nose contact and allogrooming in open field tests, and places its main emphasis on using the extracted feature space for a variety of downstream analyses (Jia et al, 2022 ).…”
Section: Machine Learning Approaches For Emotional Behavioral Analysismentioning
confidence: 99%
“…Selfee (Jia et al, 2022 ) and BehaveNet (Batty et al, 2019 ), unlike VAME and DBM, operate directly on snippets of video rather than pose data from DLC/SLEAP, employing autoencoder-style training directly on video data for nonlinear dimensionality reduction. Selfee operates on short 3-frame snippets of raw video, has been used in mice to identify behaviors such as social nose contact and allogrooming in open field tests, and places its main emphasis on using the extracted feature space for a variety of downstream analyses (Jia et al, 2022 ). BehaveNet, unlike the other self-supervised methods, performs feature extraction on individual frames rather than sequences of frames, and does not consider the temporal structure of the data until the discretization step, which uses an autoregressive hidden Markov model (Batty et al, 2019 ).…”
Section: Machine Learning Approaches For Emotional Behavioral Analysismentioning
confidence: 99%