OptiFlex: video-based animal pose estimation using deep learning enhanced by optical flow

Liu, Xiaoyu; Shao, Yu; Flierman, Nico A.; Loyola, Sebastián; Kamermans, Maarten; Hoogland, Tycho M.; Ci, De Zeeuw

doi:10.1101/2020.04.04.025494

Cited by 16 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The expectations âi c and ĝi c can be derived considering the distributions shown in Eqs. (18) and (19) and the potential functions defined in Eqs. (7), ( 8) and ( 9):…”

Section: A Structured Context Mixermentioning

confidence: 99%

Structured Context Enhancement Network for Mouse Pose Estimation

Zhou

Jiang

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

Automated analysis of mouse behaviours is crucial for many applications in neuroscience. However, quantifying mouse behaviours from videos or images remains a challenging problem, where pose estimation plays an important role in describing mouse behaviours. Although deep learning based methods have made promising advances in mouse or other animal pose estimation, they cannot properly handle complicated scenarios (e.g., occlusions, invisible keypoints, and abnormal poses). Particularly, since mouse body is highly deformable, it is a big challenge to accurately locate different keypoints on the mouse body. In this paper, we propose a novel hourglass network based model, namely Graphical Model based Structured Context Enhancement Network (GM-SCENet) where two effective modules, i.e., Structured Context Mixer (SCM) and Cascaded Multi-Level Supervision module (CMLS) are designed. The SCM can adaptively learn and enhance the proposed structured context information of each mouse part by a novel graphical model with close consideration on the difference between body parts. Then, the CMLS module is designed to jointly train the proposed SCM and the hourglass network by generating multi-level information, which increases the robustness of the whole network. Based on the multi-level predictions from the SCM and the CMLS module, we also propose an inference method to enhance the localization results. Finally, we evaluate our proposed approach against several baselines on our Parkinson's Disease Mouse Behaviour (PDMB) and the standard DeepLabCut Mouse Pose datasets, where the results show that our method can achieve better or competitive performance against the other state-of-theart approaches.

show abstract

“…The expectations âi c and ĝi c can be derived considering the distributions shown in Eqs. (18) and (19) and the potential functions defined in Eqs. (7), ( 8) and ( 9):…”

Section: A Structured Context Mixermentioning

confidence: 99%

Structured Context Enhancement Network for Mouse Pose Estimation

Zhou

Jiang

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Thus, pre-trained pose estimation algorithms save training time, increase robustness, and require substantially less training data. Indeed, most packages in Neuroscience now use pre-trained models (20,(40)(41)(42)(43)(44), although some do not (45)(46)(47), which can give acceptable performance for simplified situations with aligned individuals.…”

Section: Datasets and Data Augmentationmentioning

confidence: 99%

“…Indeed, the field has been able to rapidly adopt these tools for neuroscience. Deep learning-based markerless pose estimation applications in the laboratory have already been published for flies (20,41,43,45,70,85), rodents (20,40,41,43,45,47,70,87), horses (39), dogs (74), rhesus macaque (42,74,88,89) and marmosets (90); the original architectures were developed for humans (18,26,27). Outside of the laboratory, DeepPoseKit was used for zebras (41) and DeepLabCut for 3D tracking of cheetahs ( 80), for squirrels (91) and macaques (89), highlighting the great "in-thewild" utility of this new technology (10).…”

Section: Scope and Applicationsmentioning

confidence: 99%

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Mathis,

Schneider,

Lauer

et al. 2020

Preprint

View full text Add to dashboard Cite

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.

show abstract

“…For motion capturing of arbitrary animal movement sequences, DeepLabCut appends a stack of de-convolutional layers that can be trained in an end-to-end manner. Other deep neural network applications for motion analysis have focused on particular aspects of this approach, such as iterative improvement by manual re-labeling of pose estimates (Pereira et al, 2019 ), or exploiting movement information from subsequent frames (Liu et al, 2020 ). In all of these approaches, the output of the system is a 2D map of probabilities—so-called score maps or confidence maps—that indicate both the most likely position estimate of a particular body part and a measure of confidence of that estimate.…”

Section: Introductionmentioning

confidence: 99%

Marker-Less Motion Capture of Insect Locomotion With Deep Neural Networks Pre-trained on Synthetic Videos

Arent

Schmidt

Botsch

et al. 2021

Front. Behav. Neurosci.

View full text Add to dashboard Cite

Motion capture of unrestrained moving animals is a major analytic tool in neuroethology and behavioral physiology. At present, several motion capture methodologies have been developed, all of which have particular limitations regarding experimental application. Whereas marker-based motion capture systems are very robust and easily adjusted to suit different setups, tracked species, or body parts, they cannot be applied in experimental situations where markers obstruct the natural behavior (e.g., when tracking delicate, elastic, and/or sensitive body structures). On the other hand, marker-less motion capture systems typically require setup- and animal-specific adjustments, for example by means of tailored image processing, decision heuristics, and/or machine learning of specific sample data. Among the latter, deep-learning approaches have become very popular because of their applicability to virtually any sample of video data. Nevertheless, concise evaluation of their training requirements has rarely been done, particularly with regard to the transfer of trained networks from one application to another. To address this issue, the present study uses insect locomotion as a showcase example for systematic evaluation of variation and augmentation of the training data. For that, we use artificially generated video sequences with known combinations of observed, real animal postures and randomized body position, orientation, and size. Moreover, we evaluate the generalization ability of networks that have been pre-trained on synthetic videos to video recordings of real walking insects, and estimate the benefit in terms of reduced requirement for manual annotation. We show that tracking performance is affected only little by scaling factors ranging from 0.5 to 1.5. As expected from convolutional networks, the translation of the animal has no effect. On the other hand, we show that sufficient variation of rotation in the training data is essential for performance, and make concise suggestions about how much variation is required. Our results on transfer from synthetic to real videos show that pre-training reduces the amount of necessary manual annotation by about 50%.

show abstract

OptiFlex: video-based animal pose estimation using deep learning enhanced by optical flow

Cited by 16 publications

References 19 publications

Structured Context Enhancement Network for Mouse Pose Estimation

Structured Context Enhancement Network for Mouse Pose Estimation

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Marker-Less Motion Capture of Insect Locomotion With Deep Neural Networks Pre-trained on Synthetic Videos

Contact Info

Product

Resources

About