OptiFlex: Multi-Frame Animal Pose Estimation Combining Deep Learning With Optical Flow

Liu, Xiaole; Yu, Siyang; Flierman, Nico A.; Loyola, Sebastián; Kamermans, Maarten; Hoogland, Tycho M.; Zeeuw, Chris I. De

doi:10.3389/fncel.2021.621252

Cited by 22 publications

(7 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The 3D poses can be inferred from these 2D features by means of classical calibrated camera setups 59 ; however the 2D detection in one camera image does not benefit from the information from other cameras and the triangulation may suffer from resulting mislabeling of 2D features as well as missing detections due to occluded features. A recent approach 38,60 overcomes many of these issues by mapping from recorded images directly to 3D feature locations, again using deep learning, and is capable of classifying animal behaviors across many species 38 . An alternative approach is to use measurements that directly yield 3D information, for example RGBD 61 .…”

Section: Discussionmentioning

confidence: 99%

“…An alternative approach is to use measurements that directly yield 3D information, for example RGBD 61 . In parallel, there has been substantial developments in pose estimation of humans, including the possibility to track multiple individuals in real time 60,[62][63][64] , some of which include explicit models of kinematics 65,66 . In general, these approaches triangulate joint positions of readily detectable key points in the images, which has the advantage of not requiring application of surface markers.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Estimation of skeletal kinematics in freely moving rodents

Monsees¹,

Voit²,

Wallace³

et al. 2022

Nat Methods

View full text Add to dashboard Cite

Forming a complete picture of the relationship between neural activity and skeletal kinematics requires quantification of skeletal joint biomechanics during free behavior; however, without detailed knowledge of the underlying skeletal motion, inferring limb kinematics using surface-tracking approaches is difficult, especially for animals where the relationship between the surface and underlying skeleton changes during motion. Here we developed a videography-based method enabling detailed three-dimensional kinematic quantification of an anatomically defined skeleton in untethered freely behaving rats and mice. This skeleton-based model was constrained using anatomical principles and joint motion limits and provided skeletal pose estimates for a range of body sizes, even when limbs were occluded. Model-inferred limb positions and joint kinematics during gait and gap-crossing behaviors were verified by direct measurement of either limb placement or limb kinematics using inertial measurement units. Together we show that complex decision-making behaviors can be accurately reconstructed at the level of skeletal kinematics using our anatomically constrained model.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Estimation of skeletal kinematics in freely moving rodents

Monsees¹,

Voit²,

Wallace³

et al. 2022

Nat Methods

View full text Add to dashboard Cite

show abstract

“…To quantify the performance of markerless pose-estimation, automatically determined marker positions are needed to be compared against labels determined in another way, usually from manual labelling. To compare between evaluation metrics, the literature has established a trustable criterion, called the average Percentage of Correct Keypoints (aPCK) [39, 47, 48]. The aPCK approach needs a human assessor to manually annotate a validation set (on the top of training/test set).…”

Section: Methods and Computational Approachmentioning

confidence: 99%

Deep-Learning for Automated Markerless Tracking of Infants General Movements

Abbasi

Mollet

Williams

et al. 2022

Preprint

View full text Add to dashboard Cite

The presence of abnormal infant General Movements (GMs) is a strong predictor of progressive neurodevelopmental disorders, including cerebral palsy (CP). Automation of the assessment will overcome scalability barriers that limit its delivery to at risk individuals. Here, we report a robust markerless pose estimation scheme, based on advanced deep learning technology, to track infant movements in consumer mobile device video recordings. Two deep neural network models, namely Efficientnetb6 and resnet152, were trained on manually annotated data across twelve anatomical locations (3 per limb) in 12 videos from 6 full term infants (mean age = 17.33 (SD 2.9) wks, 4 male, 2 female), using the DeepLabCut framework. Kfold cross validation indicates the generalization capability of the deep nets for GM tracking on out of domain data with an overall performance of 95.52% (SD 2.43) from the best performing model (Efficientnetb6) across all infants (performance range: 84.32 to 99.24% across all anatomical locations). The paper further introduces an automatic, unsupervised strategy for performance evaluation on extensive out of domain recordings through a fusion of likelihoods from a Kalman filter and the deep net. Findings indicate the possibility of establishing an automated GM tracking platform, as a suitable alternative to, or support for, the current observational protocols for early diagnosis of neurodevelopmental disorders in early infancy.

show abstract

“…Hahn-Klimroth et al [24] presented a multistep CNN system to detect three typical African ungulate stances in zoo enclosures, including model averaging and postprocessing rules to make the system robust to outliers. Liu et al [25] used a ResNet backbone, three transposed convolution layers, and a final output layer to estimate the pose. The model is evaluated using data sets from four different animal species (mouse, fruit fly, zebrafish, and monkey).…”

Section: Related Workmentioning

confidence: 99%

Markerless Dog Pose Recognition in the Wild Using ResNet Deep Learning Model

2021

View full text Add to dashboard Cite

The analysis and perception of behavior has usually been a crucial task for researchers. The goal of this paper is to address the problem of recognition of animal poses, which has numerous applications in zoology, ecology, biology, and entertainment. We propose a methodology to recognize dog poses. The methodology includes the extraction of frames for labeling from videos and deep convolutional neural network (CNN) training for pose recognition. We employ a semi-supervised deep learning model of reinforcement. During training, we used a combination of restricted labeled data and a large amount of unlabeled data. Sequential CNN is also used for feature localization and to find the canine’s motions and posture for spatio-temporal analysis. To detect the canine’s features, we employ image frames to locate the annotations and estimate the dog posture. As a result of this process, we avoid starting from scratch with the feature model and reduce the need for a large dataset. We present the results of experiments on a dataset of more than 5000 images of dogs in different poses. We demonstrated the effectiveness of the proposed methodology for images of canine animals in various poses and behavior. The methodology implemented as a mobile app that can be used for animal tracking.

show abstract

OptiFlex: Multi-Frame Animal Pose Estimation Combining Deep Learning With Optical Flow

Cited by 22 publications

References 37 publications

Estimation of skeletal kinematics in freely moving rodents

Estimation of skeletal kinematics in freely moving rodents

Deep-Learning for Automated Markerless Tracking of Infants General Movements

Markerless Dog Pose Recognition in the Wild Using ResNet Deep Learning Model

Contact Info

Product

Resources

About