Fast animal pose estimation using deep neural networks

Pereira, Talmo; Aldarondo, Diego E.; Willmore, Lindsay; Kislin, Mikhail; Wang, Samuel S.‐H.; Murthy, Mala; Shaevitz, Joshua W.

doi:10.1038/s41592-018-0234-5

Cited by 521 publications

(455 citation statements)

References 52 publications

(83 reference statements)

Supporting

Mentioning

408

Contrasting

Unclassified

Order By: Relevance

“…The ability to track positions of macaques is important because of their central role in biomedical research, as well as their importance in psychology, and ethology. Recent years have witnessed the development of widely used markerless tracking systems in many species, including flies, worms, mice and rats, and humans (17,32,33). Such systems are typically not designed with the specific problems of monkey pose estimation in mind.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

OpenMonkeyStudio: Automated Markerless Pose Estimation in Freely Moving Macaques

Bala

Eisenreich

Yoo

et al. 2020

Preprint

120

View full text Add to dashboard Cite

The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and several fields of medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure its behavior, specifically, its pose (position of multiple major body landmarks) in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a novel deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 precisely calibrated and synchronized machine vision cameras that encircle an open 2.45m×2.45m×2.75m enclosure. The resulting multiview image streams allow for novel data augmentation via 3D reconstruction of hand-annotated images that in turn train a robust view-invariant deep neural network model. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track two monkey social interactions without human intervention. We also make the training data (195,228 images) and trained detection model publicly available. deep learning | behavioral tracking | rhesus macaque | convolutional pose machineCorrespondence: Benjamin Hayden,

show abstract

Section: Discussionmentioning

confidence: 99%

“…In practice, we leverage a commercial annotation service (Hive AI). As of January 2020, 33,192 images are annotated.…”

Section: Keyframe Selection For Maximally Informative Posesmentioning

confidence: 99%

OpenMonkeyStudio: Automated Markerless Pose Estimation in Freely Moving Macaques

Bala

Eisenreich

Yoo

et al. 2020

Preprint

120

View full text Add to dashboard Cite

show abstract

“…Manually tracking the limbs in 8000+ videos is not feasible. Instead, we used a recent deep-learning approach, implemented in Matlab and Python [34]. The LEAP deep-learning workflow tracks a set of user-defined points on square videos centred on a subject.…”

Section: Tracking: 'Leap' Deep-learning Trackingmentioning

confidence: 99%

Uneven substrates constrain walking speed in ants through modulation of stride frequency more than stride length

2020

View full text Add to dashboard Cite

Natural terrain is rarely flat. Substrate irregularities challenge walking animals to maintain stability, yet we lack quantitative assessments of walking performance and limb kinematics on naturally uneven ground. We measured how continually uneven 3D-printed substrates influence walking performance of Argentine ants by measuring walking speeds of workers from laboratory colonies and by testing colony-wide substrate preference in field experiments. Tracking limb motion in over 8000 videos, we used statistical models that associate walking speed with limb kinematic parameters to compare movement over flat versus uneven ground of controlled dimensions. We found that uneven substrates reduced preferred and peak walking speeds by up to 42% and that ants actively avoided uneven terrain in the field. Observed speed reductions were modulated primarily by shifts in stride frequency instead of stride length (flat R 2 : 0.91 versus 0.50), a pattern consistent across flat and uneven substrates. Mixed effect modelling revealed that walking speeds on uneven substrates were accurately predicted based on flat walking data for over 89% of strides. Those strides that were not well modelled primarily involved limb perturbations, including missteps, active foot repositioning and slipping. Together these findings relate kinematic mechanisms underlying walking performance on uneven terrain to ecologically relevant measures under field conditions.

show abstract

“…How much training is needed depends on the algorithm. Deep learning approaches intended for visual data of humans for example usually need very large training sets because humans look quite different from one another, says Talmo Pereira, a PhD student at Princeton University who has been working on the LEAP pose tracking tool 8 with Mala Murthy and Joshua Shaevitz' labs. Most lab animals are much more homogenous in both appearance and behavior, meaning a neural network can rise to par with a smaller training set.…”

Section: Under Supervisionmentioning

confidence: 99%