OpenMonkeyStudio: Automated Markerless Pose Estimation in Freely Moving Macaques

Abstract

The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and several fields of medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure its behavior, specifically, its pose (position of multiple major body landmarks) in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a novel deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 precisely calibrated and synchronized machine vision cameras that encircle an open 2.45m×2.45m×2.75m enclosure. The resulting multiview image streams allow for novel data augmentation via 3D reconstruction of hand-annotated images that in turn train a robust view-invariant deep neural network model. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track two monkey social interactions without human intervention. We also make the training data (195,228 images) and trained detection model publicly available. deep learning | behavioral tracking | rhesus macaque | convolutional pose machineCorrespondence: Benjamin Hayden,

SLEAP: Multi-animal pose tracking

“…DeepLabCut, LEAP, and DeepPoseKit, have all been adapted to perform multi-animal pose estimation in a top-down framework, but the process of identifying animal instances must be performed separately. More recent work using 3D data also employed top-down approaches for multi-animal pose but only in single species or specialized experimental conditions [5,15,12]. Other work has focused on bottom-up approaches using rodents, but these methods have not been shown to generalize to other types of animals [3,19].…”

Real-time, low-latency closed-loop feedback using markerless posture tracking

“…If neuroscience's embrace of studying the brain in its natural context of complex, contingent, and open-ended behavior (8,38,39) is smashing the champagne on a long-delayed voyage, the technical complexity of the experiments is the grim spectre of the sea. Markerless tracking has already enabled a qualitatively new class of data-dense behavioral experiments, but the heroism required to simultaneously record natural behavior from 62 cameras (40) or electrophysiology from 65,000 electrodes (41), or integrate dozens of heterogeneous custom-built components (42) hints that a central challenge facing neuroscience is scale.…”

Non-linear Dimensionality Reduction on Extracellular Waveforms Reveals Cell Type Diversity in Premotor Cortex

“…These issues led us to investigate techniques that don't require feature specification but are designed to find patterns in complex datasets through non-linear dimensionality reduction. Such methods have seen usage in diverse neuroscientific contexts such as single-cell transcriptomics 37,38 , in analyzing models of biological neural networks 39,40 , the identification of behavior [41][42][43] , and in electrophysiology [44][45][46][47] .…”