A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Mathis, Alexander; Schneider, Steffen; Lauer, Jessy

doi:10.1016/j.neuron.2020.09.017

Cited by 171 publications

(186 citation statements)

References 168 publications

Supporting

Mentioning

163

Contrasting

Unclassified

Order By: Relevance

“…Thus far we have used to TREBLE to analyze behaviors as changes in centroid velocity components. However, pose estimation methods have made analyses of other behavioral features, such as posture or limb movement, increasingly common (Pereira et al 2020; Mathis et al 2020). As such, pose estimation methods typically represent behavior in multi-dimensional spaces that may or may not include explicit velocities.…”

Section: Resultsmentioning

confidence: 99%

“…Advances in tracking technology have enabled measurements of animal movement from a wide range of species (Datta et al 2019; Pereira et al 2020; Mathis et al 2020). These datasets are often extremely rich, and include correlated movements across timescales, features that must be accounted for in efforts to link neural activity to behavior.…”

Section: Mainmentioning

confidence: 99%

“…These datasets are often extremely rich, and include correlated movements across timescales, features that must be accounted for in efforts to link neural activity to behavior. In parallel, a variety of sophisticated methods have emerged to parse such behavioral structure and to measure behavioral changes caused by ever more powerful experimental perturbations (Brown & de Bivort 2018; Datta et al 2019; Pereira et al 2020; Mathis et al 2020). However, many of these methods are complex to implement, and require extensive adaptation to specific species, contexts and experimental goals.…”

Section: Mainmentioning

confidence: 99%

See 2 more Smart Citations

Flexible analysis of animal behavior via time-resolved manifold embedding

York

Carreira-Rosario

Giocomo

et al. 2020

Preprint

View full text Add to dashboard Cite

Understanding the relationships between neural activity and behavior represents a critical challenge, one that requires generalizable statistical tools that can capture complex structures within large datasets. We developed Time-REsolved BehavioraL Embedding (TREBLE), a flexible method for analyzing behavioral data from freely moving animals. Using data from synthetic trajectories, fruit flies, and mice we show how TREBLE can capture both continuous and discrete behavioral dynamics, can uncover behavioral variation across individuals, and can detect the effects of optogenetic perturbation in an unbiased fashion. By applying TREBLE to the freely moving mouse, and medial entorhinal cortex (MEC) recordings, we show that nearly all MEC neurons encode information relevant to specific movement patterns, expanding our understanding of how navigation is related to the execution of locomotion. Thus, TREBLE provides a flexible framework for describing the structure of complex behaviors and their relationships to neural activity.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Mainmentioning

confidence: 99%

Section: Mainmentioning

confidence: 99%

See 1 more Smart Citation

Flexible analysis of animal behavior via time-resolved manifold embedding

York

Carreira-Rosario

Giocomo

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, advances in animal pose estimation–the ability to measure the geometric configuration of user-specified keypoints–have ushered in an era of high-throughput quantitative analysis of movements ( Mathis and Mathis, 2020 ). One such state-of-the-art animal pose estimation package, DeepLabCut (DLC; Mathis et al, 2018b ), provides tailored networks that predict the posture of animals of interest based on video frames, and can run swiftly in offline batch processing modes (up to 2,500 FPS on standard GPUs; Mathis et al, 2020a ; Mathis and Warren, 2018 ). This high throughput analysis has proven to be an invaluable tool to probe the neural mechanisms of behavior ( Mathis and Mathis, 2020 ; von Ziegler et al, 2020 ; Mathis et al, 2020b ).…”

Section: Introductionmentioning

confidence: 99%

“…One such state-of-the-art animal pose estimation package, DeepLabCut (DLC; Mathis et al, 2018b ), provides tailored networks that predict the posture of animals of interest based on video frames, and can run swiftly in offline batch processing modes (up to 2,500 FPS on standard GPUs; Mathis et al, 2020a ; Mathis and Warren, 2018 ). This high throughput analysis has proven to be an invaluable tool to probe the neural mechanisms of behavior ( Mathis and Mathis, 2020 ; von Ziegler et al, 2020 ; Mathis et al, 2020b ). The ability to apply these behavioral analysis tools to provide feedback to animals in real time is crucial for causally testing the behavioral functions of specific neural circuits.…”

Section: Introductionmentioning

confidence: 99%

Real-time, low-latency closed-loop feedback using markerless posture tracking

Kane

Lopes²,

Saunders

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are (1) noninvasive, (2) low-latency, and (3) provide interfaces to trigger external hardware based on posture (i.e., not just objectbased-tracking). Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. In extending our efforts towards the animal pose estimation toolbox DeepLabCut, here, we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, at >100 FPS), with an additional forwardprediction module that achieves zero-latency feedback. We also provide three options for using this tool with ease: a stand-alone GUI (called DLC-Live!GUI), integration into Bonsai and into AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.HighlightsThe DeepLabCut-Live! package is available via pip install deeplabcut-liveThe Bonsai-DLC plugin is availableThe AutoPilot-DLC plugin is availableThe DeepLabCut-Live! GUI package is available via pip install deeplabcut-live-gui

show abstract

A Time‐Division Position‐Sensitive Detector Image System for High‐Speed Multitarget Trajectory Tracking

et al. 2022

View full text Add to dashboard Cite

High‐speed trajectory tracking with real‐time processing capability is particularly important in the fields of pilotless automobiles, guidance systems, robotics, and filmmaking. The conventional optical approach to high‐speed trajectory tracking involves charge coupled device (CCD) or complementary metal–oxide–semiconductor (CMOS) image sensors, which suffer from trade‐offs between resolution and framerates, complexity of the system, and enormous data‐analysis processes. Here, a high‐speed trajectory tracking system is designed by using a time‐division position‐sensitive detector (TD‐PSD) based on a graphene–silicon Schottky heterojunction. Benefiting from the high‐speed optoelectronic response and sub‐micrometer positional accuracy of the TD‐PSD, multitarget real‐time trajectory tracking is realized, with a maximum image output framerate of up to 62 000 frames per second. Moreover, multichannel trajectory tracking and image‐distortion correction functionalities are realized by TD‐PSD systems through frequency‐related image preprocessing, which significantly improves the capacity of real‐time information processing and image quality in complicated light environments.

show abstract

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Cited by 171 publications

References 168 publications

Flexible analysis of animal behavior via time-resolved manifold embedding

Flexible analysis of animal behavior via time-resolved manifold embedding

Real-time, low-latency closed-loop feedback using markerless posture tracking

A Time‐Division Position‐Sensitive Detector Image System for High‐Speed Multitarget Trajectory Tracking

Contact Info

Product

Resources

About