Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Weinreb, Caleb; Osman, Mohammed Abdal Monium; Zhang, Libby; Lin, Sherry; Pearl, Jonah E; Annapragada, Sidharth; Conlin, Eli Benjamin; Gillis, Winthrop F.; Jay, Maya; Ye, Shaokai; Mathis, Alexander; Mathis, Mackenzie Weygandt; Pereira, Talmo; Linderman, Scott W.; Datta, Sandeep Robert

doi:10.1101/2023.03.16.532307

Cited by 48 publications

(42 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the future, we hope to expand our analyses of this extensive dataset to include identification of joint Rastermap/BSOiD (or keypoint-MOSEQ motif “syllable”; Weinreb et al, 2023; bioRxiv) transitions, and to perform further precision neurobehavioral alignment, using other methodologies such as hierarchical state-space models (Lindermann, S; https://github.com/lindermanlab/ssm), and factorial HMMs (Ghahramani and Jordan, 1997). These methods will allow increased accuracy over multiple behavioral timescales and the ability to predict transition or change points between different behavioral and/or neural activity states.…”

Section: Discussionmentioning

confidence: 99%

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

Vickers,

McCormick

2023

Preprint

View full text Add to dashboard Cite

The flow of neural activity across the neocortex during active sensory discrimination is constrained by task-specific cognitive demands, movements, and internal states. During behavior, the brain appears to sample from a broad repertoire of activation motifs. Understanding how these patterns of local and global activity are selected in relation to both spontaneous and task-dependent behavior requires in-depth study of densely sampled activity at single neuron resolution across large regions of cortex. In a significant advance toward this goal, we developed procedures to record mesoscale 2-photon Ca2+ imaging data from two novel in vivo preparations that, between them, allow simultaneous access to nearly all of the mouse dorsal and lateral neocortex. As a proof of principle, we aligned neural activity with both behavioral primitives and high-level motifs to reveal the existence of large populations of neurons that coordinated their activity across cortical areas with spontaneous changes in movement and/or arousal. The methods we detail here facilitate the identification and exploration of widespread, spatially heterogeneous neural ensembles whose activity is related to diverse aspects of behavior.

show abstract

Section: Discussionmentioning

confidence: 99%

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

Vickers,

McCormick

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We recommend that video be recorded for all trials if only to allow retrospective analysis of those data in the future. There has been an explosion of AI-based methods for extracting key points on animals (Graving et al, 2019;Mathis et al, 2018;Pereira et al, 2022) and algorithms for extracting higher-level animal behaviors from key point (Hsu & Yttri, 2021;Luxem et al, 2022;Weinreb et al, 2023) or raw pixel (Bohnslav et al, 2021) data, and rapid advances are likely to continue. By capturing video before and after all stimuli, users can incorporate their own AI-based methods or manually score behaviors for consideration above and beyond reflex measurements.…”

Section: Discussionmentioning

confidence: 99%

Reproducible and fully automated testing of nocifensive behavior in mice

Dedek

Azadgoleh

Prescott

2023

Preprint

View full text Add to dashboard Cite

Pain in rodents is often inferred from their withdrawal to noxious stimulation, using the threshold stimulus intensity or response latency to quantify pain sensitivity. This usually involves applying stimuli by hand and measuring responses by eye, which limits reproducibility and throughput to the detriment of preclinical pain research. Here, we describe a device that standardizes and automates pain testing by providing computer-controlled aiming, stimulation, and response measurement. Optogenetic and thermal stimuli are applied to the hind paw using blue and infrared light, respectively. Red light delivered through the same light path assists with aiming, and changes in its reflectance off the paw are used to measure paw withdrawal latency with millisecond precision at a fraction of the cost and data processing associated with high-speed video. Using standard video, aiming was automated by training a neural network to recognize the paws and move the stimulator using motorized linear actuators. Real-time data processing allows for closed-loop control of stimulus initiation and termination. We show that stimuli delivered with this device are significantly less variable than hand-delivered stimuli, and that reducing stimulus variability is crucial for resolving stimulus-dependent variations in withdrawal. Slower stimulus waveforms whose stable delivery is made possible with this device reveal details not evident with typical photostimulus pulses. Moreover, the substage video reveals a wealth of 'spontaneous' behaviors occurring before and after stimulation that can considered alongside withdrawal metrics to better assess the pain experience. Automation allows comprehensive testing to be standardized and carried out efficiently.

show abstract

“…To compare SaLSa's performance with an existing model, keypoint-MoSeq (Weinreb et al, 2023) was chosen for the following reasons: first, it overperformed other major models, such as B-SOiD (Hsu and Yttri, 2021) and VAME (Luxem et al, 2022). Second, the implementation is straightforward with a limited set of parameters to explore.…”

Section: Benchmark Testingmentioning

confidence: 99%

“…Over the last decade, a range of approaches has been developed to classify behavioral syllables (Kabra et al, 2013;Pereira et al, 2020;Wiltschko et al, 2020;Dunn et al, 2021;Hsu and Yttri, 2021;Segalin et al, 2021;Jia et al, 2022;Luxem et al, 2022;Harris et al, 2023;Luxem et al, 2023;Weinreb et al, 2023). Since these approaches including SaLSa are applied after body parts detection, they can be applied to videos taken in relatively dark environments as long as body parts are detected reliably.…”

Section: Comparisons To Other Approachesmentioning

confidence: 99%

“…The first category is a top-down approach, which involves pre-defining a set of rules to identify behavioral syllables or applying supervised machine learning to classify them (Kabra et al, 2013;Segalin et al, 2021;Harris et al, 2023). The second category is a bottom-up approach, which involves analyzing data patterns using unsupervised classification algorithms (Dunn et al, 2021;Hsu and Yttri, 2021;Gabriel et al, 2022;Jia et al, 2022;Luxem et al, 2022;Weinreb et al, 2023). Either approach has its own advantages and disadvantages.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SaLSa: a combinatory approach of semi-automatic labeling and long short-term memory to classify behavioral syllables

Sakata

2023

Preprint

View full text Add to dashboard Cite

Accurately and quantitatively describing mouse behavior is an important area. Although advances in machine learning have made it possible to track their behaviors accurately, reliable classification of behavioral sequences or syllables remains a challenge. In this study, we present a novel machine learning approach, called SaLSa (a combination of semi-automatic labeling and long short-term memory-based classification), to classify behavioral syllables of mice exploring an open field. This approach consists of two major steps: first, after tracking multiple body parts, spatial and temporal features of their egocentric coordinates are extracted. A fully automated unsupervised process identifies candidates for behavioral syllables, followed by manual labeling of behavioral syllables using a graphical user interface. Second, a long short-term memory (LSTM) classifier is trained with the labeled data. We found that the classification performance was marked over 96%. We applied this approach to examine how hyperactivity in a mouse model of Alzheimer's disease (AD) develops with age. When the proportion of each behavioral syllable was compared between genotypes and sexes, we found that the characteristic hyper-locomotion of female AD mice emerges between 4 and 8 months. In contrast, age-related reduction in rearing is common regardless of genotype and sex. Overall, SaLSa enables detailed characterization of mouse behavior.

show abstract

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Cited by 48 publications

References 43 publications

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

Reproducible and fully automated testing of nocifensive behavior in mice

SaLSa: a combinatory approach of semi-automatic labeling and long short-term memory to classify behavioral syllables

Contact Info

Product

Resources

About