Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Abstract

Pose estimation algorithms are shedding new light on animal behavior and intelligence. Most existing models are only trained with labeled frames (supervised learning). Although effective in many cases, the fully supervised approach requires extensive image labeling, struggles to generalize to new videos, and produces noisy outputs that hinder downstream analyses. We address each of these limitations with a semi-supervised approach that leverages the spatiotemporal statistics of unlabeled videos in two different ways. First, we introduce unsupervised training objectives that penalize the network whenever its predictions violate smoothness of physical motion, multiple-view geometry, or depart from a low-dimensional subspace of plausible body configurations. Second, we design a new network architecture that predicts pose for a given frame using temporal context from surrounding unlabeled frames. These context frames help resolve brief occlusions or ambiguities between nearby and similar-looking body parts. The resulting pose estimation networks achieve better performance with fewer labels, generalize better to unseen videos, and provide smoother and more reliable pose trajectories for downstream analysis; for example, these improved pose trajectories exhibit stronger correlations with neural activity. We also propose a Bayesian post-processing approach based on deep ensembling and Kalman smoothing that further improves tracking accuracy and robustness. We release a deep learning package that adheres to industry best practices, supporting easy model development and accelerated training and prediction. Our package is accompanied by a cloud application that allows users to annotate data, train networks, and predict new videos at scale, directly from the browser.

Exploration biases forelimb reaching strategies

“…This was not due to a lack of task engagement as animals readily performed movements with the joystick ( Figures 4E and S4A ). We investigated how mice manipulated the joystick after the lesion using markerless pose estimation from videos tracking the joystick base and the wrist 71 ( Figures S4B and S4C ; Video S2 ). As mice moved the joystick, their wrist to joystick distance did not change significantly post-stroke ( Figures S4D and S4E ), but it became more variable ( Figures S4D and S4F ).…”

“…Videos of animals before and after the cortex stroke lesion were used to train a pose estimation model (lightning pose 71 ) to track 2 key points: the base of the joystick, and the wrist of the mouse’s right hand. The joystick key point was chosen at the bottom left corner of the joystick spacer which had good contrast for high-fidelity tracking.…”

A 3D whole-face movement analysis system to uncover underlying physiology in mice

“…Recognizing the unique and underexplored potential to use whole-face dynamics as a noninvasive readout of moment-to-moment changes of body and brain states in mice, we crafted Cheese3D as a specialized high-resolution tool to study mouse facial movements, compared to and built upon emerging animal behavioral tracking methods aimed to generalize across body parts and species [19][20][21][30][31][32][33][34][35]. Moreover, in contrast to existing methods that focus on static facial images, motion of a subset of facial features, or aggregates of orofacial behavior optimized to predict cortical neural activities [3,6], Cheese3D is specifically designed to capture and represent whole-face movement while maintaining spatial and physical interpretability.…”

Visual perceptive deep learning for smartphone video-based tremor analysis: VIPER-Tremor

“…This is ascribed to the large out-of-domain variance introduced in clinical settings, such as dressings, obscuration or clutter 27,34,56 . To gauge this effect's relevance in the context of tremor, we additionally developed a tremor-speci c residual convolutional neural network using DeepLabCut 43 : DLC-RCNN 33,57 . Brie y, the RCNN was trained with >120,000 frames of clinical video material.…”