Facemap: a framework for modeling neural activity based on orofacial tracking

Abstract

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracking algorithm and a deep neural network encoder for predicting neural activity. We used the Facemap keypoints as input for the deep neural network to predict the activity of ~50,000 simultaneously-recorded neurons and in visual cortex we doubled the amount of explained variance compared to previous methods. Our keypoint tracking algorithm was more accurate than existing pose estimation tools, while the inference speed was several times faster, making it a powerful tool for closed-loop behavioral experiments. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used Facemap to find that the neuronal activity clusters which were highly driven by behaviors were more spatially spread-out across cortex. We also found that the deep keypoint features inferred by the model had time-asymmetrical state dynamics that were not apparent in the raw keypoint data. In summary, Facemap provides a stepping stone towards understanding the function of the brainwide neural signals and their relation to behavior.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

“…2B,C,D). Practitioners often detect outliers using a combination of low confidence and large temporal difference loss [16, 20, 32, 42]. Here we show that the standard approach can be complemented by multi-view and Pose PCA, which capture additional unique outliers.…”

“…We define the temporal difference loss for each body part as the Euclidean distance between consecutive predictions in pixels. Similar losses have been used by several practitioners to detect outlier predictions post-hoc [16, 32], whereas our goal here, following [34], is to incorporate these penalties directly into network training to achieve more accurate network output. Figure 2B illustrates this penalty: the cartoon in the left panel indicates a jump discontinuity we would like to penalize.…”

“…Conversely, when a keypoint’s uncertainty is high, the spatiotemporal priors and other more-confident keypoints’ predictions will be used to interpolate over these uncertain observations. Unlike previous approaches [16, 20, 22, 32, 42], EKS requires no manual selection of confidence thresholds or (suboptimal) temporal linear interpolation separately for each dropped keypoint. Moreover, the EKS post-processing approach is agnostic to the type of networks used to generate the ensemble predictions.…”

The COMBO window: A chronic cranial implant for multiscale circuit interrogation in mice

“…A locomotion time series was then created by averaging the movement time series of the left and right paw, which was then smoothed with a 0.5 sec (15 frame) sliding window. Videographic recordings during optogenetic stimulation were analyzed using FaceMap 51 . The motion energy was extracted from regions-of-interest (ROIs) placed over the whisker pad of the animal and of the wheel (without any body part visible), to obtain a proxy of whisking and locomotion, respectively.…”

Feedforward and disinhibitory circuits differentially control activity of cortical somatostatin-positive interneurons during behavioral state transitions

“…Whisker pad movement was tracked using the python framework Facemap (Syeda et al, 2022) (https://github.com/MouseLand/facemap). Whisker-pad movement was measured by selecting an ROI over the whisker pad and calculating the motion energy index (MEI) across the video.…”