Deep Imputation for Skeleton Data (DISK) for Behavioral Science

Abstract

Pose estimation methods and motion capture systems have opened doors to quantitative measurements of animal kinematics. However, these methods are not perfect and contain missing data. Our method, Deep Imputation for Skeleton data (DISK), leverages deep learning algorithms to learn dependencies between keypoints and their dynamics to impute missing tracking data. We developed an unsupervised training scheme, which does not rely on manual annotations, and tested several neural network architectures for the imputation task. We found that transformer outperforms other architectures including graph convolutional networks that were developed specifically for skeleton-based action recognition. We demonstrate the usability and performance of our imputation method on seven different animal skeletons including two multi-animal set-ups. With an optional estimated imputation error, DISK enables behavior scientists to assess the reliability of the imputed data. The imputed recordings allow to detect more episodes of motion, such as steps, and to obtain more statistically robust results when comparing these episodes between experimental conditions. While animal behavior experiments are expensive and complex, tracking errors make sometimes large portions of the experimental data unusable. DISK allows for filling in the missing information and for taking full advantage of the rich behavioral data. This stand-alone imputation package, freely available at https://github.com/bozeklab/DISK.git, is applicable to results of any tracking method (marker-based or markerless) and allows for any type of downstream analysis.