The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Analysing the behavior of individuals or groups of animals in complex environments is an important, yet difficult computer vision task. Here we present a novel deep learning architecture for classifying animal behavior and demonstrate how this end-to-end approach can significantly outperform pose estimation-based approaches, whilst requiring no intervention after minimal training. Our behavioral classifier is embedded in a first-of-its-kind pipeline (SIPEC) which performs segmentation, identification, pose-estimation and classification of behavior all automatically. SIPEC successfully recognizes multiple behaviors of freely moving mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.