Deep-learning based identification, pose estimation and end-to-end behavior classification for interacting primates and mice in complex environments

The quantification of behaviors of interest from video data is commonly used to study the effects of pharmacological interventions, genetic alterations, and the function of the brain. Existing approaches lack the capabilities to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments from raw video frames, whilst requiring no intervention after minimal human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) to perform segmentation, identification, pose-estimation, and classification of complex behavior all automatically with state-of-the-art performance. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups. ### Competing Interest Statement The authors have declared no competing interest.