Social Behavior Recognition in Mouse Video Using Agent Embedding and LSTM Modelling.

Growing demands for automated analysis of animal behavior in areas such as neuroscience, psychology, genetics and pharmacology have been witnessed in recent decades. Some progresses have been made, but studies on social behavior analysis, which is more challenging, are rarely seen and almost all of them rely on hand-crafted features. Motivated by the concept of word embedding in NLP and the success of deep learning, we present a method that extracts features for both of the mouse agents involved in social behavior events and the scenario context using embedding networks, then uses an LSTM network to model the behaviors based on the agent and context embeddings. Our method is tested on a novel dataset, RatSI [8]. We find our mouse state embedding method outperforms traditional hand-crafted feature based methods.