Cooperative Hierarchical Framework for Group Activity Recognition: From Group Detection to Multi-activity Recognition

Deep neural network algorithms have shown promising performance for many tasks in computer vision field. Several neural network-based methods have been proposed to recognize group activities from video sequences. However, there are still several challenges that are related to multiple groups with different activities within a scene. The strong correlation that exists among individual motion, groups and activities can be utilized to detect groups and recognize their concurrent activities. Motivated by these observations, we propose a unified deep learning framework for detecting multiple groups and recognizing their corresponding collective activity based on Long Short-Term Memory (LSTM) network. In this framework, we use a pre-trained convolutional neural network (CNN) to extract features from the frames and appearances of persons. An objective function has been proposed to learn the amount of pairwise interaction between persons. The obtained individual features are passed to a clustering algorithm to detect groups in the scene. Then, an LSTM based model is used to recognize group activities. Together with this, a scene level CNN followed by LSTM is used to extract and learn scene level feature. Finally, the activities from the group level and the scene context level are integrated to infer the collective activity. The proposed method is evaluated on the benchmark collective activity dataset and compared with several baselines. The experimental results show its competitive performance for the collective activity recognition task.