Video behavior recognition algorithm based on two-stream heterogeneous convolutional neural network

Abstract In video behavior recognition, making full use of the spatio-temporal information contained in video frame is the critical point to further improve the recognition accuracy. In our study, we propose a video behavior recognition algorithm based on two-stream heterogeneous network, so that we can further extract the spatio-temporal feature information in the video frame, and finally take full advantage of the spatio-temporal features in the video sequence. In view of the characteristics of RGB and optical flow images, this paper uses DenseNet121 and Inception-V4 to structure a two-stream network, to fully extract the spatio-temporal feature information in the video. In this paper, UCF101 dataset is used to evaluate the algorithm, and the recognition accuracy is 91.7%, which verified the reliability of the proposed algorithm.