Real-time anomaly detection on surveillance video with two-stream spatio-temporal generative model

Abnormal detection of surveillance video is of great significance to social security and the protection of specific scenes. However, the existing methods fail to achieve a balance between accuracy and real-time performance. In this paper, we propose a two-stream spatio-temporal generative model (TSSTGM) for surveillance videos to detect abnormal behaviors in real-time. We construct an end-to-end video reconstruction and prediction framework based on deep learning to detect the anomalies by reconstruction error and prediction error. Specifically, we elaborately design a fully convolutional structure, enabling the model to accept input videos of any size. To ensure great performance in complex scenes, appearance, temporal and motion features are fully explored and fed into the discriminator to train the model with adversarial learning. Moreover, the input design and the calculation way of optical flow ensure the model runs in real-time. Experiments on two real-world datasets show that, when satisfying the real-time requirement, TSSTGM is still competitive compared with no matter real-time or non-real-time existing methods in AUC and EER metrics. Our model has been deployed in several campus security surveillance systems to detect dangerous behaviors, ensuring the personal safety of students.