Appearance-motion heterogeneous networks for video anomaly detection

Video anomaly detection, an important aspect of intelligent surveillance systems, is an important challenge to effectively distinguish the appearance and motion differences between normal and anomalous events. However, previous work either relies on single-scale appearance (spatial) features or motion (temporal) features or treats them indiscriminately, making the model unable to exploit information specific to both. Secondly, existing clustering methods are susceptible to high-dimensional spatial “chain” mainfold distribution, which affects the accuracy of anomaly detection. In this paper, we propose a novel unsupervised prediction network, the appearance-motion heterogeneous network (AMHN) for video anomaly detection. The AMHN consists of a spatial convolutional auto-encoder (CAE) for learning appearance normality, a temporal U-Net auto-encoder for learning motion normality, and a key-value network for alleviating the “chain” mainfold distribution of apparent-motion features in high-dimensional space. In the testing phase, normal and anomalous events are distinguished by generating a regular score for each sample. The proposed AMHN framework outperforms the state-of-the-art methods with AUCs 96.71%, 86.70% and 73.88% on UCSD Ped2, CHUK Avenue and ShanghaiTech datasets, respectively.