Real-Time Deep Video Analytics on Mobile Devices

ABSTRACTReal-time mobile video analytics plays an increasingly important role in our daily life, such as smart driving, unmanned delivery, cashier free stores, and video surveillance. The existing video analytics runs complex deep models to detect and recognize objects in video frames. However, running deep models on mobile devices can not meet the real-time requirement. This paper develops a novel mobile video analytics system. Its unique features include (i) high accuracy, (ii) real-time, and (iii) running exclusively on a mobile device without the need of edge/cloud server or network connectivity. At its heart lies an effective technique to reliably extract motion from video frames and use the motion to speed up video analytics. Unlike the existing motion extraction, our technique is robust to background noise and changes in object sizes. Extensive evaluation results show that we can support real-time object tracking at 30 frames/second (fps) on Nvidia Jetson TX2. For single-object tracking, Sight improves the average Intersection-over-Union (IoU) by 88%, improves the mean Average Precision (mAP) by 207% and reduces the average hardware resource usage by 45% over state-of-the-art approach. For multi-object tracking, Sight improves IoU by 69%, improves mAP by 173% and reduces resource usage by around 32% over state-of-the-art approach.