Multiple Human Tracking Using Deep Learning with Shadow Clues

Occlusion is an inevitable problem in virtual reality systems where a single camera is used to track multiple players during the interaction. In this work, we observe that a user’s shadow is an important hint for tracking when the user is occluded. Therefore, we propose a novel tracking approach that effectively leverages the shadow information of target users, which leads to more robust tracking in complex environments. Our key idea is to train a shadow detection model based on Mask R-CNN to extract shadows from image frames. To handle different levels of occlusion, we propose to define a series of tracking statuses for occlusion representation. To better evaluate the proposed method, we also contribute a dataset that contains numerous image frames with various forms of human shadows. Experiments demonstrate that our method is not only effective for handling user tracking even in full and long-term occlusions, but also exhibits superior real-time efficiency.