FSR: a feature self-regulation network for partially occluded hand pose estimation

Hand pose estimation is important for many applications, but the performance is not satisfying when the hand is interacting with objects. To alleviate the influence of unknown objects, we propose a novel network which makes full use of the multimodal information of the RGB-D images. The network can use the color features and/or the depth features selectively according to the prediction result of whether the hand is severely occluded or slightly occluded. We also use a new principal feature enhancement structure with an irrelevant feature weakening strategy to make the pose estimation more accurate. The FHAD dataset is used in the experiments for the performance evaluation. For ‘action-split’ data group and ‘subject-split’ data group, the obtained mean joint error is 10.63 mm and 10.61mm, respectively. These results are better than those of the state-of-the-art methods. For ‘object-split’ data group, the obtained mean joint error is 17.42mm, which is on par with the best results so far. The experimental results show the effectiveness of the proposed architecture.