Integrated Registration and Occlusion Handling Based on Deep Learning for Augmented-Reality-Assisted Assembly Instruction

Augmented reality (AR) can convert complex work instructions into virtual–reality fusion contents for assembly guidance. In the past, AR registration and occlusion were usually implemented separately, with low robustness and poor timeliness. This article proposes a novel deep learning scheme, named AR-CenterNet, to integrate AR registration and occlusion handling. The proposed method mainly includes two stages, i.e., the neural network prediction stage and the AR processing stage. In the first stage, AR-CenterNet is designed for keypoint detection and depth map prediction. In the second stage, the pose matrix of the physical camera is solved with the predicted keypoints and the depth map of the virtual scene is compared with the predicted depth map for occlusion handling. The experiments demonstrate that our method is robust against different conditions for assisted assembly. This article can provide a new solution method for AR virtual–reality fusion based on monocular images.