Integrating Sensor Fusion with Pose Estimation for Simulating Human Interactions in Virtual Reality.

Virtual Reality (VR) enables users to interact with a simulated environment through a head-mounted device (HMD) and two hand controllers. These help in tracking the user's head and hand locations and movements but have no references to the rest of the body. Though these movements can be tracked using additional equipment like trackers and bodysuits, being tightly coupled with hardware would make this an inconvenient as well as an expensive option. Also, other existing computer vision solutions require calibration and expensive depth-based cameras. In this paper, we present a novel approach of using sensor fusion techniques integrated with computer vision algorithms as an alternate solution to position human joints in the virtual environment and simulate their movements. The human landmark identification and pose estimation are achieved using ML algorithms for computer vision. These landmarks are mapped to an avatar in virtual space using geometric transformations and inverse kinematics. The use of sensor fusion ensures the correctness of scale and transform operations. The HMD position helps pivot the estimated pose and set the view for the user. The hand controller feed is used to control and verify the position of the hand with respect to the upper body. Through sensor fusion, we can bypass the complicated setup required by other Computer Vision (CV) approaches. The solution is built using open source tools and frameworks and the rest of the paper discussed the related work, design overview, and results in detail.