Data and model hybrid-driven virtual reality robot operating system

To realize efficient remote human-computer interaction of robots, a robot remote operating system based on virtual reality and digital twin is proposed. The system builds a digital twin model based on the Unity 3D engine to establish a connection with the robot entity, assisting the online remote programming and real-time manipulation of the robot unit. The system uses HTC VIVE to build a virtual reality framework. To actualize the mutual drive between the real space and the virtual space, a mathematical model of the robot is constructed through the forward and inverse kinematics of the robot. Through the combination of eye-tracking-based eye movement interaction and the unique controller interaction of virtual reality system, a multi-sensory multi-input collaborative interaction method is accomplished. The method realizes the robot joints driving of users using multiple interaction methods simultaneously, simplifies the robot programming and control procedure, and optimizes the operation experience. Tests demonstrate that the system is capable of effectively providing monitoring, teleoperation and programming services for remote interaction of robots.