Augmented Telepresence: Enhancing Robot Arm Control with Mixed Reality for Dexterous Manipulation

Robotized agents are becoming increasingly important in the construction industry as an extension of human capabilities. In many applications, the complexity of a teleoperation task could only be completed alongside the same physical environment to ensure synchronized situational awareness. With the recent advances in mixed reality (MR) and reality capture technologies, a new interface for teleoperating robots with a physical world reference is made possible. This paper introduces an intuitive MR control interface that could mitigate the gap between humans and robots by using a HoloLens2 optical see-through head mounted display (HMD), a Franka Panda 7-DOF robot arm, ranging sensors, and Unity game engine. With real-time acquired spatial information, a high-fidelity digital twin of the physical workspace is teleported to the user in mixed reality. Simultaneously, a virtual space around the user is also defined by the HMD with their hand gesture processed as input commands. The two virtualized spaces are then registered together and merged to be organized by the Unity game engine. As a result, the user takes control of the robot's arm through direct manipulation of the virtual/real version of their hand. The system is tested with intricate pipe fitting tasks to validate its performance.