Augmented reality-based telepresence in a robotic manipulation task: An experimental evaluation

A spectrum of control methods in human-robot interaction was investigated, ranging from direct control to telepresence with a virtual representation of the robot arm. A total of 24 participants used a setup that included a Franka Emika Panda robot arm, Varjo XR-3 head-mounted display, and Leap Motion Controller. Participants performed a box-and-block task using the bare hand (A), and under five gesture-controlled robotic operation methods: direct sight (B), sight via video-feedthrough (C), in a 3D telepresence environment with (D) and without (E) virtual representation of the robot arm, and using a 2D video feed (F). The number of grabbing attempts did not differ significantly between conditions, but local operation (B & C) yielded more transferred blocks than teleoperation (D-F). Teleoperation using a 3D presentation was advantageous compared to teleoperation using a 2D video feed, as demonstrated by lower peak forces and smaller range in gripper heights in conditions D and E compared to condition F, a finding supported by analyses of the head movement activity. Finally, the bare hand yielded the best performance and subjective ratings. In summary, teleoperation using a 3D presentation provided a smoother interaction than teleoperation with a 2D video feed. However, direct human interaction remains a benchmark yet to surpass. The study examined different control methods in human-robot interaction, including augmented reality and digital twins. Direct control led to more transferred blocks than teleoperation, with teleoperation using a 3D representation proving more efficient than teleoperation with a 2D video feed. However, direct human interaction produced the best results and user ratings.image