Multimodal immersive digital twin platform for cyber-physical robot fleets in nuclear environments

The nuclear energy sector can benefit from mobile robots for remote inspection and handling, reducing human exposure to radiation. Advances in cyber-physical systems have improved robotic platforms in this sector through digital twin technology. Digital twins enhance situational awareness for robot operators, crucial for safety in the nuclear energy sector, and their value is anticipated to increase with the growing complexity of cyber-physical systems. We propose a multimodal immersive digital twin platform for cyber-physical robot fleets based on ROS-Unity3D. The system design integrates building information models, displays mission parameters, visualizes robot sensor data, and provides multi-modal user interaction through traditional and virtual reality interfaces. This enables real-time monitoring and management of fleets with heterogeneous robotic platforms. We performed a heuristic evaluation of our digital twin interface with robot operators from leading nuclear research institutions: Sellafield Ltd and Japan Atomic Energy Agency, performing a simulated robot inspection mission. The three usability themes that emerged from this evaluation and inspired our design recommendations include the flexibility of the interface, the individuality of each robot, and adaptation to expanding sensor visualization capabilities.