Human-Robot Full-Sentence VQA Interaction System with Highway Memory Network

Intelligent robots such as social robots and home service robots that are being developed nowadays are required to have the ability to naturally communicate with users in natural language. Beyond simple data retrieval and simple dialogue level, we propose a new Human-Robot Interaction (HRI) system that enables a robot to understand and reason the environment around a user and present information about them in natural language, whenever the user asks a question to the robot. For its intelligent HRI, based on Dynamic Memory Networks (DMN), a neural network for Visual Question Answering (VQA), we propose a new full sentence VQA network model called Full-Sentence Highway Memory Network (FSHMN). For its robot platform, a three DOF robotic head was used which has a neck with three motors and a tablet PC head. To verify the feasibility of the proposed system, an experiment is performed in which a user and a robot interact with each other in a way of question answering in a customized kitchen environment. Through the experiment, we not only demonstrated the effectiveness of applying deep learning to HRI applications in real environments but also presented a new insight into HRI.