Inter-personal social conversation in multimodal human-virtual human interaction

The goal of my work is in understanding the impact of social conversational functions in human-virtual human interaction. The work investigates the effect of social conversation in virtual human interface agents, and its impact on social factors such as engagement, satisfaction, appreciation, acceptance of the interface agent's role, and the success of task performance in an interactive social or public setting. In order to study this, a virtual human interface agent framework (VHIF) was created. VHIF is an extensible and modular framework that combines speech, graphics, voice recognition, cognitive and user modeling capability, and a real-time planner. Marve, a virtual receptionist, was built as a proof-of-concept application using VHIF. My contributions to the research on virtual humans are as follows: (1) Created a comprehensive, extensible, and modular Virtual Human Interface Framework (VHIF), as a platform for studying Human-Virtual Human interaction. VHIF provides an extensible and detailed infrastructure, including a rich task based discourse model that enables researchers to build and deploy virtual human interface agents and study interactions between the virtual human and his human interlocutors. (2) Implemented a proof-of-concept application called Marve, a virtual receptionist, using VHIF. Marve's role specific tasks were to greet everyone who passed by Marve's interaction area, and to take and deliver messages. Marve's perceptual capabilities include vision and speech recognition. (3) Performed an empirical evaluation of functional/operational characteristics of Marve's framework such as the interface agent's recognition accuracy of users, response times, and the ability to adaptively learn to recognize new users. We found that Marve's response times for interacting with someone who walks up to or away from him are on the order of less than one tenth of a second for greeting someone or saying goodbye. Marve recognizes his friends consistently (97% recognition rate) and calls them by name within three seconds. Marve is not as good at recognizing that he does not know someone (72.4% recognition rate). We also found that the number of visits necessary before Marve recognized a user as a new friend was eight, although this varied per user. (4) Marve's conversation capabilities were extended to include context independent social conversations such as an entertaining task (telling knock-knock jokes), small talk (talking about current movies), and a social informative task (discussions on the current weather) apart from his context dependent capabilities of message taking and delivery. From a study conducted to evaluate the relative popularity of social conversational dialogue in human-virtual human interaction in a deployed public setting, we found that a significant majority of users enjoyed social conversations with the interface agent and perceived as well as treated the virtual human interface agent as a human-like social entity. (5) Using VHIF, an Immersive Virtual Human Social Conversational Protocol Training system was built. Users can interact with life size virtual south Indians using speech and gesture to learn conversational verbal and non-verbal protocols for conversation initiation and disengagement in south Indian culture. (6) Conducted a controlled study that showed that immersive virtual humans can be used to teach social conversational protocols.**This dissertation is a compound document (contains both a paper copy and a CD as part of the dissertation). The CD requires the following system requirements: QuickTime; Windows MediaPlayer or RealPlayer.