Experience Surveillance Suite for Unity 3D

Monitoring the emotional state of players in games can get quite complex, taking into consideration that the game context affects the player and that a game may contain various emotional features. Furthermore, since the experience of playing a game occurs unconsciously, methods such as think aloud may interrupt the playing experience. Other methods include fitting cables and electrodes to the player in order to monitor measurements such as heart rate. Although such devices can offer significant results, they are not commonly found and may cause discomfort. In this project we propose a webcam-based heart rate monitoring method that can be used to predict the player's emotional state. The first objective was to analyze the heart rate changes with respect to the players' emotional state. The evaluation resulted in positive results, where the heart rate showed correlation with the following emotional states; frustration, fun, challenge and boredom. The second objective was to create a webcam-based method to monitor the heart rate. This was performed by extracting the RGB channels from the face region and then retrieving the underlying components using a dimensionality reduction method. Although the results obtained from the webcam-based method were not ideal, this was expected taking into consideration that the method was tested under realistic scenarios. The last objective was to predict the player's emotional state using the heart rate obtained from the webcam-based method. The accuracy of the prediction was up to 76%, which exceeds the aim of the project. Finally, by using the evaluation results it was possible to define a set of approaches on how this project can be extended by future researchers.