Happy or Sad, Smiling or Drawing with Multimodal Search and Visualisation of Movies Based on Emotions Along Time

Movies have always had a very important role in society, and have become a powerful vehicle for culture, education, leisure and even propaganda (Shah, 2011). One of the main reasons is their ability to awaken the emotions of their viewers, to influence our moods, attitudes and consequently our health and wellbeing, making a difference in our lives. The success of each movie depends on the emotions that are perceived and felt by the audience (Aurier & Guintcheva, 2015); and the emotional information related to this experience that viewers have is actually considered an important factor when searching or seeking a film to watch, also determining its success (Arriaga et al., 2019; Zhang et al., 2009). Of particular importance is the safe environment provided to experience roles and emotions we might not otherwise be free to experience (Uhrig, 2005), and film has gained a uniquely powerful ubiquity within human culture (Shah, 2011), supported by pervasive services and devices. In this context, the huge amount of movies or films we can access, and the important role of emotions, make more pertinent the ability to access, visualize and search movies based on their emotional impact. As a whole and along time: “As the frames move and tell a story, it is that movement which emotionally connects you” (Shah, 2011), and this is the journey, the path or emotional story, we want to capture and support. On the other hand, rich content of movies appeals to different senses, and the ubiquity in their access creates opportunities to use different devices, even in casual situations and environments, suggesting a multimodal access. Such situations may be when we want our current emotion taken into account, or want to draw an emotional path to search for in movies; possibly triggered by a music we are listening to, that moves us and reminds us of movies we like and how they made us feel; with the ability to get us into a movie in serendipitous moments. In this paper, we characterize and discuss main motivation, challenges and approaches in this scenario, then present and evaluate interactive means to visualize and search movies based on their emotional impact, dominant as a whole or along the movie, with different models and modalities. In particular through emotional highlights in words, colors, emojis and trajectories, by drawing emotional blueprints or through users’ emotional states in their facial expressions, with the ability to get us into a movie in serendipitous ways. The user evaluation conducted to assess perceived usefulness, usability and user experience in the search and interactive visualization features designed and developed to access movies based on emotions, and overall, had encouraging results, and informed our conclusions and perspectives for future work. This work was recently presented at HUCAPP 2023 before a very interested audience, receiving very good feedback and an award [1].