A Case Study Of Prototyping A Multimodal User Interface For A Media Annotation Tool

Media annotation is the process of adding annotations to media, like audio or video data. Annotations are, e.g., emotion descriptions of human emotions. The manual creation of annotations typically requires to repeat small tasks many times. Manual annotation is time-consuming and erroneous because user interfaces (UI) for such annotation tools often lack the possibility of multimodal interaction. In this work, we present a case study were we prototyped multimodal UIs for media annotation. First, we identified time-consuming tasks in the process of media annotation. Then we studied the human-computer interaction, to find out which modality combinations fit well for these tasks. This led us to suitable variants for modality combinations, like speech input, mouse gestures, earcons and an adapted GUI. We used the OpenInterface platform to implement prototypes of these multimodal UI variants for an existing GUI-based media annotation tool. Our prototyping approach allows easy change and adaptation of the multimodal UI. This supports the designer during the multimodal UI development and leads to UIs for media annotation tools that have a well-balanced set of modalities for interaction purposes.