Capturing and Predicting User Frustration to Support a Smart Operating System

This paper presents an IRB-approved human study to capture data to build models for human frustration prediction of computer users. First, an application was developed that ran in the user’s computer/laptop/VM with Linux 20.04. Then, the application collected a variety of data from their computers, including: mouse clicks, movements and scrolls; the pattern of keyboard keys clicks; user audio features; and head movements through the user video; System-wide information such as computation, memory usage, network bandwidth, and input/output bandwidth of the running applications in the computer and user frustrations. Finally, the application sent the data to the cloud. After two weeks of data collection, supervised and semi-supervised models were developed offline to predict user frustration with the computer using the collected data. A semi-supervised model using a generative adversarial network (GAN) resulted in the highest accuracy of 90%.