What you see is what you get: measure ABR video streaming QoE via on-device screen recording

Analyzing delivered QoE for Adaptive Bitrate (ABR) streaming over cellular networks is critical for a host of entities including content providers and mobile network providers. However, existing approaches mostly rely on network traffic analysis. In addition to potential accuracy issues, they are challenged by the increasing use of end-to-end network traffic encryption. In this paper, we explore a very different approach to QoE measurement --- utilizing the screen recording capability widely available on commodity devices to record the video displayed on the mobile device screen, and analyzing the recorded video to measure the delivered QoE. We design a novel system VideoEye to conduct such screen-recording-based QoE analysis. We identify the various technical challenges involved, including distortions introduced by the screen recording process that can make such analysis difficult. We develop techniques to accurately measure video QoE from the screen recordings even in the presence of recording distortions. Our evaluations demonstrate that VideoEye accurately detects important QoE indicators including the track played at different points in time, and stall statistics. The maximal error in detected stall duration is 0.5 s. The accuracy of detecting the displayed tracks is higher than 97%.