An Interactive Annotation Tool for Perceptual Video Compression

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach for collecting finer-grained user feedback through an interactive tool that allows direct optimization of perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study (n = 26) - and are 1.9 times more likely to be preferred by viewers. We plan on collecting a large-scale dataset using the tool for automated perceptual compression in the future. The code for the tool and dataset can be found at https://github.com/jenyap/video-annotation-tool.git.