RTCSR: Zero-latency Aware Super-resolution for WebRTC Mobile Video Streaming

The real-time video streaming market is currently undergoing rapid development. Various existing methods aim to optimize network bandwidth utilization by adjusting the sending rate and implementing forward error correction techniques. Meanwhile, other methods enhance the quality of video streams using frame-level super-resolution (SR), leveraging the computational power on the receiving end, which often consumes a lot of time and increases the end-to-end delay of a real-time video stream. To this end, we proposed a real-time video stream enhancement scheme based on patch-level super-resolution. The approach adaptively selects the locations of super-resolution patches based on the time interval between video frame receiving and rendering. Additionally, the quality benefit of SR patches continuously would be transferred to subsequent video frames during the decoding and upscaling process. We evaluate the performance of our scheme on several commercial mobile devices with real network traces and construct an open-source real-time video streaming system. Overall, our scheme can improve the PSNR of real-time video streaming by 2.3dB.