Low Rate Compression of Video with Dynamic Backgrounds

We present a technique for very low rate encoding of videos with semantically salient foregrounds and dynamic background motion, such as an outdoor sports game. Our technique uses a MaskRCNN semantic segmentation neural network to separate video into foreground and background. Foreground is stored using a traditional video codec, along with a small semantic mask video which is upsampled during decoding. Missing regions of the background are filled using nearby frames, and the resulting video is stored using dynamic video motion sprites, a technique for good visual quality compression of dynamic backgrounds using static sprites, static vibration modes, and a small number of per-frame global motion parameters. By representing the backgrounds using static descriptors, our technique achieves higher visual background quality while preserving fidelity of important foreground regions. Our technique can target lower bitrates than may be appropriate for traditional codecs such as AVC and HEVC.