A debanding algorithm for AV2

Banding is a visually unpleasing artifact appearing in flat areas of encoded content that no video standard has fully addressed. We propose a normative debanding filter to tackle banding artifacts and have tested it as an inloop and post-loop filter in AVM. Debanding is achieved by introducing dithering on a frame level to the luma component. The proposed filter shows CAMBI gains for content with banding while not affecting other content. Although the added dithering has a minor negative impact on some objective metrics, subjective improvements in banding-prone content are (informally) observed. On the test set, encoding time increases on average by ~0.5%, while decoding time increases by around 0.5% for in-loop and 1.5% for post-loop.