RGBZ Image Restoration by Patch Clone.

The RGBZ sensor is a novel imaging sensor that captures both color and depth images simultaneously in a single chip, with a specially designed color-filter-array (CFA), in which some of the RGB color pixels are replaced by “Z” pixels that capture depth information but no color information. As a result, RGB color images produced by this pixel array appear degraded, with missing RGB values or “holes” at locations occupied by the Z pixels. To fill in these “holes”, and thus restore resolution and appearance of color images, we propose a Patch-Clone method that exploits redundant texture information in the scene. Derived from the non-local approaches, our method consists of two steps: 1) a matching step to identify the candidate patch that contains the most useful information to reconstruct the color pixels missing at a particular hole; 2) a cloning step to copy the content from the candidate to fill in the hole. When higher order pixel content is copied, pixel continuity between the restored and original pixels can be enforced. The result of the proposed method is full resolution Bayer images, to which existing common demosaic algorithms can be applied. Tests show that the proposed method provides better reconstruction result in term of distortion error as well as visual appearance.