Noise Search Method of Controlling Image Transformation in Diffusion Model

The diffusion model possesses the potential to enhance individuals’ expressive capabilities. One of its functions involves image transformation, wherein an image can be converted to match specific conditions by conditioning it with text. However, it is common for other parts of the image to change alongside the text content. This cause underlying this behavior is that the diffusion model operates by reconstructing lost information through the introduction and subsequent removal of noise. In other words, the noise added during transformation is determined probabilistically, thereby rendering the transformation result noise-dependent. Our objective is to accomplish an image transformation that exclusively alters the text content while leaving other aspects unchanged. To achieve this, capturing the universal tendencies inherent in textual commands is necessary. Our approach employs a two-step process for acquiring the appropriate noise. Initially, the same image was reconstructed using a series of Gaussian noises. Subsequently, the average of these reconstructed images was computed to identify universal trends. The resultant image tends to be blurry, necessitating a gradient descent-based search for noise in order to regenerate the image. The outcomes indicate a discernible influence on the source image’s visual concepts associated with the text.