PortraitDAE: Line-Drawing Portraits Style Transfer from Photos via Diffusion Autoencoder with Meaningful Encoded Noise

The line-drawing portrait is a kind of highly abstract art that contains a sparse set of continuous graphical elements such as lines to capture a person's facial features. Due to their abstract artistic form, common style transfer methods fail to synthesize high-quality line-drawing portraits from photos. Previous works mostly concentrate on GANs, often requiring pre-calculated landmarks acquired by other models and using extra classifiers with complicated structures to capture local facial features. We propose a novel idea without these extra operations based on diffusion models, which is more flexible and stable than GAN-based methods. We utilize the diffusion-based decoder in the Diffusion Autoencoder to encode the input image to an encoded noise that contains much meaningful stochastic information by running the deterministic generative process backward. By fully utilizing the encoded noise, our method can effectively preserve the identity information and better capture facial details. We also improve the loss function to alleviate the interference of the background color. Several experiments show that our method can produce better samples with smoother lines that look more like the corresponding person, outperforming state-of-the-art methods both qualitatively and quantitatively. Our method can also be generalized to other styles such as sketch.