Fine-tuning Diffusion Models for Enhancing Face Quality in Text-to-image Generation
arxiv(2024)
Abstract
Diffusion models (DMs) have achieved significant success in generating
imaginative images given textual descriptions. However, they are likely to fall
short when it comes to real-life scenarios with intricate details.The
low-quality, unrealistic human faces in text-to-image generation are one of the
most prominent issues, hindering the wide application of DMs in practice.
Targeting addressing such an issue, we first assess the face quality of
generations from popular pre-trained DMs with the aid of human annotators and
then evaluate the alignment between existing metrics such as ImageReward, Human
Preference Score, Aesthetic Score Predictor, and Face Quality Assessment, with
human judgments. Observing that existing metrics can be unsatisfactory for
quantifying face quality, we develop a novel metric named Face Score (FS) by
fine-tuning ImageReward on a dataset of (good, bad) face pairs cheaply crafted
by an inpainting pipeline of DMs. Extensive studies reveal that FS enjoys a
superior alignment with humans. On the other hand, FS opens up the door for
refining DMs for better face generation. To achieve this, we incorporate a
guidance loss on the denoising trajectories of the aforementioned face pairs
for fine-tuning pre-trained DMs such as Stable Diffusion V1.5 and Realistic
Vision V5.1. Intuitively, such a loss pushes the trajectory of bad faces toward
that of good ones. Comprehensive experiments verify the efficacy of our
approach for improving face quality while preserving general capability.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined