Teaching Text-to-Image Models to Communicate in Dialog
CoRR(2023)
Abstract
A picture is worth a thousand words, thus, it is crucial for conversational
agents to understand, perceive, and effectively respond with pictures. However,
we find that directly employing conventional image generation techniques is
inadequate for conversational agents to produce image responses effectively. In
this paper, we focus on the innovative dialog-to-image generation task, where
the model synthesizes a high-resolution image aligned with the given dialog
context as a response. To tackle this problem, we design a tailored fine-tuning
approach on the top of state-of-the-art text-to-image generation models to
fully exploit the structural and semantic features in dialog context during
image generation. Concretely, we linearize the dialog context with specific
indicators to maintain the dialog structure, and employ in-domain data to
alleviate the style mismatch between dialog-to-image and conventional image
generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that
our approach brings consistent and remarkable improvement with 3
state-of-the-art pre-trained text-to-image generation backbones.
MoreTranslated text
Key words
models,communicate,text-to-image
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined