InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following
CoRR(2023)
Abstract
The ability to provide fine-grained control for generating and editing visual
imagery has profound implications for computer vision and its applications.
Previous works have explored extending controllability in two directions:
instruction tuning with text-based prompts and multi-modal conditioning.
However, these works make one or more unnatural assumptions on the number
and/or type of modality inputs used to express controllability. We propose
InstructAny2Pix, a flexible multi-modal instruction-following system that
enables users to edit an input image using instructions involving audio,
images, and text. InstructAny2Pix consists of three building blocks that
facilitate this capability: a multi-modal encoder that encodes different
modalities such as images and audio into a unified latent space, a diffusion
model that learns to decode representations in this latent space into images,
and a multi-modal LLM that can understand instructions involving multiple
images and audio pieces and generate a conditional embedding of the desired
output, which can be used by the diffusion decoder. Additionally, to facilitate
training efficiency and improve generation quality, we include an additional
refinement prior module that enhances the visual quality of LLM outputs. These
designs are critical to the performance of our system. We demonstrate that our
system can perform a series of novel instruction-guided editing tasks. The code
is available at https://github.com/jacklishufan/InstructAny2Pix.git
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined