SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models
CoRR(2024)
Abstract
We are witnessing a revolution in conditional image synthesis with the recent
success of large scale text-to-image generation methods. This success also
opens up new opportunities in controlling the generation and editing process
using multi-modal input. While spatial control using cues such as depth,
sketch, and other images has attracted a lot of research, we argue that another
equally effective modality is audio since sound and sight are two main
components of human perception. Hence, we propose a method to enable
audio-conditioning in large scale image diffusion models. Our method first maps
features obtained from audio clips to tokens that can be injected into the
diffusion model in a fashion similar to text tokens. We introduce additional
audio-image cross attention layers which we finetune while freezing the weights
of the original layers of the diffusion model. In addition to audio conditioned
image generation, our method can also be utilized in conjuction with diffusion
based editing methods to enable audio conditioned image editing. We demonstrate
our method on a wide range of audio and image datasets. We perform extensive
comparisons with recent methods and show favorable performance.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined