I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models
CoRR(2023)
摘要
Text-guided image-to-video (I2V) generation aims to generate a coherent video
that preserves the identity of the input image and semantically aligns with the
input prompt. Existing methods typically augment pretrained text-to-video (T2V)
models by either concatenating the image with noised video frames channel-wise
before being fed into the model or injecting the image embedding produced by
pretrained image encoders in cross-attention modules. However, the former
approach often necessitates altering the fundamental weights of pretrained T2V
models, thus restricting the model's compatibility within the open-source
communities and disrupting the model's prior knowledge. Meanwhile, the latter
typically fails to preserve the identity of the input image. We present
I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the
unnoised input image to subsequent noised frames through a cross-frame
attention mechanism, maintaining the identity of the input image without any
changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few
trainable parameters, significantly alleviating the training cost and also
ensures compatibility with existing community-driven personalized models and
control tools. Moreover, we propose a novel Frame Similarity Prior to balance
the motion amplitude and the stability of generated videos through two
adjustable control coefficients. Our experimental results demonstrate that
I2V-Adapter is capable of producing high-quality videos. This performance,
coupled with its agility and adaptability, represents a substantial advancement
in the field of I2V, particularly for personalized and controllable
applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要