AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment
arxiv(2024)
摘要
Recent video editing advancements rely on accurate pose sequences to animate
subjects. However, these efforts are not suitable for cross-species animation
due to pose misalignment between species (for example, the poses of a cat
differs greatly from that of a pig due to differences in body structure). In
this paper, we present AnimateZoo, a zero-shot diffusion-based video generator
to address this challenging cross-species animation issue, aiming to accurately
produce animal animations while preserving the background. The key technique
used in our AnimateZoo is subject alignment, which includes two steps. First,
we improve appearance feature extraction by integrating a Laplacian detail
booster and a prompt-tuning identity extractor. These components are
specifically designed to capture essential appearance information, including
identity and fine details. Second, we align shape features and address
conflicts from differing subjects by introducing a scale-information remover.
This ensures accurate cross-species animation. Moreover, we introduce two
high-quality animal video datasets featuring a wide variety of species. Trained
on these extensive datasets, our model is capable of generating videos
characterized by accurate movements, consistent appearance, and high-fidelity
frames, without the need for the pre-inference fine-tuning that prior arts
required. Extensive experiments showcase the outstanding performance of our
method in cross-species action following tasks, demonstrating exceptional shape
adaptation capability. The project page is available at
https://justinxu0.github.io/AnimateZoo/.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要