STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting
arxiv(2024)
摘要
The creation of 4D avatars (i.e., animated 3D avatars) from text description
typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars in
the canonical space and subsequently applies animation with target motions.
However, such an optimization-by-animation paradigm has several drawbacks. (1)
For pose-agnostic optimization, the rendered images in canonical pose for naive
Score Distillation Sampling (SDS) exhibit domain gap and cannot preserve
view-consistency using only T2I priors, and (2) For post hoc animation, simply
applying the source motions to target 3D avatars yields translation artifacts
and misalignment. To address these issues, we propose Skeleton-aware Text-based
4D Avatar generation with in-network motion Retargeting (STAR). STAR considers
the geometry and skeleton differences between the template mesh and target
avatar, and corrects the mismatched source motion by resorting to the
pretrained motion retargeting techniques. With the informatively retargeted and
occlusion-aware skeleton, we embrace the skeleton-conditioned T2I and
text-to-video (T2V) priors, and propose a hybrid SDS module to coherently
provide multi-view and frame-consistent supervision signals. Hence, STAR can
progressively optimize the geometry, texture, and motion in an end-to-end
manner. The quantitative and qualitative experiments demonstrate our proposed
STAR can synthesize high-quality 4D avatars with vivid animations that align
well with the text description. Additional ablation studies shows the
contributions of each component in STAR. The source code and demos are
available at:
\href{https://star-avatar.github.io}{https://star-avatar.github.io}.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要