DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors
CoRR(2024)
Abstract
Dynamic 3D interaction has witnessed great interest in recent works, while
creating such 4D content remains challenging. One solution is to animate 3D
scenes with physics-based simulation, and the other is to learn the deformation
of static 3D objects with the distillation of video generative models. The
former one requires assigning precise physical properties to the target object,
otherwise the simulated results would become unnatural. The latter tends to
formulate the video with minor motions and discontinuous frames, due to the
absence of physical constraints in deformation learning. We think that video
generative models are trained with real-world captured data, capable of judging
physical phenomenon in simulation environments. To this end, we propose
DreamPhysics in this work, which estimates physical properties of 3D Gaussian
Splatting with video diffusion priors. DreamPhysics supports both image- and
text-conditioned guidance, optimizing physical parameters via score
distillation sampling with frame interpolation and log gradient. Based on a
material point method simulator with proper physical parameters, our method can
generate 4D content with realistic motions. Experimental results demonstrate
that, by distilling the prior knowledge of video diffusion models, inaccurate
physical properties can be gradually refined for high-quality simulation. Codes
are released at: https://github.com/tyhuang0428/DreamPhysics.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined