Physical Property Understanding from Language-Embedded Feature Fields
CVPR 2024(2024)
Abstract
Can computers perceive the physical properties of objects solely through
vision? Research in cognitive science and vision science has shown that humans
excel at identifying materials and estimating their physical properties based
purely on visual appearance. In this paper, we present a novel approach for
dense prediction of the physical properties of objects using a collection of
images. Inspired by how humans reason about physics through vision, we leverage
large language models to propose candidate materials for each object. We then
construct a language-embedded point cloud and estimate the physical properties
of each 3D point using a zero-shot kernel regression approach. Our method is
accurate, annotation-free, and applicable to any object in the open world.
Experiments demonstrate the effectiveness of the proposed approach in various
physical property reasoning tasks, such as estimating the mass of common
objects, as well as other properties like friction and hardness.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined