Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture
CoRR(2024)
摘要
In this paper, we touch on the problem of markerless multi-modal human motion
capture especially for string performance capture which involves inherently
subtle hand-string contacts and intricate movements. To fulfill this goal, we
first collect a dataset, named String Performance Dataset (SPD), featuring
cello and violin performances. The dataset includes videos captured from up to
23 different views, audio signals, and detailed 3D motion annotations of the
body, hands, instrument, and bow. Moreover, to acquire the detailed motion
annotations, we propose an audio-guided multi-modal motion capture framework
that explicitly incorporates hand-string contacts detected from the audio
signals for solving detailed hand poses. This framework serves as a baseline
for string performance capture in a completely markerless manner without
imposing any external devices on performers, eliminating the potential of
introducing distortion in such delicate movements. We argue that the movements
of performers, particularly the sound-producing gestures, contain subtle
information often elusive to visual methods but can be inferred and retrieved
from audio cues. Consequently, we refine the vision-based motion capture
results through our innovative audio-guided approach, simultaneously clarifying
the contact relationship between the performer and the instrument, as deduced
from the audio. We validate the proposed framework and conduct ablation studies
to demonstrate its efficacy. Our results outperform current state-of-the-art
vision-based algorithms, underscoring the feasibility of augmenting visual
motion capture with audio modality. To the best of our knowledge, SPD is the
first dataset for musical instrument performance, covering fine-grained hand
motion details in a multi-modal, large-scale collection.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要