HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models
CVPR 2024(2024)
Abstract
Recent years have witnessed a trend of the deep integration of the generation
and reconstruction paradigms. In this paper, we extend the ability of
controllable generative models for a more comprehensive hand mesh recovery
task: direct hand mesh generation, inpainting, reconstruction, and fitting in a
single framework, which we name as Holistic Hand Mesh Recovery (HHMR). Our key
observation is that different kinds of hand mesh recovery tasks can be achieved
by a single generative model with strong multimodal controllability, and in
such a framework, realizing different tasks only requires giving different
signals as conditions. To achieve this goal, we propose an all-in-one diffusion
framework based on graph convolution and attention mechanisms for holistic hand
mesh recovery. In order to achieve strong control generation capability while
ensuring the decoupling of multimodal control signals, we map different
modalities to a shared feature space and apply cross-scale random masking in
both modality and feature levels. In this way, the correlation between
different modalities can be fully exploited during the learning of hand priors.
Furthermore, we propose Condition-aligned Gradient Guidance to enhance the
alignment of the generated model with the control signals, which significantly
improves the accuracy of the hand mesh reconstruction and fitting. Experiments
show that our novel framework can realize multiple hand mesh recovery tasks
simultaneously and outperform the existing methods in different tasks, which
provides more possibilities for subsequent downstream applications including
gesture recognition, pose generation, mesh editing, and so on.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined