4DHands: Reconstructing Interactive Hands in 4D with Transformers
CoRR(2024)
Abstract
In this paper, we introduce 4DHands, a robust approach to recovering
interactive hand meshes and their relative movement from monocular inputs. Our
approach addresses two major limitations of previous methods: lacking a unified
solution for handling various hand image inputs and neglecting the positional
relationship of two hands within images. To overcome these challenges, we
develop a transformer-based architecture with novel tokenization and feature
fusion strategies. Specifically, we propose a Relation-aware Two-Hand
Tokenization (RAT) method to embed positional relation information into the
hand tokens. In this way, our network can handle both single-hand and two-hand
inputs and explicitly leverage relative hand positions, facilitating the
reconstruction of intricate hand interactions in real-world scenarios. As such
tokenization indicates the relative relationship of two hands, it also supports
more effective feature fusion. To this end, we further develop a
Spatio-temporal Interaction Reasoning (SIR) module to fuse hand tokens in 4D
with attention and decode them into 3D hand meshes and relative temporal
movements. The efficacy of our approach is validated on several benchmark
datasets. The results on in-the-wild videos and real-world scenarios
demonstrate the superior performances of our approach for interactive hand
reconstruction. More video results can be found on the project page:
https://4dhands.github.io.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined