Multi-spectrum Fusion Towards 3D Human Pose Estimation Using mmWave Radar

Range-Doppler Map (RDM) and Range-Angle Map (RAM) are two kinds of representation forms of spectrum data commonly used in human pose estimation with mmWave radars. It is often that using them separately will fail to work well when the reflection signals received are insufficient for reconstructing a non-destructive human pose. To remediate the destructiveness of human pose estimations with only RDM or RAM data, this paper will explore a novel paradigm of data fusion to take full use of RDM and RAM data in human pose estimation. In doing this, two issues arise: one is how to associate the features of the two spectrums and the other is how to remove redundant features. To address these issues, inspired by the structure of Transformer encoder, a novel multi-spectrum fusion method is presented, in which a cross-attention fusion mechanism is used to facilitate the corresponding velocity to angle at the same distance region but without introducing redundant distance features. Experiment demonstrated the effectiveness of our method.