Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion
arxiv(2023)
Abstract
This paper presents a self-supervised temporal video alignment framework
which is useful for several fine-grained human activity understanding
applications. In contrast with the state-of-the-art method of CASA, where
sequences of 3D skeleton coordinates are taken directly as input, our key idea
is to use sequences of 2D skeleton heatmaps as input. Unlike CASA which
performs self-attention in the temporal domain only, we feed 2D skeleton
heatmaps to a video transformer which performs self-attention both in the
spatial and temporal domains for extracting effective spatiotemporal and
contextual features. In addition, we introduce simple heatmap augmentation
techniques based on 2D skeletons for self-supervised learning. Despite the lack
of 3D information, our approach achieves not only higher accuracy but also
better robustness against missing and noisy keypoints than CASA. Furthermore,
extensive evaluations on three public datasets, i.e., Penn Action, IKEA ASM,
and H2O, demonstrate that our approach outperforms previous methods in
different fine-grained human activity understanding tasks. Finally, fusing 2D
skeleton heatmaps with RGB videos yields the state-of-the-art on all metrics
and datasets. To our best knowledge, our work is the first to utilize 2D
skeleton heatmap inputs and the first to explore multi-modality fusion for
temporal video alignment.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined