Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal
arxiv(2022)
摘要
Most deep-learning-based continuous sign language recognition (CSLR) models
share a similar backbone consisting of a visual module, a sequential module,
and an alignment module. However, due to limited training samples, a
connectionist temporal classification loss may not train such CSLR backbones
sufficiently. In this work, we propose three auxiliary tasks to enhance the
CSLR backbones. The first task enhances the visual module, which is sensitive
to the insufficient training problem, from the perspective of consistency.
Specifically, since the information of sign languages is mainly included in
signers' facial expressions and hand movements, a keypoint-guided spatial
attention module is developed to enforce the visual module to focus on
informative regions, i.e., spatial attention consistency. Second, noticing that
both the output features of the visual and sequential modules represent the
same sentence, to better exploit the backbone's power, a sentence embedding
consistency constraint is imposed between the visual and sequential modules to
enhance the representation power of both features. We name the CSLR model
trained with the above auxiliary tasks as consistency-enhanced CSLR, which
performs well on signer-dependent datasets in which all signers appear during
both training and testing. To make it more robust for the signer-independent
setting, a signer removal module based on feature disentanglement is further
proposed to remove signer information from the backbone. Extensive ablation
studies are conducted to validate the effectiveness of these auxiliary tasks.
More remarkably, with a transformer-based backbone, our model achieves
state-of-the-art or competitive performance on five benchmarks, PHOENIX-2014,
PHOENIX-2014-T, PHOENIX-2014-SI, CSL, and CSL-Daily. Code and Models are
available at https://github.com/2000ZRL/LCSA_C2SLR_SRM.
更多查看译文
关键词
continuous sign language recognition,signer removal,consistency
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要