Video-Based Vietnamese Sign Language Recognition Using Local Descriptors

Sign Language is one of the method for non-verbal communication. It is most commonly used by deaf or dumb people who have hearing or speech problems to communicate among themselves or with normal people. Vietnamese Sign Language (VSL) is a sign language system used in the community of Vietnamese hearing impaired individuals. VSL recognition aims to develop algorithms and methods to correctly identify a sequence of produced signs and to understand their meaning in Vietnamese. However, automatic VSL recognition in video has many challenges due to the orientation of camera, hand position and movement, inter hand relation, etc. In this paper, we present some feature extraction approaches for VSL recognition includes spatial feature, scene-based feature, and especially motion-based feature. Instead of relying on a static image, we specifically capture motion information between frames in a video sequence. We evaluated the proposed framework on our acquired VSL dataset including 23 alphabets, 3 diacritic marks and 5 tones in Vietnamese language with 2D camera. Additionally, in order to gain more information of hand movement and hand position, we also used the data augmentation technique. All these helpful information would contribute to an effective VSL recognition system. The experiments achieved the satisfactory results with 86.61%. It indicates that data augmentation technique provides more information about the orientation of hand. Moreover, the combination of spatial, scene and especially motion information could help the system to be able to capture information from both single frame and from multiple frames, and thus the performance of VSL recognition system could be improved.