TwohandsMusic: Multitask Learning-Based Egocentric Piano-Playing Gesture Recognition System for Two Hands
2019 IEEE International Conference on Image Processing (ICIP)(2019)
Abstract
We present TwohandsMusic, a new real-time system for recognizing egocentric piano-playing gestures on planar objects by using a depth camera. Existing methods have usually recognized single tap gestures of one hand using a sensor installed in front of or under the user's hand. In contrast, we consider recognizing multi-tap gestures of both hands using a depth camera installed near the user's head. Our approach consists of two steps: hand detection and gesture recognition. At the hand detection step, we detect both hands using a 2DCNN (Convolutional Neural Network), called SegNet, and generate cropped hand images, which is to be used in the next step. In the gesture recognition step, we estimate 3D hand poses and classify multi-tap gestures simultaneously using a 3DCNN with multitask learning, called MusicNet. For training and validating of our system, we collect 85K dataset including tapping chords and show improved results over state-of-the-art methods.
MoreTranslated text
Key words
piano-playing gesture,hand pose estimation,gesture classification,multitask learning
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined