TwohandsMusic: Multitask Learning-Based Egocentric Piano-Playing Gesture Recognition System for Two Hands

We present TwohandsMusic, a new real-time system for recognizing egocentric piano-playing gestures on planar objects by using a depth camera. Existing methods have usually recognized single tap gestures of one hand using a sensor installed in front of or under the user's hand. In contrast, we consider recognizing multi-tap gestures of both hands using a depth camera installed near the user's head. Our approach consists of two steps: hand detection and gesture recognition. At the hand detection step, we detect both hands using a 2DCNN (Convolutional Neural Network), called SegNet, and generate cropped hand images, which is to be used in the next step. In the gesture recognition step, we estimate 3D hand poses and classify multi-tap gestures simultaneously using a 3DCNN with multitask learning, called MusicNet. For training and validating of our system, we collect 85K dataset including tapping chords and show improved results over state-of-the-art methods.