Classification of Functional and Nonfunctional Hand Movement Using Deep Learning and First-Person View Video

Upper extremity (UE) rehabilitation of motor function is needed post-stroke. Although UE rehabilitation is widely used in practice, it is difficult to evaluate the effectiveness of UE treatments. There are various approaches to evaluate UE motor function such as performing tasks in the clinician's office, patient's self-report, or wearable devices equipped with accelerometer/gyro sensors to analyze the data captured from stroke survivors. Our previous study reported that using a Random Forest machine learning algorithm with accelerometry sensor data obtained by single wrist-worn sensor, it is possible to differentiate between functional and non-functional UE movements in stroke survivors with good accuracy. However, to build the model, a third-person view video was captured and manually annotated by experts to provide ground truth labels for training. This task is very time-consuming and subjective. In this study, we propose a framework that uses deep learning (DL) to classify functional and non-functional hand movement in videos captured from a wearable camera (i.e., first-person view videos). The proposed system is fully automated and consists of two DL networks. The first DL network performs hand pose estimation from a 2-second sequence of frames which extract pre-defined key points on the hands. The second DL network then takes these key points as input and classifies the sequence of frames to either functional or non-functional movement. The proposed system has two major impacts on rehabilitation. First, it can provide initial annotations on the frames of videos that can significantly reduce the time required for human annotations. Second, analyzing the effectiveness of UE rehabilitation using the first-person view videos enables us to measure the outcomes of UE treatments in stroke survivors' homes. In addition, the system can deliver timely feedback to stroke survivors that has the potential to increase the use of the affected UE in the community.