VCOACH: A Virtual Coaching System Based on Visual Streaming

A virtual coaching system is key for monitoring the performance during exercise regimes and rendering timely feedback to avoid potential physical harm. Ideally, the core technologies behind such a system imply exercise recognition/assessment for performance evaluation and subject identification for retrieval/update of personal profiles. In this work, we develop techniques based on learning models for inferences regarding exercise recognition, prediction of the subject's biometrics, exercise evaluation, and simultaneous recognition and evaluation. We address four indoor exercises: squats, push-ups, shoulder-press, and lunges. Different sorts of exercise evaluation are provided: (1) the overall-binary assessment, with a detailed assessment rendering the exact mistakes performed, and (2) the score-based evaluation giving a metric in the range of 0-10 for each aspect of the exercise. Thermal and RGB imagery are examined with various descriptors, such as raw images, Gait Energy Image (GEI), and skeletal pose, with a comparison between their performance. The GEI descriptor extracted from RGB imagery is introduced for exercise recognition and evaluation tasks achieving remarkable results. The GEI descriptor is further utilized for subject recognition and verification from the performed exercise showing the ability of subject discrimination from the way of exercising. Consequently, we introduce new kinds of biometrics based on indoor exercising inspired by the notion that individuals are recognizable from the fingerprint motion dynamics in exercise regimes. To the best of our knowledge, our work is the first to use GEI in such directions and show its remarkable effectiveness.