Multiview Video-Based 3-D Pose Estimation of Patients in Computer-Assisted Rehabilitation Environment (CAREN)

The computer-assisted rehabilitation environment (CAREN) system plays an important role in the training of rehabilitation patients, where the capture of the patient's 3-D pose and gait is critical for assessing the patient's requirements for effective training. Vision-based methods are highly effective for this task due to their low cost, high speed, and noninterference. Although various general vision-based pose estimation methods were developed recently, their performance is limited in the CAREN system due to the specific environment. To address these problems, we propose an improved framework for accurate 2-D and 3-D pose estimation for the CAREN system through using multiview videos. First, for 2-D pose estimation, we propose a coarse-to-fine heatmap shrinking (CFHS) strategy that gradually reduces the kernel size of the heatmap of joints during training to improve the performance. Second, to further obtain 3-D pose estimations, we propose a novel spatial-temporal perception network that fuses the 2-D results from multiple views and multiple moments; multiview early fusion uses complementary spatial information from different views, and multimoment late fusion leverages temporal information from the sequential input for higher accuracy. The experimental results, based on CAREN videos of 225 orthopedic patients, showed that the accuracy of 2-D human pose estimations with the CFHS training strategy reached 99.85% PCKh@0.5. For 3-D results, the mean per joint position error was 25.22 mm, and the 3DPCK reached 98.71%, which outperformed existing general video-based methods. The results showed that the proposed system is capable of estimating human poses with high accuracy for clinical applications.