Convolutional neural network and sensor fusion for obstacle classification in the context of powered prosthetic leg applications

Powered prostheses for lower extremities are an advancing technology, with recent developments reducing their weight while increasing their autonomy and capabilities. One problem that powered prosthesis users face is the change of gait pattern that is associated with navigating structures such as staircases or ramps. While a powered prosthesis may implement more than one mode of locomotion to address those transitions, a recent area of study is their automatic detection. Previous developments have used different sensor combinations, including wearable cameras, to anticipate the change between level-ground walking and stair climbing. We propose combining information from a camera with signals from an inertial measurement unit, thus creating a system that is less susceptible to false classifications. The principle behind using inertial data is to provide useful information to classify an object as an obstacle only when the user is going to pass or is passing through it. To this end, we modify the architectures of different neural networks that were designed to classify images, extending them to handle inertial information as well. Our results show that the introduction of inertial information increases several performance metrics for each network, effectively reducing misclassifications. All chosen network architectures provide real-time performance, even on embedded hardware. Introducing inertial sensor information increases the Matthews correlation coefficient (MCC) of the neural networks, with gains ranging from 0.85% to 3.49%. The proposed sensor fusion model achieves accuracy over 98%, and an MCC of 0.9137.