Visual perception system design for rock breaking robot based on multi-sensor fusion

In recent years, mining automation has received significant attention as a critical focus area. Rock breaking robots are commonly used equipment in the mining industry, and their automation requires an accurate and fast visual perception system. Currently, rock detection and determination of rock breaking surfaces heavily rely on operator experience. To address this, this paper leverages multi-sensor fusion techniques, specifically camera and lidar fusion, as the perception system for the rock breaking robot. The advanced PP-YOLO series algorithm is employed for 2D detection, enabling the generation of specific detection results based on the breaking requirements. Furthermore, 3D reconstruction of rocks detected in the 2D area is performed using point cloud data. The extraction of rock breaking surfaces is achieved through point cloud segmentation and statistical filtering methods. Experimental results demonstrate a rock detection speed of 13.8 ms and the mAP value of 91.2%. The segmentation accuracy for rock breaking surfaces is 75.46%, with an average recall of 91.08%. The segmentation process takes 73.09 ms, thus meeting the real-time detection and segmentation needs within the specified rock breaking range. This study effectively addresses the limitations associated with single sensor information.