Artificial intelligence-aided intelligent obstacle and trespasser detection based on locomotive-mounted forward-facing camera data

Most fatalities in the railroad industry are trespasser deaths. Additional obstacles such as fallen trees, construction materials, and other debris also pose a danger to railroad operations. Understanding where, when, and how these events occur can help railroads develop trespasser mitigation strategies and improve rail safety. However, obtaining information from forward-facing footage requires immense manual effort. We developed a forward-facing trespassing and obstacle detection system that utilizes artificial intelligence to recognize and understand unsafe events and logs them for later review and analysis. The system dynamically identifies regions of interest, trespassers, and obstacles with a customized detection methodology which uses a semantic segmentation model called DeepLab and an object detection model called YOLOv5. These models were trained by a dataset containing over 10,000 images from various sources, including open-source datasets and videos provided by industry collaborators. The novelties of this research are threefold: the development of an algorithm capable of detecting the railroad's restricted area around the track area, optimizing the trade-off between accuracy and latency to achieve real-time performance, and proposing a universal obstacle detection algorithm. The final system was able to analyze 400-pixel x 400-pixel videos at a rate of 14.5 frames per second (FPS) in an edge-computing device which identified trespassers and obstacles with more than 92% accuracy. The system can adaptively detect both obstacles and trespassers in the train's path, as well as the railroad property area surrounding the tracks. This research offers the railroad industry tools which can provide precise trespasser and obstacle information to improve railroad safety and operations.