Graph-based linguistic and visual information integration for on-site occupational hazards identification

Construction sites are hazardous with various potential hazards that can occur at any time. The combination of different factors always causes the construction fatalities, and the majority of these fatalities could be prevented if workers followed on-site regulatory rules. However, compliance of regulatory rules is not strictly enforced among workers due to all kinds of reasons. Although previously proposed vision-based approaches are available for occupational hazards identification, the practicality is limited by the lack of automated understanding and adaptability to regulatory rules changes. In response to these gaps, this paper proposes a novel graph-based framework that integrates linguistic and visual information to process regulatory rule sentences and images for on-site occupational hazards identification. Particularly, a regulatory rules processing approach is presented to automatically extract and represent the key linguistic information of regulatory rules and a vision-based image scene information understanding approach is introduced to process on-site images by the combination of deep learning-based object detection and individual detection using geometric relationships analysis. Additionally, an automated reasoning approach is proposed to provide the integration of the processed linguistic and visual information and perform hazards identification. The hazards of two scenes, i.e., "working on height" and "oper-ating a grinder", were successfully identified with significantly higher performance compared to the baseline model.