Contextual multimodal approach for recognizing concurrent activities of equipment in tunnel construction projects

In order to accurately track progress and improve efficiency in complex construction projects, it's important to effectively monitor individual tasks and measure the time taken to complete a cycle of tasks. Tunnel construction involves a variety of activities, where multiple pieces of equipment are engaged in different actions that occur simultaneously or sequentially during a single activity. This study introduces a contextual audio-visual (multimodal) approach to better recognize multi-equipment activities in a tunnel construction site for monitoring purposes. By incorporating both audio and visual data, and by integrating both spatial and cyclical temporal contexts, the model accurately recognizes the activity being performed by multiple pieces of equipment more often than single-mode models. Tested against real-world operation data, the model achieved a remarkable Fscore of 96.3% in recognizing construction activities, demonstrating its superiority over traditional methods in scenarios involving multiple, simultaneously operating pieces of equipment. The results emphasize the potential of contextual multimodal models in enhancing operational efficiency in complex construction sites.