Providing a greater precision of Situational Awareness of urban floods through Multimodal Fusion

Floods are a source of anxiety for the people living in the city of São Paulo, Brazil. Every year, the city suffers a financial loss of more than US$ 35 million caused by damage to property, and countless lives are lost as a result of the flooding. Strategies such as Disaster Management can reduce and prevent flash floods and also assist their victims. Moreover, social networks such as Twitter can play a crucial role in offering assistance at the Disaster Management response stage because they produce a massive number of localized geo messages, which can help identify the flood victims. We argue that the mining of social network opinion raises a severe challenge since the Machine Learning (ML) algorithms cannot reflect the context of the messages in-depth, and thus this needs to be improved by combining textual data with contextual data. In this study, we combine multiple sources of weather data with the social network posts to obtain a Situational Awareness (SAW) of flash floods and hence be able to support the Disaster Management Response stage in São Paulo. We show that by combining meteorological with social network data, we can identify the flood victims with a greater degree of precision. The model that was designed for identifying the victims of flooding in São Paulo achieved a 87.69% rate of precision. Furthermore, contextual data inclusion led to a 22.8% increase in SAW of urban floods from tweets and contextual data, which shows that multimodal approaches are more promising than unimodal strategies. Finally, this work adopts a novel approach first by demonstrating that simply applying social network posts to ML strategies is not an efficient method of obtaining a SAW of flash floods. Second, we proved through this study that empirical strategies for establishing potential flood areas are more effective than the adoption of geostatistical approaches (Semivariogram) because the Semivariogram technique is more suitable for understanding scenarios that have not had any prior human interference or damage (e.g., when locating mineral reserves). Thus, particularly in the case of São Paulo, when the trash is disposed of close to drainage systems, this causes clogging of gutters and hence leads to floods.