Towards Smart Big Weather Data Management

Smart management of weather data is pivotal to achieving sustainable agriculture since weather monitoring is linked to crop water requirement estimation and consequently to efficient irrigation systems. Advances in technologies such as remote sensing and the Internet of Things (IoT) have led to the generation of this data with a high temporal resolution which requires adequate infrastructure and processing tools to gain insights from it. To this end, this paper presents a smart weather data management system composed of three layers: the data acquisition layer, the data storage layer, and the application layer. The data can be sourced from station sensors, real-time IoT sensors, third-party services (APIs), or manually imported from files. It is then checked for errors and missing values before being stored using the distributed database MongoDB. The platform provides various services related to weather data: (i) forecast univariate weather time series, (ii) perform advanced analysis and visualization, (iii) use machine learning to estimate and model important climatic parameters such as the reference evapotranspiration (ET0) estimation using the XGBoost model (R2 = 0.96 and RMSE = 0.39). As part of a test phase, the system uses data from a meteorological station installed in the study area in Morocco.