Beyond quadratic error: Case-study of a multiple criteria approach to the performance assessment of numerical forecasts of solar irradiance in the tropics

As the penetration of photovoltaic power in the electrical grid increases, reliable forecasts of solar irradiance are becoming ever more critical. For forecasting horizons beyond a few hours, Numerical Weather Prediction models (NWP) are stapled to yield the best results. For potential users of NWP forecasts, choosing which model to use can be challenging, as a large palette of NWP models and configurations is available. To tackle this issue, several studies have conducted systematic comparisons between available models and configurations. Most of these studies, however, only evaluate forecast skills with a small number of metrics, such as the Root Mean Square Error (RMSE) or the Mean Average Error (MAE). To better understand the nature and specificities of various NWP models, we developed an evaluation approach that combines multiple-criteria analysis, Fourier analysis and classification of daily profiles of irradiance. In this paper, we present this novel approach and demonstrate its strengths by applying it to a practical use-case, using 108 different NWP forecasts models based on different parametrizations of the mesoscale NWP Weather Research and Forecasting model (WRF) in Singapore. Our approach allowed a clearer overview of the forecasts’ skills, as well as better discrimination between models. It also significantly improved our understanding of the nature of the NWP errors. In particular, WRF forecasts in Singapore were found unfit to timely resolve irradiance at an hourly scale, but better adapted to predict daily profiles of irradiance. Furthermore, the proposed multiple criteria approach was applied to a sensitivity analysis of WRF physical schemes, providing a better insight into the impact of each model. This work shows that it is critical to go beyond RMSE when evaluating forecasts and that more holistic approaches such as the one proposed here should be preferred.