A Two-Stage Framework for Bias and Reliability Tests of Ensemble Hydroclimatic Forecasts

The popular probability integral transform (PIT) uniform plot presents informative empirical illustrations of five types of ensemble forecasts, that is, reliable, under-confident, over-confident, negatively biased and positively biased. This paper has built a novel two-stage framework upon the PIT uniform plot to quantitatively examine the forecast attributes of bias and reliability. The first stage utilizes the test statistic on bias to examine whether the mean of PIT values is equal to the theoretical mean of standard uniform distribution. Then, the second stage uses the test statistic on reliability to examine whether the mean squared deviation from the theoretical mean is equal to the theoretical variance of standard uniform distribution. Therefore, by using the two-tailed bootstrap hypothesis testing, the first stage identifies unbiased ensemble forecasts, negatively biased forecasts and positively biased forecasts; the second stage focuses on unbiased ensemble forecasts to furthermore identify reliable forecasts, under-confident forecasts and over-confident forecasts. Numerical experiments are devised for the National Centers for Environmental Prediction's Climate Forecast System version 2 ensemble forecasts of global precipitation. The results highlight the existence of various shapes of the PIT uniform plots. Due to extreme values of observed precipitation, the PIT uniform plots in some cases can substantially deviate from the 1:1 line even though the mean and variance of ensemble forecasts are respectively in accordance with the mean and variance of observations. Nevertheless, the two-stage framework along with the two test statistics serves as a robust tool for the verification of ensemble hydroclimatic forecasts.