A hybrid cloud detection and cloud phase classification algorithm using classic threshold-based tests and extra randomized tree model

The detection of clouds from the latest-generation satellites is crucial for the accuracy of downstream remote sensing products. Classic threshold-based algorithms or machine-learning-based algorithms have limitations in all-day cloud detection, usually sacrificing measurements from visible bands to develop all-day cloud detections. It is difficult to find a day-night consistent cloud detection that takes advantage of all band measurements even if there is an abrupt absence of visible bands on one sides of the terminator. This study explains a cloud detection algorithm for the Cloud, Atmospheric Radiation, and renewal Energy application (CARE) datasets that synergistically uses threshold-based tests and an extra randomized tree (ERT) model to detect all-day clouds from Himawari-8 full-disk measurements. First, we developed a visible-to-infrared band daytime cloud detection algorithm (D algorithm) to generate high-precision training datasets, based on which an infrared band all-day cloud detection algorithm (DN algorithm) is built with an ERT model. Second, we merge the D and DN algorithm into CARE algorithm by weighting the results of the two algorithms differently according to day/night and land/sea conditions. In developing D algorithm, we use two-layer curved-surface thresholds from a look-up-table (LUT), that can effectively improve the dusk-dawn cloud detection accuracy over high latitudes and reduce the misclassification of clouds over sunglint regions. Validations against CALIPSO measurements show that the hit rate (HR) of CARE results (daytime cloudy HR 76.95%, clear HR 87.97%, nighttime cloudy HR 73.19%, clear HR 81.77%) outperform MODIS (daytime cloudy HR 77.35%, clear HR 66.09%, nighttime cloudy HR 78.08%, clear HR 59.72%) especially in nighttime clear sky detections, daytime CARE results show lower and higher HR in water/ice cloud detections than JAXA and MODIS results, nighttime CARE (water cloud HR 47.82%, ice cloud HR 93.59%) results show higher HR in both water and ice clouds than MODIS (water cloud HR 43.39%, ice cloud HR 74.41%). The comparison between the CARE and JAXA products shows high consistency during the daytime, and the CARE products have better cloud detection in the sunglint center and in dusk-dawn scenarios.