Lessons learned from cloudsen12 dataset: identifying incorrect annotations in cloud semantic segmentation datasets

In Earth observation, deep learning models rely heavily on comprehensive datasets for training and evaluation. However, the relevance of data quality is often underestimated, leading to subpar generalization in real-world remote sensing scenarios. This study aims to bridge this gap by proposing a straight-forward method to identify critical human annotation errors in semantic segmentation datasets. The approach is based on two indices: trustworthiness and hardness. By implementing these indices, we estimate the extent of human annotation errors in CloudSEN12, a global dataset specifically designed for cloud detection in Sentinel-2 imagery. Considering only the trustworthiness index, our approach identified 1794 potential labelling errors among 10,000 image patches. Out of these, 106 were confirmed as human errors, resulting in a true positive rate of 9.86%. When this method was applied to other extensive cloud masking datasets, such as KappaSet and Sentinel-2 Cloud Mask Catalogue, it was found that over 44% of the human labels were inaccurate. These results do not imply the inferior quality of these datasets, instead, they highlight the considerable shift between the annotation protocols, making inter-dataset benchmarking exercises inequitable.