Intra- and Inter-Image Causal Intervention for Robust Semantic Segmentation in Remote-Sensing Images

Semantic segmentation in remote-sensing (RS) images is a critical component for advanced RS scene understanding. Traditional methods learn to segment using direct supervision of paired data, yet fall short by not fully addressing the noncausal biases behind RS images. These include intra-image biases, such as varying background and object appearance, and inter-image biases, such as inconsistent geographical environments and class distributions, which can mislead models to infer by relying on these biases. Given the high cost and effort needed for pixelwise annotation, the limited volume of RS images exacerbates this problem. In this letter, we propose a novel intra- and inter-image causal intervention (I-3-CI) learning paradigm, designed specifically to mitigate the influence of both biases on RS semantic segmentation. Specifically, to reduce the intra-image biases, our I-3-CI utilizes intra-image contrastive learning to promote closer alignment of features from the same category and distancing those of disparate categories in the same image. Consequently, the compact feature space is more robust to the varying appearance. As for eliminating the inter-image level biases, our I-3-CI further explores the inter-image contrastive learning of all pixels within the entire dataset. To overcome the difficulty of learning from all pixels of the dataset simultaneously, our I-3-CI introduces a set of proxy prototype features to keep track of the global centroids for the features of different categories. We substantiate the I-3-CI paradigm's efficacy through rigorous testing on the LoveDA, Vaihingen, and Potsdam datasets with an average improvement of 0.55% on the mIoU metric, proving the value of causal interventions in achieving robust and accurate semantic segmentation.