Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining
CoRR(2024)
摘要
Self-supervised pretraining on large-scale satellite data has raised great
interest in building Earth observation (EO) foundation models. However, many
important resources beyond pure satellite imagery, such as land-cover-land-use
products that provide free global semantic information, as well as vision
foundation models that hold strong knowledge of the natural world, tend to be
overlooked. In this work, we show these free additional resources not only help
resolve common contrastive learning bottlenecks, but also significantly boost
the efficiency and effectiveness of EO pretraining.
Specifically, we first propose soft contrastive learning that optimizes
cross-scene soft similarity based on land-cover-generated multi-label
supervision, naturally solving the issue of multiple positive samples and too
strict positive matching in complex scenes. Second, we explore cross-domain
continual pretraining for both multispectral and SAR imagery, building
efficient EO foundation models from strongest vision models such as DINOv2.
Integrating simple weight-initialization and Siamese masking strategies into
our soft contrastive learning framework, we demonstrate impressive continual
pretraining performance even when the input channels and modalities are not
aligned.
Without prohibitive training, we produce multispectral and SAR foundation
models that achieve significantly better results in 9 out of 10 downstream
tasks than most existing SOTA models. For example, our ResNet50/ViT-S achieve
84.8/85.0 linear probing mAP scores on BigEarthNet-10% which are better than
most existing ViT-L models; under the same setting, our ViT-B sets a new record
of 86.8 in multispectral, and 82.5 in SAR, the latter even better than many
multispectral models. Dataset and models are available at
https://github.com/zhu-xlab/softcon.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要