PuzzleTuning: Explicitly Bridge Pathological and Natural Image with Puzzles
arXiv (Cornell University)(2023)
Abstract
Pathological image analysis is a crucial field in computer vision. Due to the
annotation scarcity in the pathological field, recently, most of the works have
leveraged self-supervised learning (SSL) trained on unlabeled pathological
images, hoping to mine the representation effectively. However, there are two
core defects in current SSL-based pathological pre-training: (1) they do not
explicitly explore the essential focuses of the pathological field, and (2)
they do not effectively bridge with and thus take advantage of the knowledge
from natural images. To explicitly address them, we propose our large-scale
PuzzleTuning framework, containing the following innovations. Firstly, we
identify three task focuses that can effectively bridge knowledge of
pathological and natural domain: appearance consistency, spatial consistency,
and restoration understanding. Secondly, we devise a novel multiple puzzle
restoring task, which explicitly pre-trains the model regarding these focuses.
Thirdly, we introduce an explicit prompt-tuning process to incrementally
integrate the domain-specific knowledge, aligning the large domain gap between
natural and pathological images. Additionally, a curriculum-learning training
strategy is designed to regulate task difficulty, making the model adaptive to
the puzzle restoring complexity. Experimental results show that our
PuzzleTuning framework outperforms the previous state-of-the-art methods in
various downstream tasks on multiple datasets. The code, demo, and pre-trained
weights are available at https://github.com/sagizty/PuzzleTuning.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined