Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification
arxiv(2024)
Abstract
Parameter-efficient transfer learning (PETL) is proposed as a cost-effective
way to transfer pre-trained models to downstream tasks, avoiding the high cost
of updating entire large-scale pre-trained models (LPMs). In this work, we
present Fine-grained Prompt Tuning (FPT), a novel PETL method for medical image
classification. FPT significantly reduces memory consumption compared to other
PETL methods, especially in high-resolution input contexts. To achieve this, we
first freeze the weights of the LPM and construct a learnable lightweight side
network. The frozen LPM takes high-resolution images as input to extract
fine-grained features, while the side network is fed low-resolution images to
reduce memory usage. To allow the side network to access pre-trained knowledge,
we introduce fine-grained prompts that summarize information from the LPM
through a fusion module. Important tokens selection and preloading techniques
are employed to further reduce training cost and memory requirements. We
evaluate FPT on four medical datasets with varying sizes, modalities, and
complexities. Experimental results demonstrate that FPT achieves comparable
performance to fine-tuning the entire LPM while using only 1.8
learnable parameters and 13
a 512 x 512 input resolution.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined