AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models Adapting
CoRR(2024)
Abstract
Recently, prompt-based methods have emerged as a new alternative
`parameter-efficient fine-tuning' paradigm, which only fine-tunes a small
number of additional parameters while keeping the original model frozen.
However, despite achieving notable results, existing prompt methods mainly
focus on `what to add', while overlooking the equally important aspect of
`where to add', typically relying on the manually crafted placement. To this
end, we propose a region-based Adaptive Visual Prompt, named AdaViPro, which
integrates the `where to add' optimization of the prompt into the learning
process. Specifically, we reconceptualize the `where to add' optimization as a
problem of regional decision-making. During inference, AdaViPro generates a
regionalized mask map for the whole image, which is composed of 0 and 1, to
designate whether to apply or discard the prompt in each specific area.
Therefore, we employ Gumbel-Softmax sampling to enable AdaViPro's end-to-end
learning through standard back-propagation. Extensive experiments demonstrate
that our AdaViPro yields new efficiency and accuracy trade-offs for adapting
pre-trained models.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined