Re-parameterized Low-rank Prompt: Generalize a Vision-Language Model within 0.5K Parameters
CoRR(2023)
摘要
With the development of large pre-trained vision-language models, how to
effectively transfer the knowledge of such foundational models to downstream
tasks becomes a hot topic, especially in a data-deficient scenario. Recently,
prompt tuning has become a popular solution. When adapting the vision-language
models, researchers freeze the parameters in the backbone and only design and
tune the prompts. On the one hand, the delicate design of prompt tuning
exhibits strong performance. On the other hand, complicated structures and
update rules largely increase the computation and storage cost. Motivated by
the observation that the evolution pattern of the generalization capability in
visual-language models aligns harmoniously with the trend of rank variations in
the prompt matrix during adaptation, we design a new type of prompt,
Re-parameterized Low-rank Prompt (RLP), for both efficient and effective
adaptation. Our method could largely reduce the number of tunable parameters
and storage space, which is quite beneficial in resource-limited scenarios.
Extensive experiments further demonstrate the superiority of RLP. In
particular, RLP shows comparable or even stronger performance than the latest
state-of-the-art methods with an extremely small number of parameters. On a
series of tasks over 11 datasets, RLP significantly increases the average
downstream accuracy of classic prompt tuning by up to 5.25
parameters.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要