Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model
arxiv(2023)
摘要
Parameter-efficient fine-tuning (PEFT) is an effective methodology to unleash
the potential of large foundation models in novel scenarios with limited
training data. In the computer vision community, PEFT has shown effectiveness
in image classification, but little research has studied its ability for image
segmentation. Fine-tuning segmentation models usually require a heavier
adjustment of parameters to align the proper projection directions in the
parameter space for new scenarios. This raises a challenge to existing PEFT
algorithms, as they often inject a limited number of individual parameters into
each block, which prevents substantial adjustment of the projection direction
of the parameter space due to the limitation of Hidden Markov Chain along
blocks. In this paper, we equip PEFT with a cross-block orchestration mechanism
to enable the adaptation of the Segment Anything Model (SAM) to various
downstream scenarios. We introduce a novel inter-block communication module,
which integrates a learnable relation matrix to facilitate communication among
different coefficient sets of each PEFT block's parameter space. Moreover, we
propose an intra-block enhancement module, which introduces a linear projection
head whose weights are generated from a hyper-complex layer, further enhancing
the impact of the adjustment of projection directions on the entire parameter
space. Extensive experiments on diverse benchmarks demonstrate that our
proposed approach consistently improves the segmentation performance
significantly on novel scenarios with only around 1K additional parameters.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要