Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
arxiv(2024)
摘要
Large models represent a groundbreaking advancement in multiple application
fields, enabling remarkable achievements across various tasks. However, their
unprecedented scale comes with significant computational costs. These models,
often consisting of billions of parameters, require vast amounts of
computational resources for execution. Especially, the expansive scale and
computational demands pose considerable challenges when customizing them for
particular downstream tasks, particularly over the hardware platforms
constrained by computational capabilities. Parameter Efficient Fine-Tuning
(PEFT) provides a practical solution by efficiently adapt the large models over
the various downstream tasks. In particular, PEFT refers to the process of
adjusting the parameters of a pre-trained large models to adapt it to a
specific task while minimizing the number of additional parameters introduced
or computational resources required. This approach is particularly important
when dealing with large language models with high parameter counts, as
fine-tuning these models from scratch can be computationally expensive and
resource-intensive, posing considerable challenges in the supporting system
platform design. In this survey, we present comprehensive studies of various
PEFT algorithms, examining their performance and computational overhead.
Moreover, we provide an overview of applications developed using different PEFT
algorithms and discuss common techniques employed to mitigate computation costs
for PEFT. In addition to the algorithmic perspective, we overview various
real-world system designs to investigate the implementation costs associated
with different PEFT algorithms. This survey serves as an indispensable resource
for researchers aiming to understand both the PEFT algorithm and its system
implementation, offering detailed insights into recent advancements and
practical applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要