Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning
CoRR(2024)
摘要
Recently, multi-task instruction tuning has been applied into sentence
representation learning, which endows the capability of generating specific
representations with the guidance of task instruction, exhibiting strong
generalization ability on new tasks. However, these methods mostly neglect the
potential interference problems across different tasks and instances, which may
affect the training and convergence of the model. To address it, we propose a
data curriculum method, namely Data-CUBE, that arranges the orders of all the
multi-task data for training, to minimize the interference risks from the two
views. In the task level, we aim to find the optimal task order to minimize the
total cross-task interference risk, which is exactly the traveling salesman
problem, hence we utilize a simulated annealing algorithm to find its solution.
In the instance level, we measure the difficulty of all instances per task,
then divide them into the easy-to-difficult mini-batches for training.
Experiments on MTEB sentence representation evaluation tasks show that our
approach can boost the performance of state-of-the-art methods. Our code and
data are publicly available at the link:
.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要