Communication-Constrained Bayesian Active Knowledge Distillation
CoRR(2023)
摘要
Consider an active learning setting in which a learner has a training set
with few labeled examples and a pool set with many unlabeled inputs, while a
remote teacher has a pre-trained model that is known to perform well for the
learner's task. The learner actively transmits batches of unlabeled inputs to
the teacher through a constrained communication channel for labeling. This
paper addresses the following key questions: (i) Active batch selection: Which
batch of inputs should be sent to the teacher to acquire the most useful
information and thus reduce the number of required communication rounds? (ii)
Batch encoding: How do we encode the batch of inputs for transmission to the
teacher to reduce the communication resources required at each round? We
introduce Communication-Constrained Bayesian Active Knowledge Distillation
(CC-BAKD), a novel protocol that integrates Bayesian active learning with
compression via a linear mix-up mechanism. Bayesian active learning selects the
batch of inputs based on their epistemic uncertainty, addressing the
"confirmation bias" that is known to increase the number of required
communication rounds. Furthermore, the proposed mix-up compression strategy is
integrated with the epistemic uncertainty-based active batch selection process
to reduce the communication overhead per communication round.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要