Spectral Co-Distillation for Personalized Federated Learning
NeurIPS 2023(2024)
摘要
Personalized federated learning (PFL) has been widely investigated to address
the challenge of data heterogeneity, especially when a single generic model is
inadequate in satisfying the diverse performance requirements of local clients
simultaneously. Existing PFL methods are inherently based on the idea that the
relations between the generic global and personalized local models are captured
by the similarity of model weights. Such a similarity is primarily based on
either partitioning the model architecture into generic versus personalized
components, or modeling client relationships via model weights. To better
capture similar (yet distinct) generic versus personalized model
representations, we propose spectral distillation, a novel
distillation method based on model spectrum information. Building upon spectral
distillation, we also introduce a co-distillation framework that establishes a
two-way bridge between generic and personalized model training. Moreover, to
utilize the local idle time in conventional PFL, we propose a wait-free local
training protocol. Through extensive experiments on multiple datasets over
diverse heterogeneous data settings, we demonstrate the outperformance and
efficacy of our proposed spectral co-distillation method, as well as our
wait-free training protocol.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要