TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Data
CoRR(2024)
摘要
Distributed training of deep neural networks faces three critical challenges:
privacy preservation, communication efficiency, and robustness to fault and
adversarial behaviors. Although significant research efforts have been devoted
to addressing these challenges independently, their synthesis remains less
explored. In this paper, we propose TernaryVote, which combines a ternary
compressor and the majority vote mechanism to realize differential privacy,
gradient compression, and Byzantine resilience simultaneously. We theoretically
quantify the privacy guarantee through the lens of the emerging f-differential
privacy (DP) and the Byzantine resilience of the proposed algorithm.
Particularly, in terms of privacy guarantees, compared to the existing
sign-based approach StoSign, the proposed method improves the dimension
dependence on the gradient size and enjoys privacy amplification by mini-batch
sampling while ensuring a comparable convergence rate. We also prove that
TernaryVote is robust when less than 50
matches that of SIGNSGD with majority vote. Extensive experimental results
validate the effectiveness of the proposed algorithm.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要