Accelerating Distributed Deep Learning using Lossless Homomorphic Compression
CoRR(2024)
摘要
As deep neural networks (DNNs) grow in complexity and size, the resultant
increase in communication overhead during distributed training has become a
significant bottleneck, challenging the scalability of distributed training
systems. Existing solutions, while aiming to mitigate this bottleneck through
worker-level compression and in-network aggregation, fall short due to their
inability to efficiently reconcile the trade-offs between compression
effectiveness and computational overhead, hindering overall performance and
scalability. In this paper, we introduce a novel compression algorithm that
effectively merges worker-level compression with in-network aggregation. Our
solution is both homomorphic, allowing for efficient in-network aggregation
without CPU/GPU processing, and lossless, ensuring no compromise on training
accuracy. Theoretically optimal in compression and computational efficiency,
our approach is empirically validated across diverse DNN models such as NCF,
LSTM, VGG19, and BERT-base, showing up to a 6.33× improvement in
aggregation throughput and a 3.74× increase in per-iteration training
speed.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要