AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning
CoRR(2024)
Abstract
Communication bottlenecks hinder the scalability of distributed neural
network training, particularly on distributed-memory computing clusters. To
significantly reduce this communication overhead, we introduce AB-training, a
novel data-parallel training method that decomposes weight matrices into
low-rank representations and utilizes independent group-based training. This
approach consistently reduces network traffic by 50
scenarios, increasing the training potential on communication-constrained
systems. Our method exhibits regularization effects at smaller scales, leading
to improved generalization for models like VGG16, while achieving a remarkable
44.14 : 1 compression ratio during training on CIFAR-10 and maintaining
competitive accuracy. Albeit promising, our experiments reveal that large batch
effects remain a challenge even in low-rank training regimes.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined