Learn To be Efficient: Build Structured Sparsity in Large Language Models
CoRR(2024)
Abstract
Large Language Models (LLMs) have achieved remarkable success with their
billion-level parameters, yet they incur high inference overheads. The
emergence of activation sparsity in LLMs provides a natural approach to reduce
this cost by involving only parts of the parameters for inference. However,
existing methods only focus on utilizing this naturally formed activation
sparsity in a post-training setting, overlooking the potential for further
amplifying this inherent sparsity. In this paper, we hypothesize that LLMs can
learn to be efficient by achieving more structured activation sparsity. To
achieve this, we introduce a novel training algorithm, Learn-To-be-Efficient
(LTE), designed to train efficiency-aware LLMs to learn to activate fewer
neurons and achieve a better trade-off between sparsity and performance.
Furthermore, unlike SOTA MoEfication methods, which mainly focus on ReLU-based
models, LTE can also be applied to LLMs like LLaMA using non-ReLU activations.
Extensive evaluation on language understanding, language generation, and
instruction tuning tasks show that LTE consistently outperforms SOTA baselines.
Along with our hardware-aware custom kernel implementation, LTE reduces
LLaMA2-7B inference latency by 25
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined