Distributed Speculative Inference of Large Language Models
CoRR(2024)
摘要
Accelerating the inference of large language models (LLMs) is an important
challenge in artificial intelligence. This paper introduces distributed
speculative inference (DSI), a novel distributed inference algorithm that is
provably faster than speculative inference (SI) [leviathan2023fast,
chen2023accelerating, miao2023specinfer] and traditional autoregressive
inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
requiring no training or architectural modifications, and it preserves the
target distribution.
Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
often do not have matching drafters that are sufficiently fast and accurate. We
show a gap: SI gets slower than non-SI when using slower or less accurate
drafters. We close this gap by proving that DSI is faster than both SI and
non-SI given any drafters. By orchestrating multiple instances of the target
and drafters, DSI is not only faster than SI but also supports LLMs that cannot
be accelerated with SI.
Our simulations show speedups of off-the-shelf LLMs in realistic settings:
DSI is 1.29-1.92x faster than SI.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要