Optimizing Speculative Decoding for Serving Large Language Models Using Goodput
CoRR(2024)
摘要
Reducing the inference latency of large language models (LLMs) is crucial,
and speculative decoding (SD) stands out as one of the most effective
techniques. Rather than letting the LLM generate all tokens directly,
speculative decoding employs effective proxies to predict potential outputs,
which are then verified by the LLM without compromising the generation quality.
Yet, deploying SD in real online LLM serving systems (with continuous batching)
does not always yield improvement – under higher request rates or low
speculation accuracy, it paradoxically increases latency. Furthermore, there is
no best speculation length work for all workloads under different system loads.
Based on the observations, we develop a dynamic framework SmartSpec. SmartSpec
dynamically determines the best speculation length for each request (from 0,
i.e., no speculation, to many tokens) – hence the associated speculative
execution costs – based on a new metric called goodput, which characterizes
the current observed load of the entire system and the speculation accuracy. We
show that SmartSpec consistently reduces average request latency by up to 3.2x
compared to non-speculative decoding baselines across different sizes of target
models, draft models, request rates, and datasets. Moreover, SmartSpec can be
applied to different styles of speculative decoding, including traditional,
model-based approaches as well as model-free methods like prompt lookup and
tree-style decoding.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要