On Speculative Decoding for Multimodal Large Language Models
arxiv(2024)
摘要
Inference with Multimodal Large Language Models (MLLMs) is slow due to their
large-language-model backbone which suffers from memory bandwidth bottleneck
and generates tokens auto-regressively. In this paper, we explore the
application of speculative decoding to enhance the inference efficiency of
MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
the need for image tokens and their associated processing components from the
draft model. Our experiments across three different tasks show that speculative
decoding can achieve a memory-bound speedup of up to 2.37× using a 115M
parameter language model that we trained from scratch. Additionally, we
introduce a compact LLaVA draft model incorporating an image adapter, which
shows marginal performance gains in image captioning while maintaining
comparable results in other tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要