Dodo: Dynamic Contextual Compression for Decoder-only LMs
CoRR(2023)
Abstract
Transformer-based language models (LMs) are inefficient in long contexts. We
propose Dodo, a solution for context compression. Instead of one vector per
token in a standard transformer model, Dodo represents text with a dynamic
number of hidden states at each layer, reducing the cost of self-attention to a
fraction of typical time and space. Moreover, off-the-shelf models such as
LLaMA can be adapted to Dodo by efficient parameter tuning methods such as
LoRA. In use, Dodo can act as either an autoregressive LM or a context
compressor for downstream tasks. We demonstrate through experiments in language
modeling, question answering, and summarization that Dodo retains capabilities
in these tasks, while drastically reducing the overhead during decoding. For
example, in the autoencoding task, Dodo shrinks context at a 20x compression
ratio with a BLEU score of 98
encoding.
MoreTranslated text
Key words
dynamic contextual compression,scaling,language,decoder-only
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined