Training LLMs over Neurally Compressed Text
arxiv(2024)
摘要
In this paper, we explore the idea of training large language models (LLMs)
over highly compressed text. While standard subword tokenizers compress text by
a small factor, neural text compressors can achieve much higher rates of
compression. If it were possible to train LLMs directly over neurally
compressed text, this would confer advantages in training and serving
efficiency, as well as easier handling of long text spans. The main obstacle to
this goal is that strong compression tends to produce opaque outputs that are
not well-suited for learning. In particular, we find that text naïvely
compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
this, we propose Equal-Info Windows, a novel compression technique whereby text
is segmented into blocks that each compress to the same bit length. Using this
method, we demonstrate effective learning over neurally compressed text that
improves with scale, and outperforms byte-level baselines by a wide margin on
perplexity and inference speed benchmarks. While our method delivers worse
perplexity than subword tokenizers for models trained with the same parameter
count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
require fewer autoregressive generation steps, and reduce latency. Finally, we
provide extensive analysis of the properties that contribute to learnability,
and offer concrete suggestions for how to further improve the performance of
high-compression tokenizers.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要