Set the Clock: Temporal Alignment of Pretrained Language Models
CoRR(2024)
Abstract
Language models (LMs) are trained on web text originating from many points in
time and, in general, without any explicit temporal grounding. This work
investigates the temporal chaos of pretrained LMs and explores various methods
to align their internal knowledge to a target time, which we call "temporal
alignment." To do this, we first automatically construct a dataset containing
20K time-sensitive questions and their answers for each year from 2000 to 2023.
Based on this dataset, we empirically show that pretrained LMs (e.g., LLaMa2),
despite having a recent pretraining cutoff (e.g., 2022), mostly answer
questions using earlier knowledge (e.g., in 2019). We then develop several
methods, from prompting to finetuning, to align LMs to use their most recent
knowledge when answering questions, and investigate various factors in this
alignment. Our experiments show that aligning LLaMa2 to the year 2022 can boost
its performance by up to 62
mentioning time information explicitly, indicating the possibility of aligning
models' internal sense of time after pretraining. Finally, we find that
alignment to a historical time is also possible, with up to 2.8× the
performance of the unaligned LM in 2010 if finetuning models to that year.
These findings hint at the sophistication of LMs' internal knowledge
organization and the necessity of tuning them properly.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined