A Realistic Evaluation of LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3
arxiv(2024)
Abstract
Large Language Models (LLMs) zero-shot and few-shot performance are subject
to memorization and data contamination, complicating the assessment of their
validity. In literary tasks, the performance of LLMs is often correlated to the
degree of book memorization. In this work, we carry out a realistic evaluation
of LLMs for quotation attribution in novels, taking the instruction fined-tuned
version of Llama3 as an example. We design a task-specific memorization measure
and use it to show that Llama3's ability to perform quotation attribution is
positively correlated to the novel degree of memorization. However, Llama3
still performs impressively well on books it has not memorized nor seen. Data
and code will be made publicly available.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined