Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
conf_acl(2023)
摘要
Evaluation of natural language generation (NLG) is complex and
multi-dimensional. Generated text can be evaluated for fluency, coherence,
factuality, or any other dimensions of interest. Most frameworks that perform
such multi-dimensional evaluation require training on large manually or
synthetically generated datasets. In this paper, we study the efficacy of large
language models as multi-dimensional evaluators using in-context learning,
obviating the need for large training datasets. Our experiments show that
in-context learning-based evaluators are competitive with learned evaluation
frameworks for the task of text summarization, establishing state-of-the-art on
dimensions such as relevance and factual consistency. We then analyze the
effects of factors such as the selection and number of in-context examples on
performance. Finally, we study the efficacy of in-context learning based
evaluators in evaluating zero-shot summaries written by large language models
such as GPT-3.
更多查看译文
关键词
text summarization,evaluation,learning,multi-dimensional,in-context
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要