Leveraging LLMs for Dialogue Quality Measurement
arxiv(2024)
Abstract
In task-oriented conversational AI evaluation, unsupervised methods poorly
correlate with human judgments, and supervised approaches lack generalization.
Recent advances in large language models (LLMs) show robust zeroshot and
few-shot capabilities across NLP tasks. This paper explores using LLMs for
automated dialogue quality evaluation, experimenting with various
configurations on public and proprietary datasets. Manipulating factors such as
model size, in-context examples, and selection techniques, we examine
"chain-of-thought" (CoT) reasoning and label extraction procedures. Our results
show that (1) larger models yield more accurate dialogue labels; (2)
algorithmic selection of in-context examples outperforms random selection; (3)
CoT reasoning where an LLM is asked to provide justifications before outputting
final labels improves performance; and (4) fine-tuned LLMs outperform
out-of-the-box ones. Our results indicate that LLMs that are suitably
fine-tuned and have sufficient reasoning capabilities can be leveraged for
automated dialogue evaluation.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined