DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models
CoRR(2024)
Abstract
We present DIALIGHT, a toolkit for developing and evaluating multilingual
Task-Oriented Dialogue (ToD) systems which facilitates systematic evaluations
and comparisons between ToD systems using fine-tuning of Pretrained Language
Models (PLMs) and those utilising the zero-shot and in-context learning
capabilities of Large Language Models (LLMs). In addition to automatic
evaluation, this toolkit features (i) a secure, user-friendly web interface for
fine-grained human evaluation at both local utterance level and global dialogue
level, and (ii) a microservice-based backend, improving efficiency and
scalability. Our evaluations reveal that while PLM fine-tuning leads to higher
accuracy and coherence, LLM-based systems excel in producing diverse and
likeable responses. However, we also identify significant challenges of LLMs in
adherence to task-specific instructions and generating outputs in multiple
languages, highlighting areas for future research. We hope this open-sourced
toolkit will serve as a valuable resource for researchers aiming to develop and
properly evaluate multilingual ToD systems and will lower, currently still
high, entry barriers in the field.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined