MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
ICLR 2024(2024)
摘要
Multimodal intent recognition poses significant challenges, requiring the
incorporation of non-verbal modalities from real-world contexts to enhance the
comprehension of human intentions. Existing benchmark datasets are limited in
scale and suffer from difficulties in handling out-of-scope samples that arise
in multi-turn conversational interactions. We introduce MIntRec2.0, a
large-scale benchmark dataset for multimodal intent recognition in multi-party
conversations. It contains 1,245 dialogues with 15,040 samples, each annotated
within a new intent taxonomy of 30 fine-grained classes. Besides 9,304 in-scope
samples, it also includes 5,736 out-of-scope samples appearing in multi-turn
contexts, which naturally occur in real-world scenarios. Furthermore, we
provide comprehensive information on the speakers in each utterance, enriching
its utility for multi-party conversational research. We establish a general
framework supporting the organization of single-turn and multi-turn dialogue
data, modality feature extraction, multimodal fusion, as well as in-scope
classification and out-of-scope detection. Evaluation benchmarks are built
using classic multimodal fusion methods, ChatGPT, and human evaluators. While
existing methods incorporating nonverbal information yield improvements,
effectively leveraging context information and detecting out-of-scope samples
remains a substantial challenge. Notably, large language models exhibit a
significant performance gap compared to humans, highlighting the limitations of
machine learning methods in the cognitive intent understanding task. We believe
that MIntRec2.0 will serve as a valuable resource, providing a pioneering
foundation for research in human-machine conversational interactions, and
significantly facilitating related applications. The full dataset and codes are
available at https://github.com/thuiar/MIntRec2.0.
更多查看译文
关键词
intent recognition,multimodal dataset,out-of-scope detection,multi-turn conversations,benchmark framework,benchmark evaluation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要