Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders
arxiv(2024)
摘要
Conversational systems often rely on embedding models for intent
classification and intent clustering tasks. The advent of Large Language Models
(LLMs), which enable instructional embeddings allowing one to adjust semantics
over the embedding space using prompts, are being viewed as a panacea for these
downstream conversational tasks. However, traditional evaluation benchmarks
rely solely on task metrics that don't particularly measure gaps related to
semantic understanding. Thus, we propose an intent semantic toolkit that gives
a more holistic view of intent embedding models by considering three tasks–
(1) intent classification, (2) intent clustering, and (3) a novel triplet task.
The triplet task gauges the model's understanding of two semantic concepts
paramount in real-world conversational systems– negation and implicature. We
observe that current embedding models fare poorly in semantic understanding of
these concepts. To address this, we propose a pre-training approach to improve
the embedding model by leveraging augmentation with data generated by an
auto-regressive model and a contrastive loss term. Our approach improves the
semantic understanding of the intent embedding model on the aforementioned
linguistic dimensions while slightly effecting their performance on downstream
task metrics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要