Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness
arxiv(2024)
摘要
The task of Information Retrieval (IR) requires a system to identify relevant
documents based on users' information needs. In real-world scenarios,
retrievers are expected to not only rely on the semantic relevance between the
documents and the queries but also recognize the nuanced intents or
perspectives behind a user query. For example, when asked to verify a claim, a
retrieval system is expected to identify evidence from both supporting vs.
contradicting perspectives, for the downstream system to make a fair judgment
call. In this work, we study whether retrievers can recognize and respond to
different perspectives of the queries – beyond finding relevant documents for
a claim, can retrievers distinguish supporting vs. opposing documents? We
reform and extend six existing tasks to create a benchmark for retrieval, where
we have diverse perspectives described in free-form text, besides root, neutral
queries. We show that current retrievers covered in our experiments have
limited awareness of subtly different perspectives in queries and can also be
biased toward certain perspectives. Motivated by the observation, we further
explore the potential to leverage geometric features of retriever
representation space to improve the perspective awareness of retrievers in a
zero-shot manner. We demonstrate the efficiency and effectiveness of our
projection-based methods on the same set of tasks. Further analysis also shows
how perspective awareness improves performance on various downstream tasks,
with 4.2
viewpoints on essay writing, compared to non-perspective-aware baselines.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要