ExcluIR: Exclusionary Neural Information Retrieval
CoRR(2024)
Abstract
Exclusion is an important and universal linguistic skill that humans use to
express what they do not want. However, in information retrieval community,
there is little research on exclusionary retrieval, where users express what
they do not want in their queries. In this work, we investigate the scenario of
exclusionary retrieval in document retrieval for the first time. We present
ExcluIR, a set of resources for exclusionary retrieval, consisting of an
evaluation benchmark and a training set for helping retrieval models to
comprehend exclusionary queries. The evaluation benchmark includes 3,452
high-quality exclusionary queries, each of which has been manually annotated.
The training set contains 70,293 exclusionary queries, each paired with a
positive document and a negative document. We conduct detailed experiments and
analyses, obtaining three main observations: (1) Existing retrieval models with
different architectures struggle to effectively comprehend exclusionary
queries; (2) Although integrating our training data can improve the performance
of retrieval models on exclusionary retrieval, there still exists a gap
compared to human performance; (3) Generative retrieval models have a natural
advantage in handling exclusionary queries. To facilitate future research on
exclusionary retrieval, we share the benchmark and evaluation scripts on
.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined