What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification
Conference of the European Chapter of the Association for Computational Linguistics(2024)
摘要
Biomedical claim verification fails if no evidence can be discovered. In
these cases, the fact-checking verdict remains unknown and the claim is
unverifiable. To improve upon this, we have to understand if there are any
claim properties that impact its verifiability. In this work we assume that
entities and relations define the core variables in a biomedical claim's
anatomy and analyze if their properties help us to differentiate verifiable
from unverifiable claims. In a study with trained annotation experts we prompt
them to find evidence for biomedical claims, and observe how they refine search
queries for their evidence search. This leads to the first corpus for
scientific fact verification annotated with subject-relation-object triplets,
evidence documents, and fact-checking verdicts (the BEAR-Fact corpus). We find
(1) that discovering evidence for negated claims (e.g., X-does-not-cause-Y) is
particularly challenging. Further, we see that annotators process queries
mostly by adding constraints to the search and by normalizing entities to
canonical names. (2) We compare our in-house annotations with a small
crowdsourcing setting where we employ medical experts and laypeople. We find
that domain expertise does not have a substantial effect on the reliability of
annotations. Finally, (3), we demonstrate that it is possible to reliably
estimate the success of evidence retrieval purely from the claim text (.82),
whereas identifying unverifiable claims proves more challenging (.27). The
dataset is available at http://www.ims.uni-stuttgart.de/data/bioclaim.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要