Fake News Detection and Manipulation Reasoning via Large Vision-Language Models
arxiv(2024)
Abstract
Fake news becomes a growing threat to information security and public opinion
with the rapid sprawl of media manipulation. Therefore, fake news detection
attracts widespread attention from academic community. Traditional fake news
detection models demonstrate remarkable performance on authenticity binary
classification but their ability to reason detailed faked traces based on the
news content remains under-explored. Furthermore, due to the lack of external
knowledge, the performance of existing methods on fact-related news is
questionable, leaving their practical implementation unclear. In this paper, we
propose a new multi-media research topic, namely manipulation reasoning.
Manipulation reasoning aims to reason manipulations based on news content. To
support the research, we introduce a benchmark for fake news detection and
manipulation reasoning, referred to as Human-centric and Fact-related Fake News
(HFFN). The benchmark highlights the centrality of human and the high factual
relevance, with detailed manual annotations. HFFN encompasses four realistic
domains with fake news samples generated through three manipulation approaches.
Moreover, a Multi-modal news Detection and Reasoning langUage Model (M-DRUM) is
presented not only to judge on the authenticity of multi-modal news, but also
raise analytical reasoning about potential manipulations. On the feature
extraction level, a cross-attention mechanism is employed to extract
fine-grained fusion features from multi-modal inputs. On the reasoning level, a
large vision-language model (LVLM) serves as the backbone to facilitate
fact-related reasoning. A two-stage training framework is deployed to better
activate the capacity of identification and reasoning. Comprehensive
experiments demonstrate that our model outperforms state-of-the-art (SOTA) fake
news detection models and powerful LVLMs like GPT-4 and LLaVA.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined