A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions
arxiv(2024)
摘要
Peer review often involves reviewers submitting their independent reviews,
followed by a discussion among reviewers of each paper. A question among
policymakers is whether the reviewers of a paper should be anonymous to each
other during the discussion. We shed light on this by conducting a randomized
controlled trial at the UAI 2022 conference. We randomly split the reviewers
and papers into two conditions–one with anonymous discussions and the other
with non-anonymous discussions, and conduct an anonymous survey of all
reviewers, to address the following questions: 1. Do reviewers discuss more in
one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2.
Does seniority have more influence on final decisions when non-anonymous? Yes,
the decisions are closer to senior reviewers' scores in the non-anonymous
condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite
in one of the conditions? No significant difference in politeness of reviewers'
text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported
experiences differ across the two conditions? No significant difference for
each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer
one condition over the other? Yes, there is a weak preference for anonymous
discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider
important to make policy on anonymity among reviewers? Reviewers' feeling of
safety in expressing their opinions was rated most important, while polite
communication among reviewers was rated least important (n = 159). 7. Have
reviewers experienced dishonest behavior due to non-anonymity in discussions?
Yes, roughly 7
experiment reveals evidence supporting an anonymous discussion setup in the
peer-review process, in terms of the evaluation criteria considered.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要