What Can We Do About Our (Untrustworthy) Literature?

The primary focus of the replication movement has been on changing research and analysis practices to enhance the reliability of future research. Much less discussed is what should be done with existing scientific literatures, which are increasingly regarded with suspicion but cannot feasibly be rechecked in their entirety. The problem is vexing because criteria once seen as sufficient markers of validity (such as multiple published “conceptual replications”) are now recognized as wholly unconvincing. We suggest that the discipline needs to adopt a strong truth-in-labeling standard, admitting that all findings not yet confirmed in well-powered preregistered studies need to be regarded as preliminary and merely suggestive. Adopting this conservative standard will help alleviate the catastrophic drop in the public reputation of behavioral science that has begun in recent years, and will focus incentives and resources on finding out what portion of our supposed knowledge base is real and what is not.