“Gee, P = .06; Let’s Run Two More Participants!”: Wicked Effects of Motivated Conditional Data Collection on False Positives in Null-Hypothesis Significance Testing

It has been shown in simulation studies that the probability of a false positive research result increases rapidly when additional data is repeatedly collected and tested for statistical significance. I extend this scenario, also known as optional or conditional stopping, by a less extreme but more realistic scenario: Researchers are likely motivated to run additional participants when the initially observed p-value is close to the desired significance level. Results from two Monte-Carlo simulations show the consequences of such behavior: The probability of a false positive result increases up to 40% for studies in which researchers run additional participants and the α-error of a study can increase by 45%. Potential false beliefs that rationalize this behavior are identified and discussed.