How Smart are Online Workers? A Student Versus MTurk Participant Comparison

The use of online workers as research participants has dramatically escalated in recent years, prompting interest in how online workers compare to traditional research participants (e.g., Paolacci and Chandler 2014; Farrell, Grenier, and Leiby 2016). To date, no study has compared the general intelligence level of online workers to traditional research participants. We broadly investigate fluid intelligence (McGrew 2009) using two easy-to-administer, psychometrically validated cognitive ability measures: the Cognitive Reflection Test (CRT) and Raven’s Progressive Matrices (RPM). We also compare participant performance on the Luft and Shields (2001) forecasting task. Our main findings are (1) low-cost Amazon Mechanical Turk (MTurk) workers and traditional student subjects have comparable fluid intelligence and (2) MTurk workers can outperform traditional student subjects under certain conditions. In spite of our main findings, we caution researchers about, and provide guidance to mitigate, a “fat tail” of extremely poor performers in the MTurk participant pool.