Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

The method of pairwise comparison has been used in a wide range of contexts. In educational measurement, many pairwise comparisons are required for reliable measurement when the comparisons to be performed are selected using the commonly-used semi-random selection algorithm (SSA). We proposed a Bayesian selection algorithm (BSA) to obtain smaller standard errors of parameter estimates and higher reliability compared with the SSA, and we evaluated the performance of these algorithms in a simulation study. We conclude that 1) the BSA should be preferred to the SSA, 2) the number of comparisons required for reliable measurement depends on the object variance, and 3) the Scale Separation Reliability (SSR) may systematically overestimate reliability even when the SSA is used.