Simulation Studies of Item Bias Estimation Accuracy

The item-parameter invariance assumption of Cognitive Diagnostic Models (CDMs) states that item parameters are not functionally dependent upon the characteristics of an examinee with respect to a specific examinee population. Violations of this assumption for a test item indicate the presence of item-bias for that item. The purpose of this study is to investigate how the evidence for item parameter invariance varies as a function of the number of times an item is administered to an examinee. Towards this end, three different data sets are used: (i) ECPE-Grammar (Templin and Hoffman, Educational Measurement: Issues and Practice, 32(2):37–50, 2013) with n=2922, (ii) TIMSS (Mullis et al. TIMSS 2011 International Results in Mathematics. International Association for the Evaluation of Educational Achievement, 2012) with n = 1010 and (iii) Social Psychology class conducted in University of Texas at Dallas (UTD) (Social-UTD) n = 136. To detect the presence of item parameter invariance, Bootstrap Mean Absolute Bias (Bootstrap-AB) metric for each item parameter is calculated by comparing two examinee subpopulations generated by sampling with replacement multiple times from the original examinee population. This process is repeated over pairs of examinee population samples of varying sizes to get different Item Administered Count (IAC) values. Item-specific guess and slip parameters are estimated using the DINA model in the CDM package in R (George et al. Journal of Statistical Software 74(2), 2016). Violations of item invariance are observed for smaller IAC, but such violations decrease as the IAC increases. The standard parameter DINA parameter estimation method was then compared with a bagging approach for estimating DINA model parameters (Breiman, Machine Learning, 24(2):123–140, 1996). The fitted bagged-DINA model demonstrated fewer violations of item invariance for smaller IAC values for all three data sets.