Evaluating two small-sample corrections for fixed-effects standard errors and inferences in multilevel models with heteroscedastic, unbalanced, clustered data

Multilevel modeling (MLM) is commonly used in psychological research to model clustered data. However, data in applied research usually violate one of the essential assumptions of MLM—homogeneity of variance. While the fixed-effect estimates produced by the maximum likelihood method remain unbiased, the standard errors for the fixed effects are misestimated, resulting in inaccurate inferences and inflated or deflated type I error rates. To correct the bias in fixed effects standard errors and provide valid inferences, small-sample corrections such as the Kenward-Roger (KR) adjustment and the adjusted cluster-robust standard errors (CR-SEs) with the Satterthwaite approximation for t tests have been used. The current study compares KR with random slope (RS) models and the adjusted CR-SEs with ordinary least squares (OLS), random intercept (RI) and RS models to analyze small, heteroscedastic, clustered data using a Monte Carlo simulation. Results show the KR procedure with RS models has large biases and inflated type I error rates for between-cluster effects in the presence of level 2 heteroscedasticity. In contrast, the adjusted CR-SEs generally yield results with acceptable biases and maintain type I error rates close to the nominal level for all examined models. Thus, when the interest is only in within-cluster effect, any model with the adjusted CR-SEs could be used. However, when the interest is to make accurate inferences of the between-cluster effect, researchers should use the adjusted CR-SEs with RS to have higher power and guard against unmodeled heterogeneity. We reanalyzed an example in Snijders & Bosker (2012) to demonstrate the use of the adjusted CR-SEs with different models.