Asymptotic distribution of likelihood ratio test statistics for variance components in nonlinear mixed effects models.

Mixed effects models are widely used to describe heterogeneity in a population. A crucial issue when adjusting such a model to data consists in identifying fixed and random effects. Testing the nullity of the variances of a subset of random effects can help to investigate this issue. Some authors have proposed to use the likelihood ratio test and have established its asymptotic distribution in some particular cases. Extending the existing results, a likelihood ratio test procedure is studied, to test that the variances of any subset of the random effects are equal to zero in nonlinear mixed effects model. More precisely, the asymptotic distribution of the test statistics is shown to be a chi-bar-square distribution, that is to say a mixture of chi-square distributions, and the corresponding weights are identified. In particular, it is highlighted that the limiting distribution depends strongly on the presence of correlations between the random effects. The finite sample size properties of the test procedure are illustrated through simulation studies and the test procedure is applied to two real datasets of dental growth and of coucal growth.