Comparison of profile-likelihood-based confidence intervals with other rank-based methods for the two-sample problem in ordered categorical data

For ordered categorical data from randomized clinical trials, the relative effect, the probability that observations in one group tend to be larger, has been considered appropriate for a measure of an effect size. Although the Wilcoxon-Mann-Whitney test is widely used to compare two groups, the null hypothesis is not just the relative effect of 50%, but the identical distribution between groups. The null hypothesis of the Brunner-Munzel test, another rank-based method used for arbitrary types of data, is just the relative effect of 50%. In this study, we compared actual type I error rates (or 1 - coverage probability) of the profile-likelihood-based confidence intervals for the relative effect and other rank-based methods in simulation studies at the relative effect of 50%. The profile-likelihood method, as with the Brunner- Munzel test, does not require any assumptions on distributions. Actual type I error rates of the profile-likelihood method and the Brunner-Munzel test were close to the nominal level in large or medium samples, even under unequal distributions. Those of the Wilcoxon-Mann-Whitney test largely differed from the nominal level under unequal distributions, especially under unequal sample sizes. In small samples, the actual type I error rates of Brunner-Munzel test were slightly larger than the nominal level and those of the profile-likelihood method were even larger. We provide a paradoxical numerical example: only the Wilcoxon-Mann-Whitney test was significant under equal sample sizes, but by changing only the allocation ratio, it was not significant but the profile-likelihood method and the Brunner-Munzel test were significant. This phenomenon might reflect the nature of the Wilcoxon-Mann-Whitney test in the simulation study, that is, the actual type I error rates become over and under the nominal level depending on the allocation ratio.