A Comparison of Two-Sample Tests of Significance When Used With Variable Treatment Effects

This article compares the performance of many two-sample tests of significance that might be used to test the equality of means when the effect of the treatment is variable. Of the 19 tests that were compared, the normal scores test is recommended for general use in testing the null hypothesis of no treatment effect against the alternative that the distributions are stochastically ordered when the ratio of the larger standard deviation to the smaller standard deviation does not exceed 1.3. The Baumgartner-Wei ss-Schindler tests and an adaptive test also have higher power than the pooled t-test, the unequal variance t-test, and the rank-sum test for many distributions. In the simulation studies, data in the first sample are generated from nine distributions, including long-tailed and skewed distributions. Data in the second sample are generated by adding a random treatment effect to a random variable that was generated from the same distribution that was used in the first sample. Because we restricted our power studies to treatment effects that are positive or zero, the population distributions will be stochastically ordered. The results of these studies demonstrate that the normal scores test is often more powerful than the t-tests and the rank-sum test. If the ratio of the standard deviations does exceed 1.3, then one of the t-tests is recommended.