Null Hypothesis Significance Testing

Suppose that we have some hypothesis (i.e., proposed idea) about a random variable, and we want to explore the plausibility of our hypothesis using a sample of data collected from the population. For example, suppose that we believe that the random variable’s mean μ is equal to some hypothesized value, say μ0. Given a sample of n independent observations x1, . . . , xn from the population, we could estimate the population mean μ using the sample mean x̄ = 1 n ∑n i=1 xi. Assuming that our random variable of interest is continuous, the probability that x̄ is exactly equal to μ0 will be zero, i.e., P (x̄ = μ0) = 0. In other words, given any random sample of data from the population, we would expect the sample estimate x̄ to differ from the hypothesized population mean μ0 to some degree. But how small of a difference is “small enough” such that we should assume that the hypothesis μ = μ0 is reasonable? And how large of a difference is “too large” such that we should assume that the hypothesis μ = μ0 is unreasonable? Statistical tests, also known as “significance tests” or “null hypothesis significance tests” (NHST), attempt to answer these sorts of questions. The commonly used procedure for NHST was first developed by Sir Ronald A. Fisher (1925) and further developed by Neyman and Pearson (1933). As a result, if the filed of statistics, you may hear the idea of NHST referred to as the Neyman-Pearson procedure for testing the significance of a hypothesis. The idea of NHST has caused quite a bit of controversy in the field of psychology (e.g., Nickerson, 2000) and has lead to many heated arguments. As will be discussed towards the end of this chapter, this controversy is mainly due to the traditions of misunderstanding and misusing (and sometimes outright abusing) the ideas of NHST in the field of psychology.