Likelihood-Free Hypothesis Testing

Consider the problem of binary hypothesis testing. Given Z coming from either P ^⊗m or Q ^⊗m , to decide between the two with small probability of error it is sufficient, and in many cases necessary, to have m ≍ 1/ε ² , where ε measures the separation between P and Q in total variation (TV). Achieving this, however, requires complete knowledge of the distributions and can be done, for example, using the Neyman-Pearson test. In this paper we consider a variation of the problem which we call likelihood-free hypothesis testing, where access to P and Q is given through n i.i.d. observations from each. In the case when P and Q are assumed to belong to a non-parametric family, we demonstrate the existence of a fundamental trade-off between n and m given by nm ≍ n ² _GoF (ε), where n _GoF (ε) is the minimax sample complexity of testing between the hypotheses H ₀ : P = Q vs H ₁ : TV(P, Q) ≥ ε. We show this for three families of distributions, in addition to the family of all discrete distributions for which we obtain a more complicated trade-off exhibiting an additional phase-transition. Our results demonstrate the possibility of testing without fully estimating P and Q, provided m ≫ 1/ε ² .