How Certain are You of Your Minimum AIC or BIC Values?

In choosing a candidate model in likelihood-based inference by minimizing an information criterion, the practitioner is often faced with the difficult task of deciding how far up the ranked list to look. Motivated by this pragmatic necessity, we derive an approximation to the quantiles of a generalized (model selection) information criterion (ZIC), defined as a criterion for which the limit in probability is identical to that of the normalized log-likelihood, and which includes common special cases such as AIC and BIC. The method starts from the joint asymptotic normality of the ZIC values, and proceeds by deriving the (asymptotically) exact distribution of the minimum, which can be efficiently (numerically) computed. High quantiles can then be obtained by inverting this distribution function, resulting in what we call a certainty envelope (CE) of plausible models, intended to provide a heuristic upper bound on the location of the actual minimum. The theory is established for three data settings of perennial classical interest: (i) independent and identically distributed, (ii) regression, and (iii) time series. The development in the latter two cases invokes Lindeberg-Feller type conditions for, respectively, normalized: sums of conditional distributions and quadratic forms, in the observations. The performance of the methodology is examined on simulated data by assessing CE nominal coverage probabilities, and comparing them to the bootstrap. Both approaches give coverages close to nominal for large samples, but the bootstrap is on average two orders of magnitude slower. Finally, we hint at the possibility of producing confidence intervals for individual parameters by pivoting the distribution of the minimum ZIC, thus naturally accounting for post-model selection uncertainty.