A Link between Coding Theory and Cross-Validation with Applications
CoRR(2021)
Abstract
How many different binary classification problems a single learning algorithm
can solve on a fixed data with exactly zero or at most a given number of
cross-validation errors? While the number in the former case is known to be
limited by the no-free-lunch theorem, we show that the exact answers are given
by the theory of error detecting codes. As a case study, we focus on the AUC
performance measure and leave-pair-out cross-validation (LPOCV), in which every
possible pair of data with different class labels is held out at a time. We
shown that the maximal number of classification problems with fixed class
proportion, for which a learning algorithm can achieve zero LPOCV error, equals
the maximal number of code words in a constant weight code (CWC), with certain
technical properties. We then generalize CWCs by introducing light CWCs and
prove an analogous result for nonzero LPOCV errors and light CWCs. Moreover, we
prove both upper and lower bounds on the maximal numbers of code words in light
CWCs. Finally, as an immediate practical application, we develop new LPOCV
based randomization tests for learning algorithms that generalize the classical
Wilcoxon-Mann-Whitney U test.
MoreTranslated text
Key words
coding theory,applications,cross-validation
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined