Don't Waste Your Time: Early Stopping Cross-Validation
CoRR(2024)
Abstract
State-of-the-art automated machine learning systems for tabular data often
employ cross-validation; ensuring that measured performances generalize to
unseen data, or that subsequent ensembling does not overfit. However, using
k-fold cross-validation instead of holdout validation drastically increases the
computational cost of validating a single configuration. While ensuring better
generalization and, by extension, better performance, the additional cost is
often prohibitive for effective model selection within a time budget. We aim to
make model selection with cross-validation more effective. Therefore, we study
early stopping the process of cross-validation during model selection. We
investigate the impact of early stopping on random search for two algorithms,
MLP and random forest, across 36 classification datasets. We further analyze
the impact of the number of folds by considering 3-, 5-, and 10-folds. In
addition, we investigate the impact of early stopping with Bayesian
optimization instead of random search and also repeated cross-validation. Our
exploratory study shows that even a simple-to-understand and easy-to-implement
method consistently allows model selection to converge faster; in 94
datasets, on average by 214
model selection to explore the search space more exhaustively by considering
+167
overall performance.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined