Multi-Model Ensemble Optimization

Methodology and optimization algorithms for sparse regression are extended to multi-model regression ensembles. In particular, we adapt optimization algorithms for l0-penalized problems to learn ensembles of sparse and diverse models. To generate an initial solution for our algorithm, we generalize forward stepwise regression to multi-model regression ensembles. The sparse and diverse models are learned jointly from the data and constitute alternative explanations for the relationship between the predictors and the response variable. Beyond the advantage of interpretability, in prediction tasks the ensembles are shown to outperform state-of-the-art competitors on both simulated and gene expression data. We study the effect of the number of models and show how the ensembles achieve excellent prediction accuracy by exploiting the accuracy-diversity tradeoff of ensembles. The optimization algorithms are implemented in publicly available R/C++ software packages.