Comparative Performance Evaluation of State-of-the-Art Hyperparameter Optimization Frameworks

Machine learning (ML) has proven to be highly effective in solving complex problems in various domains, thanks to its ability to identify specific data tasks, perform feature engineering, and learn quickly. However, designing and training ML models is a complicated task and requires optimization. The effectiveness of ML models is highly dependent on the selection of hyperparameters that determines their performance. Hyperparameter optimization (HPO) is a systematic search process to find the optimal combinations of hyperparameters to achieve robust performance. Traditional HPO methods such as grid and random search take a lot of computing time when used in large-scale applications. Recently, various automated search strategies, such as Bayesian optimization (BO) and evolutionary algorithms, have been developed to significantly reduce the computing time. In this paper, we use state-of-the-art HPO frameworks, namely BO, Optuna, HyperOpt, and Keras Tuner, for optimizing the ML and deep learning models for the classification tasks and evaluate their comparative performance using two different sets of experiments. The first one uses different ML classifiers to solve the optimal parameter selection problem with HPO. The second one attempts to optimize the convolutional neural network (CNN) architecture using HPO frameworks to improve its performance in the image classification task. We use four publicly available real-world datasets including one image dataset. The experimental results show that HyperOpt - TPE outperforms the other HPO frameworks for the ML classifiers and achieves up to 94.12% of accuracy with 30 minutes for performing the optimization. Similarly, for the CNN model, HyperOpt-TPE outperforms the other HPO frameworks by improving 34% of the classification accuracy, while taking 2 hours and 24 minutes of computing time.