Deep reinforcement learning-based framework for constrained any-objective optimization

Optimization problems are widely used in many real-world applications. These problems are rarely unconstrained and are usually considered constrained optimization problems. Regarding the number of objectives, the optimization problems can be categorized into single- (for one), multi- (usually for two and three), and many- (more than three) objective optimization problems. In this paper, an Any-Objective Optimization (AOO) framework is introduced based on Deep Reinforcement Learning (DRL) models. The term any-objective optimization is coined to indicate the generalized structure of the proposed algorithm that regardless of the number of objectives, can solve the constrained optimization problems with any number of objectives. To trade off the multiple conflicting objectives, RL algorithms can be extended to a framework called Multi-Objective Reinforcement Learning (MORL). By converting a constrained optimization problem into an environment that can be explored by the MORL and deep learning algorithms, any constrained optimization problem can be tackled. In this research, to solve a constrained optimization problem with any number of objective functions, a novel reward function is introduced, and the algorithm begins a heuristic search in the environment to find the optimal solution(s) and generates an archive of the optimal Pareto front solution. The corresponding environment is constructed modular, such that any RL algorithm with arbitrary reward function types (scalar or vector) can be utilized. To evaluate the proposed algorithm, some popular test function-defined constrained optimization problems with continuous variable and objective spaces as illustrative examples are considered, and five of the widely used DRL algorithms are implemented to test the case studies. To demonstrate the capabilities of the proposed algorithm, the obtained results are compared with structurally similar GA-based well-known existing single-, multi-, and many-objective optimization algorithms, respectively. The results show that the proposed framework can be a well-performing baseline for a new type of DRL-based optimization algorithm.