Hyperparameter Selection in Reinforcement Learning Using the “Design of Experiments” Method

Artificial Intelligence and Machine Learning is a highly active area of research across numerous subgenres. One such example is Reinforcement Learning, which relies on trial and error based sampling of the environment to train an agent at completing a given task, commonly applied to Atari 2600 games within research applications. Many variants of Reinforcement Learning algorithms exist, all of which apply numerous hyperparameters to control the learning process in some way, from strength of backpropagation updates through to rates of exploration. The breadth of choice across these hyperparameters makes optimal training a challenging task, with no feedback given until significant time has lapsed in training. What is of interest here is the relative importance across these hyperparameters as well as any relationship amongst them at deriving high performance agents. Through this research we apply the common statistical approach of Design of Experiments to the task of understanding the state space of the numerous hyperparameters present in Reinforcement Learning algorithms such as the Double Deep Q-Network and Prioritized Experience Replay methods. We identify the learning rate as the only primary contributor of success or failure of value based Reinforcement Learning approaches to achieve optimal reward gain. This finding suggests the possibility of significantly reduced effort and time for considering the effects of non-dependent hyperparameters.