An Alternative Curriculum Learning Approach with Macro Actions for Deep Reinforcement Learning

It is challenging to train deep reinforcement learning (DRL) agents for tasks with sparse rewards, especially for those that require combining discrete actions with flexible control, e.g., the scoring task for robots in RoboCup soccer. Curriculum learning has been considered as a useful tool for DRL to accelerate the learning process by splitting the task into a sequence of progressively more difficult subtasks. In this paper, we propose an alternative curriculum learning approach that firstly makes the task easier by introducing macro actions, i.e., predefined procedures of actions for solving several subtasks. Once a DRL agent has been well trained for the easier task, the approach will disable one of the macro actions and continuously train the DRL agent for the refined task. The approach continues the process until all macro actions were disabled and a DRL agent were trained for the original task. Experiments on the Half Field Offense (HFO) platform for simulated RoboCup soccer show that the approach is effective and able to learn robust policies for challenging tasks, like HFO 2v2, which has not been reported to be solved by DRL as far as the knowledge of the authors. A demonstration video is available online at https://youtu.be/e6kunbAhhaY.