Scheduling of twin automated stacking cranes based on Deep Reinforcement Learning

Effective scheduling of twin automated stacking cranes (ASCs) in automated storage yard is critical to maximize operational efficiency. While Deep Reinforcement Learning (DRL) is promising in solving NP-hard scheduling problems, twin ASCs scheduling is challenging due to its unique properties including sequence-dependent setup and potential ASC interferences. In this paper, we propose a novel DRL method to learn high-quality policy for scheduling twin ASCs. We propose a Markov Decision Process model that enables the DRL agent to learn to minimize makespan and possible interferences. Based on the problem characteristics, we design a self-attention based neural architecture to effectively capture the relationships between containers under certain block state. Experiments show that the agent with the proposed feature extraction network can learn high-quality policies from training instances. These learned policies can be employed to produce effective scheduling solutions within seconds. Compared to traditional scheduling methods, the learned policy performs best in most problem sizes, and the performance improvement amplifies as the scales increase. Moreover, the policies show remarkable generalization ability on unseen instances with different distributions or scales.