CAAC: An effective reinforcement learning algorithm for sparse reward in automatic control systems

Nowadays, reinforcement learning (RL) is increasingly being employed to optimize automatic control systems. This allows these systems to autonomously learn and improve their operations, enhancing their ability to adapt to changing environments and conditions. However, RL heavily relies on reward signals to guide learning, and in practical tasks, these signals are often sparse. This sparsity hinders the learning procedure as only a small amount of feedback signals are available. Most of the current methods to solve the sparse reward problem need to introduce a lot of hyperparameters, and the sample data utilization is relatively low. To address the issue of sparse rewards in RL-based automatic control systems, we propose the Cosine Attenuation Monte Carlo Augmented Actor-Critic (CAAC) algorithm. CAAC uses a cosine decay function to adjust the Q-value during training, optimizing the effect of final rewards and improving RL algorithm performance in the area of automatic control systems. In addition, we conduct experiments in three simulation environments to validate the proposed approach. The results demonstrate that CAAC outperforms the baseline algorithms in terms of learning speed and obtains 10