Distributed Secure Surrounding Control for Multiple USVs against Deception Attacks: A Stackelberg Game Approach with Reinforcement Learning

This article investigates the Stackelberg game-based distributed secure surrounding control (SSC) problem for multiple unmanned surface vehicles (USVs) with unknown dynamics under deception attacks (DAs). The proposed scheme is rooted in the Stackelberg game with reinforcement learning (RL), in which the attacker and controller respectively play the roles of the follower and leader, making sequential decisions of the DA and the SSC. Specifically, a distributed target estimator is established to access the target position. By utilizing this estimated position to formulate an intermediate control law, the target surrounding scenario is effectively transformed into the Stackelberg game-solving problem. The RL approach with a neural networks-based actor-critic learning structure is deployed to directly derive the distributed optimal SSC and DA policies from the Bellman error, whilst learning the unknown dynamics of USVs. Moreover, a value function decomposition technique is applied optimally for using the designed control parameters, thereby accelerating the acquisition of optimal policies. A rigorous theoretical analysis is employed to ensure the closed-loop stability of multiple USVs. Simulations are provided to validate the effectiveness of the proposed distributed SSC scheme for multiple USVs.