Deep reinforcement learning for competitive DER pricing problem of virtual power plants

The pricing competition between virtual power plants (VPPs) for distributed energy resources (DERs) is considered in this paper. Due to the limited amount of DERs in one distributed area, VPPs have to compete for the rights to work with DERs and then sell the electricity from internal DERs in the wholesale market. To address this pricing problem, a Markov decision process (MDP) with continuous state and action spaces is formulated for the VPP to consider the future rewards brought by the contract statuses of DERs. The deep deterministic policy gradient (DDPG) algorithm is applied to solve the pricing problem in the MDP form. To deal with the non-stationary environment in the training process brought by the competing VPP, a fictitious adversary method is put forward in this paper to combine with the DDPG algorithm for the first time. The proposed fictitious adversary method can help the VPP in finding competitive and robust pricing strategies under competition. Numerical results demonstrate the effectiveness of the proposed methodology in finding satisfying pricing strategies that consider the competitor's behavior and long-term values of DERs.