A path selection scheme for detecting malicious behavior based on deep reinforcement learning in SDN/NFV-Enabled network

The SDN/NFV network is prone to different types of attacks. The Distributed Denial of Service (DDoS) attack has the most severe impact as it can overwhelm the critical components of SDN/NFV to degrade its performance. We propose a closed-loop security architecture (SFCSA) and virtualize detection methods as network service functions in this article. Combining the detection methods forms detection paths, in which different detection paths affect security performance differently. Further, we model the path selection problem as a Markov Decision Process, where the reward balances the malicious traffic detection capability and end-to-end latency. Then, an integrated deep reinforcement learning and convolution neural network path selection algorithm (CNNQ) is proposed. Furthermore, we define a total path malicious traffic detection capability metric. The defined metrics and common metrics are applied to evaluate the building prototype, with the corresponding experimental results demonstrating that the detection performance when combining multiple detection modules outperforms a single detection-based module. Besides, we verify the effectiveness of the CNNQ method under various DDoS attacks scenarios and present the fine-grained classification results of the selected detection modules.