Defeating insider attacks via autonomic self-protective networks

There has been a constant growing security concern with insider attacks on network accessible computer systems. Users with power credentials can do almost anything they want with the systems they own with very little control or oversight. Most breaches occurring nowadays by power users are considered legitimate access and not necessarily intrusions. Developing a solution for such problems is challenging because power users need flexible requirements to administer or maintain their systems. The increased usage of virtual environments, virtual systems, teleworking, and remote usage has made network access the preferred method for system administration. This dissertation describes the design and implementation of a network Autonomic Violation Prevention System (AVPS) framework that is intended to defeat the insider threat in organizations. The AVPS sits between privileged users and applications. It monitors traffic that traverses the network and takes actions as needed. A proof of concept prototype for the system was developed in a virtualized environment. FTP and Telnet were part of the application testbed. Rules that pertain to privileged user administration were applied. Actions that were tested successfully included traffic monitoring, replacement, blocking, and dropping. This work also examined the scalability of the AVPS design. An experimental testbed was built to obtain performance measures of the AVPS overhead, throughput, and response time. FTP, Database and Web servers were used in the application testbed. A variety of tests were performed including automated simultaneous transactions and manual simultaneous transactions. An M/M/N//M analytic queuing model was used to assess how well the AVPS system would perform for a finite population where the number of applications, users and AVPS engines vary under different load levels. The results showed that the AVPS exhibits a very low overhead and is therefore scalable. The AVPS architecture design was further enhanced to automate how signatures are created. Autonomic self-protection capabilities were added into the framework by implementing high level rules that set the goal for how violations are detected and signatures are created. Supervised self-learning capabilities were added via the use of Support Vector Machines (SVM) in order to classify the raw data and make final decisions on what is considered a violation and what is considered normal insider behavior.