Classification Of Diversified Web Crawler Accesses Inspired By Biological Adaptation

To discover and prevent attacks, it is necessary to collect data about the attacks using honeypots and to identify malicious accesses from collected data. In this study, we focus on detecting a massive number of crawler accesses, which complicates the detection of malicious accesses. We adapt AntTree, a bio-inspired clustering scheme that is highly scalable and adaptable, for crawler detection. We also designed a feature vector for crawler detection and propose a cluster interpretation method of AntTree. Our results show that the proposed bio-inspired mechanism can detect crawlers with a low false-negative rate, which is an advantage over conventional schemes for detecting various types of crawler.