Improving the performance of focused crawlers based on Multi-Agent system

Using Multi-Agent theory to achieve communication and collaboration between focused crawlers is a new topic, and it has been proved that it can improve the precision significantly. In this paper, we improved the focused crawler based on Multi-Agent System, and proposed a new ability function for evaluating whether an agent needs to request for assistance. This ability function not only took the semantic importance of downloaded pages into account, but also the scores of links from downloaded pages. Finally, we have achieved the improved focused crawler based on JADE, and the experimental results showed that our focused crawler based on Multi-Agent had higher precision and crawling efficiency. Copyright © 2011 Binary Information Press.