Design and Implementation of Firmware Data Acquisition System Based on Scrapy Framework

In recent years, the scale of Internet data grows exponentially with the development of Internet technology. Such huge amount of Internet data is valuable. Web crawler is one of the most popular technology, which is often used to obtain these data. Scrapy is a popular framework of web crawler which is widely used in various Internet information collection systems. This paper optimizes the framework of Scrapy, designs a firmware data acquisition system based on the framework of Scrapy based on the technologies of distributed, anti-crawler, ELK and automatic construction, and crawls the firmware information of various manufacturers on the Internet. The experimental results show that after employing the distributed and prevent anti-crawler technology, the number of target acquisition increases by 10%, and the time is shortened by 70%. The large-scale logs analysis using ELK solves the problem that the log number is too large for analysis, and the crawler can be automatically constructed and crawls through the code automatic construction technology. This method is efficient for the optimization of firmware crawler.