A Framework of Petroleum Information Retrieval System Based on Web Scraping with Python

It is very necessary to build a customized retrieval system in the era of the big information explosion. This paper gives a framework of petroleum information retrieval system which will be used by petroleum exploration and development researchers. First, we use the open source framework SCRAPY to build a crawler system to crawl the information that business people pay attention to. Then k-means algorithm is used to cluster the crawled documents, therefore the key information is extracted and presented in the system. The actual effect in production and operation shows that this customized retrieval system is efficient and agile, it improves the efficiency, accuracy and automation level of the work.