Study on Focused Crawler's Application in Searching Petroleum News

Abstract

The paper aims for the application of focused crawler in the petroleum news topic crawling, studies the related technologies of the focused crawler, and put forward a crawling engine strategy and review strategy on the petroleum news topics, adopt different extracting methods for different types of pages through web page classification, and design a corresponding link topic correlation calculating method for the crawling engine strategy; test and verify the above-mentioned crawling engine strategies through experiments, and the experimental results show that the strategy can greatly balance its accuracy and width for focused crawler.