Design and Implementation of a Crawling System in Shopping Search Engine

Abstract

This paper presents a crawling system， that helps building a shopping search engine. The system's core is a vertical crawler that is specially used in shopping search engines. The crawler is divided into 5 parts and it realized automatic generation of crawling templates by its analysis module which is a regular expressions method set. A manual generation of crawling template is designed to support the automatic function, and these modified templates 100% passed the test. Five parts' perfectly combination makes the system being such a fool-proof system that anyone who has basic computer operation ability can use it.With the fast development of search engine, topical crawler that severs vertical search engines has been an important research direction. There are many literatures describing the crawler design algorithm and crawling strategies, however, due to the competitive nature of the shopping search engine business, there are few papers in the literature describing the design and implementation in shopping search engines' topical crawlers. This paper's main contribution is to fill that gap. The crawling system described in this paper is a prototype of crawler system in shopping search engine. It will influence the later shopping search engines' searching and development.

A survey of Web crawlers for information retrieval

“…These techniques do not use strict mathematical definitions and are tolerant to imprecision, uncertainty, and partial truth to achieve a solution. In the starting paragraph of each category, we will discuss common characteristics of that technique and their subsequent 109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,<...>…”

Studies and evaluation on meta search engines

“…Liu, etc put forward the policy of making use of decision tree technology for classification to dynamically generate the tactics according to the recorded performance of member search engines [2]. Wang, etc put forward an improved genetic algorithm to dynamically change the sequence of members according to their dynamic performance [3]. Du, etc put forward an intelligent scheduling strategy based on statistical study and information classification [4].…”