Query Triggered Crawling Strategy: Build a Time Sensitive Vertical Search Engine

Wu, Yu; Shou, Lidan; Hu, Tianlei; Chen, Gang

doi:10.1109/cw.2008.35

Cited by 6 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These techniques do not use strict mathematical definitions and are tolerant to imprecision, uncertainty, and partial truth to achieve a solution. In the starting paragraph of each category, we will discuss common characteristics of that technique and their subsequent 109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,<...>…”

Section: Focused Web Crawler Using Soft Computing Techniquesmentioning

confidence: 99%

“…As in twitterEcho, 33 19 YouTube, 20 Twitter, 33 Wikipedia, 48 compass.com, 50 netriu. fr, 62 Virgilio.it, 63 Netease military channel, 64 dangdang.com, 69 163. com, 134 Chinese travel site, 136 chemicaljournal, 137 only Twitter can be the subject system, in a similar way for videos crawling YouTube is used as a subject system. 20 We have found 24 studies in total that belongs to this category as shown in Table 10.…”

Section: Subject Systemsmentioning

confidence: 99%

See 1 more Smart Citation

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

Section: Focused Web Crawler Using Soft Computing Techniquesmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

show abstract

“…The previous work on search personalization [13], [17], [18], [19], [20], [21] is usually characterized through the data source utilized to learn about user interest, and the approach where a user is relating to these data. The factors that are mostly considered for web personalization are "user queries", "weblog data" and "clickthrough data".…”

Section: Related Workmentioning

confidence: 99%

“…The most commonly used method is to use keyword-based search methods [10] to unearth the relevant web pages and to present suitable ranking schemes. In addition, as demand for user satisfaction has increased, "vertical search engines" have presented specific value information and interrelated services for specific areas, specific individuals, and specific needs (eg, travel searches, online purchases, and search for educational resources) [17]. However, vertical and general search engines still do not get detailed and accurate information.…”

Section: Related Workmentioning

confidence: 99%

A Web Search Personalization Based on Probability of Semantic Similarity between User Log and Query with Web Page

Raju¹,

Babu²,

Anuradha³

2018

IJET

View full text Add to dashboard Cite

Web search personalization is recognized as a competent solution to address the problem of query-relevant search as per the user interest, while it able to present dissimilar search results based upon the preferences and information requirements of users. The popular search engines provide their search results interpreting the user query only, which mostly have unrelated results due to the keywords ambiguity problem. In order to have satisfied and user interesting result, it is important to personalize the results according to their relevancies. In this paper, we propose a Web search Personalization based on a Probability of Semantic Similarity (WP-PSS) between user log and query with search result webpage. It performs a probability of semantic similarities computation between the user query and search result webpage snippet, and compute the frequency of link associated with the log data. Based on these two computed factors a probability of similarities association is computed to group and re-rank the search results for the personalization. Experiment evaluation over a set of multi-domain web searched data collection shows an accuracy improvisation.

show abstract

“…On the base of [8], [9], and [10] which focused on improving the crawling efficiency, [11] proposes a distributed crawling concept which is designed to avoid approximations, to limit the network overhead, and to run on relatively inexpensive hardware. [12] implements a TSVS (Time Sensitive Vertical Search Engine) prototype named Velocisaurus that focused on time-critical airfare discount information search to investigate the time critical requirements for vertical search and proposes a QTC (query triggered crawling) strategy to coordinate the crawling systems by real-time user queries.…”

Section: Literature Reviewmentioning

confidence: 99%

Design and Implementation of a Crawling System in Shopping Search Engine

Wang

2009

2009 Second International Workshop on Computer Science and Engineering

View full text Add to dashboard Cite

This paper presents a crawling system， that helps building a shopping search engine. The system's core is a vertical crawler that is specially used in shopping search engines. The crawler is divided into 5 parts and it realized automatic generation of crawling templates by its analysis module which is a regular expressions method set. A manual generation of crawling template is designed to support the automatic function, and these modified templates 100% passed the test. Five parts' perfectly combination makes the system being such a fool-proof system that anyone who has basic computer operation ability can use it.With the fast development of search engine, topical crawler that severs vertical search engines has been an important research direction. There are many literatures describing the crawler design algorithm and crawling strategies, however, due to the competitive nature of the shopping search engine business, there are few papers in the literature describing the design and implementation in shopping search engines' topical crawlers. This paper's main contribution is to fill that gap. The crawling system described in this paper is a prototype of crawler system in shopping search engine. It will influence the later shopping search engines' searching and development.

show abstract

Query Triggered Crawling Strategy: Build a Time Sensitive Vertical Search Engine

Cited by 6 publications

References 10 publications

A survey of Web crawlers for information retrieval

A survey of Web crawlers for information retrieval

A Web Search Personalization Based on Probability of Semantic Similarity between User Log and Query with Web Page

Design and Implementation of a Crawling System in Shopping Search Engine

Contact Info

Product

Resources

About