Filtering the Web Pages that are not Modified at Remote Site Without Downloading using Mobile Crawlers

Bal, Satinder; Nath, Rajender

doi:10.3923/itj.2010.376.380

Cited by 5 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are currently over 1,500 active crawlers [8]. By some early estimates, crawlers consume up to 40% of the Internet's text web bandwidth [9]. At an abstract level, the web crawling algorithm is straightforward; given a set of seed URLs a crawler places them in the Frontier structure, from which it iteratively selects URLs, downloads them, extracts the hyperlinks contained in them and adds the new URLs to the Frontier.…”

Section: Web Crawlingmentioning

confidence: 99%

Set Cover at Web Scale

Stergiou

Tsioutsiouliklis

2015

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

The classic Set Cover problem requires selecting a minimum size subset A ⊆ F from a family of finite subsets F of U such that the elements covered by A are the ones covered by F . It naturally occurs in many settings in web search, web mining and web advertising. The greedy algorithm that iteratively selects a set in F that covers the most uncovered elements, yields an optimum (1 + ln |U |)-approximation but is inherently sequential. In this work we give the first MapReduce Set Cover algorithm that scales to problem sizes of ∼ 1 trillion elements and runs in log p ∆ iterations for a nearly optimum approximation ratio of p ln ∆, where ∆ is the cardinality of the largest set in F .A web crawler is a system for bulk downloading of web pages. Given a set of seed URLs, the crawler downloads and extracts the hyperlinks embedded in them and schedules the crawling of the pages addressed by those hyperlinks for a subsequent iteration. While the average page out-degree is ∼ 50, the crawled corpus grows at a much smaller rate, implying a significant outlink overlap. Using our MapReduce Set Cover heuristic as a building block, we present the first large-scale seed generation algorithm that scales to ∼ 20 billion nodes and discovers new pages at a rate ∼ 4x faster than that obtained by prior art heuristics.

show abstract

Section: Web Crawlingmentioning

confidence: 99%

Set Cover at Web Scale

Stergiou

Tsioutsiouliklis

2015

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

show abstract

“…Search engines do not index sites equally, may not index new pages for months, and no engine indexes more than about 16% of the web [23]. A crawling technique to reduce the load of the network using mobile agents were developed by Bal and Nath [25]. The working of a comprehensive full text search engine called WebCrawler is also studied [27].…”

Section: Background Literaturementioning

confidence: 99%

Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers

Jose¹,

Lal²

2013

View full text Add to dashboard Cite

World Wide Web is growing at a tremendous rate in terms of the number of visitors and number of web pages. Search engine crawlers are highly automated programs that periodically visit the web and index web pages. The behavior of search engines could be used in analyzing server load, quality of search engines, dynamics of search engine crawlers, ethics of search engines etc. The more the number of visits of a crawler to a web site, the more it contributes to the workload. The time delay between two consecutive visits of a crawler determines the dynamicity of the crawlers. The ARIMA(1,1,0) Model in time series analysis works well with the forecasting of the time delay between the visits of search crawlers at web sites. We considered 5 search engine crawlers, all of which could be modeled using ARIMA(1,1,0).The results of this study is useful in analyzing the server load.

show abstract

“…In Mobile crawling idea, place the mobile crawler at the web server. That small web crawler resides on web server, perform their operations on web server and send updates in return [6].…”

Section: Related Workmentioning

confidence: 99%

Smart Approach to Reduce the Web Crawling Traffic of Existing System using HTML based Update File at Web Server

Mishra¹,

Jain²,

Sachan³

2010

IJCA

View full text Add to dashboard Cite

Web crawler is used for downloading information from web. Web pages are changed without any notice. Web crawler frequently revisits websites to check updates. It is expected that 40% of present internet traffic is because of web crawling. In this paper we propose a file which maintains the list of updated URLs of web pages of web site. Format of file is based on HTML. Crawler will only visit the UPDATE File, and need not have to revisit the full website to know the updates. This scheme can easily implement on today's system with little modification on web application and web crawler. In simulator we test proposed method; using a website of 13 pages for experiment. Experiment results shows that this scheme is very promising.

show abstract

Filtering the Web Pages that are not Modified at Remote Site Without Downloading using Mobile Crawlers

Cited by 5 publications

References 1 publication

Set Cover at Web Scale

Set Cover at Web Scale

Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers

Smart Approach to Reduce the Web Crawling Traffic of Existing System using HTML based Update File at Web Server

Contact Info

Product

Resources

About