A Query based Approach to Reduce the Web Crawler Traffic using HTTP Get Request and Dynamic Web Page

Mishra, Shekhar; Jain, Anurag; Sachan, Amit

doi:10.5120/1826-2406

Cited by 6 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rate of crawling and downloading is compared with general crawling rate and it is observed that instead of all pages of site, only those pages were actually downloaded which were modified after last visit. This scheme was approx 4.83 times faster than simple crawling [16].…”

Section: (F) Algorithms Based On Reducing Traffic On Crawler-mentioning

confidence: 95%

“…When a website is visited and downloaded by the crawler (especially dynamic website) then crawler download only updated pages of that website rather than downloading the whole website again. This query based approach ensures that all updates are being informed to the web crawler to prevent it traversing and downloading whole site again and again that means it reduces number of pages to be visit by the crawler for updation [16]. (F) Other Methods-20.…”

Section: By Http Get Request and Dynamic Webmentioning

confidence: 99%

See 1 more Smart Citation

Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

Shrivastava¹

2019

IJRTE

View full text Add to dashboard Cite

World Wide Web (WWW) is a massive resource of information that contains data in zeta bytes. Large number of users surfing Internet for various purposes need a quick response from web. Search engines are preferable, easily available and reliable source to retrieve data very rapidly with minimum resources. Search engine uses Web Crawler to search abundant data from web. It is a program that automatically passes through hypertext structure of the web and access webpage and hyperlink within it. Web crawling is a technique that focuses on obtaining relevant and fruitful links. To make the traversing accurate and productive, a properly designed web crawling algorithm is required. Web crawler and crawling techniques are kept secret as it is a hidden formula for the success of any search engine. In this paper, we have presented various web crawling algorithms along with their pros and cons to be utilized towards web search. Also we have proposed a novel web crawling algorithm to enhance the security and improving the performance of web crawler using single threaded and multi-threaded crawling algorithms

show abstract

Section: (F) Algorithms Based On Reducing Traffic On Crawler-mentioning

confidence: 95%

Section: By Http Get Request and Dynamic Webmentioning

confidence: 99%

Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

Shrivastava¹

2019

IJRTE

View full text Add to dashboard Cite

show abstract

“…It is a Query based Approach to minimize the Web Crawler or spider Traffic by using HTTP Get Request and also Dynamic Web Page. According to the author it is a query based approach to inform all updates on the web site by web crawler using by Dynamic web page and also HTTP GET Request [17]. And crawler download only updated web pages after the last visit.…”

Section: By Http Get Request and Dynamic Web Pagementioning

confidence: 99%

Untitled

2016

AVC

View full text Add to dashboard Cite

show abstract

“…Systems like Lixto [1], Wargo [11], and WICCAP [9] provide visual platforms of Web database querying. Samples of wrapper induction systems were introduced in [4,13]; and useful approaches in reducing Web crawling traffic can be found in [2,10].…”

Section: Introductionmentioning

confidence: 99%

Accelerating Structured Web Crawling without Losing Data

El-Gamil

Winiwarter

2013

Proceedings of International Conference on Information Integration and Web-Based Applications &Amp; Services

View full text Add to dashboard Cite

Size of retrieved data versus crawling time formulate a wellknown dilemma in the structured Web crawling community. The real challenge within this dilemma is to optimize the settings of a given wrapper to obtain maximum available data in shortest possible time. In this paper, we try to tune these settings, by introducing a threaded algorithm that guarantees accessing all available detail pages within crawling scope; and using this algorithm, we try to reduce the time consumed by the crawler, via simple adjustments of sleeping time after each detail page visit.

show abstract

A Query based Approach to Reduce the Web Crawler Traffic using HTTP Get Request and Dynamic Web Page

Cited by 6 publications

References 8 publications

Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

Untitled

Accelerating Structured Web Crawling without Losing Data

Contact Info

Product

Resources

About