A reliability based approach for securing migrating crawlers

Singhal, Niraj; Dixit, Ashutosh; Agarwal, Rohit; Sharma, Ashok Kumar

doi:10.1007/s41870-017-0065-0

Cited by 4 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are many types of crawler exist such as Parallel Crawler, Migrating Crawler, Focussed Crawler, Hidden Crawler, Incremental Crawler etc. In this proposed work, the capabilities of migrating crawler (Singhal et al, 2012) are being utilized for designing an efficient crawling system. The proposed Crawler tries to achieve all the design issues needed for efficient crawling.…”

mentioning

confidence: 99%

Design of a Migrating Crawler Based on a Novel URL Scheduling Mechanism using AHP

Punj

Dixit

2017

International Journal of Rough Sets and Data Analysis

View full text Add to dashboard Cite

In order to manage the vast information available on web, crawler plays a significant role. The working of crawler should be optimized to get maximum and unique information from the World Wide Web. In this paper, architecture of migrating crawler is proposed which is based on URL ordering, URL scheduling and document redundancy elimination mechanism. The proposed ordering technique is based on URL structure, which plays a crucial role in utilizing the web efficiently. Scheduling ensures that URLs should go to optimum agent for downloading. To ensure this, characteristics of both agents and URLs are taken into consideration for scheduling. Duplicate documents are also removed to make the database unique. To reduce matching time, document matching is made on the basis of their Meta information only. The agents of proposed migrating crawler work more efficiently than traditional single crawler by providing ordering and scheduling of URLs.

show abstract

mentioning

confidence: 99%