Evaluation of crawling policies for a web-repository crawler

McCown, Frank; Nelson, Michael L.

doi:10.1145/1149941.1149972

Cited by 24 publications

(18 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ten randomly selected repositories running EPrints were selected from the Registry of Open Access Repositories (ROAR) [27]. The same methodology from previous reconstruction experiments were used to determine reconstruction success [19,16,22]: the repositories were crawled using the Heritrix web crawler [23] and then reconstructed with Warrick, and the reconstructions were compared to the crawled sites. Of course the reconstructions were only able to recover the client-side representation of the sites and none of the server components.…”

Section: Reconstructing 10 Digital Repositoriesmentioning

confidence: 99%

“…For 19 weeks, the Monarch Repository was crawled with Heritrix and reconstructed with Warrick (using the Comprehensive policy which attempts to recover a maximum of all lost resources [19]) at the end of the week as was performed in the previous reconstruction experiment. The crawls were matched with the reconstructions to produce an accurate assessment as to how much of the website was being successfully reconstructed each week.…”

Section: Setupmentioning

confidence: 99%

See 1 more Smart Citation

Recovering a website's server components from the web infrastructure

McCown

Nelson

2008

Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

Our previous research has shown that the collective behavior of search engine caches (e.g., Google, Yahoo, Live Search) and web archives (e.g., Internet Archive) results in the uncoordinated but large-scale refreshing and migrating of web resources. Interacting with these caches and archives, which we call the Web Infrastructure (WI), allows entire websites to be reconstructed in an approach we call lazy preservation. Unfortunately, the WI only captures the client-side view of a web resource. While this may be useful for recovering much of the content of a website, it is not helpful for restoring the scripts, web server configuration, databases, and other server-side components responsible for the construction of the website's resources. This paper proposes a novel technique for storing and recovering the server-side components of a website from the WI. Using erasure codes to embed the server-side components as HTML comments throughout the website, we can effectively reconstruct all the server components of a website when only a portion of the client-side resources have been extracted from the WI. We present the results of a preliminary study that baselines the lazy preservation of ten EPrints repositories and then examines the preservation of an EPrints repository that uses the erasure code technique to store the server-side EPrints software throughout the website. We found nearly 100% of the EPrints components were recoverable from the WI just two weeks after the repository came online, and it remained recoverable four months after it was "lost".

show abstract

Section: Reconstructing 10 Digital Repositoriesmentioning

confidence: 99%

Section: Setupmentioning

confidence: 99%

Recovering a website's server components from the web infrastructure

McCown

Nelson

2008

Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, we reconstructed an academic conference website that was lost due to a fire [12]. On behalf of the Library of Congress, we reconstructed a Congressman's website when it was suddenly shutdown due to allegations of misconduct [8].…”

Section: Background and Related Workmentioning

confidence: 99%

“…Seven different hosts are currently deployed, allowing seven reconstructions to be running concurrently. Warrick is executed using the Exhaustive policy [12] which means all four web repositories are initially asked to list all URLs they have stored for a website (we call these lister queries). This discovery process is somewhat limited for large websites since search engines will only reveal at most 1000 URLs they have cached.…”

Section: Reconstruction Mechanicsmentioning

confidence: 99%

Usage analysis of a public website reconstruction tool

McCown

Nelson

2008

Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

The Web is increasingly the medium by which information is published today, but due to its ephemeral nature, web pages and sometimes entire websites are often "lost" due to server crashes, viruses, hackers, run-ins with the law, bankruptcy and loss of interest. When a website is lost and backups are unavailable, an individual or third party can use Warrick to recover the website from several search engine caches and web archives (the Web Infrastructure). In this short paper, we present Warrick usage data obtained from Brass, a queueing system for Warrick hosted at Old Dominion University and made available to the public for free. Over the last six months, 520 individuals have reconstructed more than 700 websites with 800K resources from the Web Infrastructure. Sixty-two percent of the static web pages were recovered, and 41% of all website resources were recovered. The Internet Archive was the largest contributor of recovered resources (78%).

show abstract

“…A URL selected by the crawler from the frontier, downloads the web resource, collects URLs from the downloaded resource and adds the new URLs to the frontier. The crawler proceeds in this manner till the frontier is empty or some other condition causes it to stop [15], [16]. Duplicate and near duplicate web page detection becomes an important step in web Crawling [5].…”

Section: Introductionmentioning

confidence: 99%

Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Kumar¹,

Govindarajulu²

2013

IJCIS

View full text Add to dashboard Cite

Duplicate and near-duplicate web pages are the chief concerns for web search engines. In reality, they incur enormous space to store the indexes, ultimately slowing down and increasing the cost of serving results. A variety of techniques have been developed to identify pairs of web pages that are "similar" to each other. The problem of finding near-duplicate web pages has been a subject of research in the database and web-search communities for some years. In order to identify the near duplicate web pages, we make use of sentence level features along with fingerprinting method. When a large number of web documents are in consideration for the detection of web pages, then at first, we use K-mode clustering and subsequently sentence feature and fingerprint comparison is used. Using these steps, we exactly identify the near duplicate web pages in an efficient manner. The experimentation is carried out on the web page collections and the results ensured the efficiency of the proposed approach in detecting the near duplicate web pages.

show abstract

Evaluation of crawling policies for a web-repository crawler

Cited by 24 publications

References 40 publications

Recovering a website's server components from the web infrastructure

Recovering a website's server components from the web infrastructure

Usage analysis of a public website reconstruction tool

Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Contact Info

Product

Resources

About