Moved but not gone: an evaluation of real-time methods for discovering replacement web pages

Klein, Martin; Nelson, Michael L.

doi:10.1007/s00799-014-0108-0

Cited by 17 publications

(11 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The correlations between the features are considered towards data retrieval. In [7], different methods of retrieval have been evaluated for their performance under standalone and combined way. Toward corpus retrieval of newspapers, an log based approach is presented in [8].…”

Section: Literature Reviewmentioning

confidence: 99%

Hierarchical Semantic Relational Coverage Measure Based Web Document Clustering Using Semantic Ontology

Selvalakshmi¹,

Subramaniam²

2019

IJITEE

View full text Add to dashboard Cite

The problem of web document clustering has been well studied. Web documents has been grouped based various features like textual, topical and semantic features. Number of approaches has been discussed earlier for the clustering of web documents. However the method does not produce promising results towards web document clustering. To overcome this, an efficient hierarchical semantic relational coverage based approach is presented in this paper. The method extracts the features of web document by preprocessing the document. The features extracted have been used to measure the semantic relational coverage measure in different levels. As the documents are grouped in a hierarchical manner, the method estimates the relational coverage measure in each level of the cluster. Based on the semantic relational measure at different level, the method estimates the topical semantic support measure. Using these two, the method computes the class weight. The estimated class weight has been used to perform document clustering. The proposed method improves the performance of document clustering and reduces the false classification ratio.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Hierarchical Semantic Relational Coverage Measure Based Web Document Clustering Using Semantic Ontology

Selvalakshmi¹,

Subramaniam²

2019

IJITEE

View full text Add to dashboard Cite

show abstract

“…In summary, anchor texts are related to real queries, and target documents' titles. In addition to this, anchor text is available not only for pages in the archive, but also for pages that have not been archived when there are pointers to them from pages in the Web archive [29,33,42].…”

Section: Query Setmentioning

confidence: 99%

Quantifying retrieval bias in Web archive search

et al. 2017

View full text Add to dashboard Cite

A Web archive usually contains multiple versions of documents crawled from the Web at different points in time. One possible way for users to access a Web archive is through full-text search systems. However, previous studies have shown that these systems can induce a bias, known as the retrievability bias, on the accessibility of documents in community-collected collections (such as TREC collections). This bias can be measured by analyzing the distribution of the retrievability scores for each document in a collection, quantifying the likelihood of a document's retrieval. We investigate the suitability of retrievability scores in retrieval systems that consider every version of a document in a Web archive as an independent document. We show that the retrievability of documents can vary for different versions of the same document and that retrieval systems induce biases to different extents. We quantify this bias for a retrieval system which is adapted to handle multiple versions of the same document. The retrieval system indexes each version of a document independently, and we refine the search results using two techniques to aggregate similar versions. The first approach is to collapse similar versions of a document based on content similarity. The second approach is to collapse all versions of the same document based on their URLs. In both cases, we found that the degree of bias Universiteit Utrecht, Utrecht, The Netherlands is related to the aggregation level of versions of the same document. Finally, we study the effect of bias across time using the retrievability measure. Specifically, we investigate whether the number of documents crawled in a particular year correlates with the number of documents in the search results from that year. Assuming queries are not inherently temporal in nature, the analysis is based on the timestamps of documents in the search results returned using the retrieval model for all queries. The results show a relation between the number of documents per year and the number of documents retrieved by the retrieval system from that year. We further investigated the relation between the queries' timestamps and the documents' timestamps. First, we split the queries into different time frames using a 1-year granularity. Then, we issued the queries against the retrieval system. The results show that temporal queries indeed retrieve more documents from the assumed time frame. Thus, the documents from the same time frame were preferred by the retrieval system over documents from other time frames.

show abstract

“…Links and anchor texts can be used to locate missing webpages, of which the original URL is not accessible anymore. Klein and Nelson [13] computed lexical signatures of lost webpages, using the top n words of link anchors, and used these and other methods to retrieve alternative URLs for lost webpages. The use of the link structure and anchor texts to uncover and reconstruct target pages that were not archived was studied in [9], based on a depth-first crawl of manually selected websites.…”

Section: Related Workmentioning

confidence: 99%

Comparing Topic Coverage in Breadth-First and Depth-First Crawls Using Anchor Texts

Samar

Traub

Ossenbruggen

et al. 2016

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

Abstract. Web archives preserve the fast changing Web by repeatedly crawling its content. The crawling strategy has an influence on the data that is archived. We use link anchor text of two Web crawls created with different crawling strategies in order to compare their coverage of past popular topics. One of our crawls was collected by the National Library of the Netherlands (KB ) using a depth-first strategy on manually selected websites from the .nl domain, with the goal to crawl websites as completes as possible. The second crawl was collected by the Common Crawl foundation using a breadth-first strategy on the entire Web, this strategy focuses on discovering as many links as possible. The two crawls differ in their scope of coverage, while the KB dataset covers mainly the Dutch domain, the Common Crawl dataset covers websites from the entire Web. Therefore, we used three different sources to identify topics that were popular on the Web; both at the global level (entire Web) and at the national level (.nl domain): Google Trends, WikiStats, and queries collected from users of the Dutch historic newspaper archive. The two crawls are different in terms of their size, number of included websites and domains. To allow fair comparison between the two crawls, we created sub-collections from the Common Crawl dataset based on the .nl domain and the KB seeds. Using simple exact string matching between anchor texts and popular topics from the three different sources, we found that the breadth-first crawl covered more topics than the depth-first crawl. Surprisingly, this is not limited to popular topics from the entire Web but also applies to topics that were popular in the .nl domain.

show abstract

Moved but not gone: an evaluation of real-time methods for discovering replacement web pages

Cited by 17 publications

References 62 publications

Hierarchical Semantic Relational Coverage Measure Based Web Document Clustering Using Semantic Ontology

Hierarchical Semantic Relational Coverage Measure Based Web Document Clustering Using Semantic Ontology

Quantifying retrieval bias in Web archive search

Comparing Topic Coverage in Breadth-First and Depth-First Crawls Using Anchor Texts

Contact Info

Product

Resources

About