Evaluating methods to rediscover missing web pages from the web infrastructure

Klein, Martin; Nelson, Michael L.

doi:10.1145/1816123.1816133

Cited by 14 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though they are expensive to compute, similar to tags, they may provide an alternative if no copy of a missing page can be found in the web infrastructure. Further research in [9,10] has shown that titles of web pages are a very strong alternative to lexical signatures. The results also prove that we can increase the retrieval performance by applying both methods combined.…”

Section: Content and Link Based Methods To Rediscover Web Pagesmentioning

confidence: 99%

“…This establishes a binary relevance case. More precisely, similar to our evaluation in [9] the first performance measure distinguishes between four retrieval cases where the returned URI is:…”

Section: Performance Measurementioning

confidence: 99%

“…We have seen in previous work [9] that for datasets based on randomly sampled URIs tags are very sparse and it is hard to aggregate a somewhat representative corpus. Heymann et al [5] supports this point by showing that compared to a search engine's index the number of URIs annotated with tags is diminishing.…”

Section: Data Gatheringmentioning

confidence: 99%

“…In previously generated corpora containing randomly sampled URIs we experienced that tags were very sparse. In [9] for example we only found tags for 15% of all URIs. This led us to the creation of a new, "tag-centric" corpus introduced here.…”

Section: Introductionmentioning

confidence: 96%

“…In previous work [8,9] we have provided two content based approaches to generate search engine queries that rediscover such missing pages. We are following our intuition that information on the web is rarely completely lost, it is just missing and we can utilize the web infrastructure (search engine, their caches, archives, etc) to rediscover and preserve these resources.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Find, New, Copy, Web, Page - Tagging for the (Re-)Discovery of Web Pages

Klein

Nelson

2011

Research and Advanced Technology for Digital Libraries

Self Cite

View full text Add to dashboard Cite

Abstract. The World Wide Web has a very dynamic character with resources constantly disappearing and (re-)surfacing. A ubiquitous result is the "404 Page not Found" error as the request for missing web pages. We investigate tags obtained from Delicious for the purpose of rediscovering such missing web pages with the help of search engines. We determine the best performing tag based query length, quantify the relevance of the results and compare tags to retrieval methods based on a page's content. We find that tags are only useful in addition to content based methods. We further introduce the notion of "ghost tags", terms used as tags that do not occur in the current but did occur in a previous version of the web page. One third of these ghost tags are ranked high in Delicious and also occurred frequently in the document which indicates their importance to both the user and the content of the document.

show abstract

Section: Content and Link Based Methods To Rediscover Web Pagesmentioning

confidence: 99%

Section: Performance Measurementioning

confidence: 99%

Section: Data Gatheringmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Find, New, Copy, Web, Page - Tagging for the (Re-)Discovery of Web Pages

Klein

Nelson

2011

Research and Advanced Technology for Digital Libraries

Self Cite

View full text Add to dashboard Cite

show abstract

Identifying “Soft 404” Error Pages: Analyzing the Lexical Signatures of Documents in Distributed Collections

Meneses

Furuta

Shipman

2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Collections of Web-based resources are often decentralized; leaving the task of identifying and locating removed resources to collection managers who must rely on http response codes. When a resource is no longer available, the server is supposed to return a 404 error code. In practice and to be friendlier to human readers, many servers respond with a 200 OK code and indicate in the text of the response that the document is no longer available. In the reported study, 3.41% of servers respond in this manner. To help collection managers identify these "friendly" or "soft" 404s, we developed two methods that use a Naïve Bayes classifier based on known valid responses and known 404 responses. The classifier was able to predict soft 404 pages with a precision of 99% and a recall of 92%. We will also elaborate on the results obtained from our study and will detail the lessons learned.Keywords: Soft 404, Web resource management, distributed collections. IntroductionVannevar Bush in his pioneering 1945 essay "As We May Think" [1] envisions a time in which the world's knowledge is accessible by machine and in which the connections that describe the higher-level relationships among sources are themselves objects of scholarship that can be shared with colleagues. We can see this today on the Web, with the utility of resource lists such as Yahoo and the investigation of mechanisms such as our own Walden's Paths [2,3]. Such interconnections of documents is a natural side effect of collaboration and cooperation, so as the problems to be solved grow beyond the technical abilities of an individual scholar and as social media becomes more embedded into our work practices, the presence of resources that situate knowledge into the broader environment will become ever more prevalent. A factor not considered by Bush but critical in today's networked world is that of administrative ownership of data. Information today is not contained in neatlydefined book-like units that can be replicated and stored locally in libraries. Instead the administrative control of information related to a topic may be spread across 198 L. Meneses, R. Furuta, and F. Shipman digital collections maintained by multiple scholars in multiple institutions. Administrative decentralization often is a critical factor in engaging a scholar to put in the work needed to create a valuable resource-the sense of ownership and control is motivating and often a necessary condition both for scholar and also for institution. Some of this need also centers on the desire to have a canonical copy of the resource-multiple copies in multiple locations can, and often do, diverge over time.Administrative decentralization, though, leads to changes that are unexpected by the maintainer of a "meta-resource"-a resource created by tying together the existing resources. Individual collections can change in many ways, both intentional and unintentional. Change may be because of deliberate actions on part of the collector-for example, reorganization of the structure of the collect...

show abstract