Search the past with the portuguese web archive

Abstract

The web was invented to quickly exchange data between scientists, but it became a crucial communication tool to connect the world. However, the web is extremely ephemeral. Most of the information published online becomes quickly unavailable and is lost forever. There are several initiatives worldwide that struggle to archive information from the web before it vanishes. However, search mechanisms to access this information are still limited and do not satisfy their users who demand performance similar to live-web search engines.This demo presents the Portuguese Web Archive, which enables search over 1.2 billion files archived from 1996 to 2012. It is the largest full-text searchable web archive publicly available [17]. The software developed to support this service is also publicly available as a free open source project at Google Code, so that it can be reused and enhanced by other web archivists. A short video about the Portuguese Web Archive is available at vimeo.com/59507267. The service can be tried live at archive.pt.

Desiderata for Exploratory Search Interfaces to Web Archives in Support of Scholarly Activities

“…Obviously, browsing is only useful if one knows the exact URL of the desired content. Since this is often not the case, search is an obvious solution [4,6], but unfortunately, most web archives do not support full-text search. There has been academic work on searching timestamped collections (such as web archives) [12,10,2,8], but these systems have not been deployed in production to our knowledge.…”

Infrastructure for supporting exploration and discovery in web archives

“…Since this is often not the case, temporal search capabilities are the next most desired feature in web archives [7]. Unfortunately, most web archives do not support full-text search, with a few exceptions such as the Portuguese Web Archive [9], the British Library, and the Internet Archive's Archive-It service. There has been academic work on searching timestamped collections [18,12,3,11,22], but these systems have not been deployed in production at scale.…”

“…Similarly, a web page should only receive an inbound link from a contemporaneous page. We solve the problem in a manner similar to Gomes et al [9] by leveraging the fact that most web content is gathered in periodic (e.g., monthly) crawls, and thus we can process the webgraph from each crawl separately. We can easily determine where these "break points" should be with a Hadoop job to build a histogram of document timestamps.…”

The ARCOMEM Architecture for Social- and Semantic-Driven Web Archiving

“…It adapts Nutch by searching against Web archives rather than crawling the Web. The Portuguese Web Archive information retrieval system [63] adapted NutchWAX to support versions of the same URL across time and to improve the ranking of results.…”