Efficient, automatic web resource harvesting

Abstract

There are two problems associated with conventional web crawling techniques: a crawler cannot know if all resources at a non-trivial web site have been discovered and crawled ("the counting problem") and the human-readable format of the resources are not always suitable for machine processing ("the representation problem"). We introduce an approach that solves these two problems by implementing support for both the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and MPEG-21 Digital Item Declaration Language (DIDL) into the web server itself. We present the Apache module "mod oai", which can be used to address the counting problem by listing all valid URIs at a web server and efficiently discovering updates and additions on subsequent crawls. Our experiments indicated comparable performance for initial crawls, and dramatic increases in update speed. mod oai can also be used to address the representation problem by providing "preservation ready" versions of web resources aggregated with their respective forensic metadata in MPEG-21 DIDL format.

Detecting Emergent Conflicts through Web Mining and Visualization

“…This work was supported by the R&D programme of the Swedish Armed Forces and by Recorded Future AB, and is also a supporting work for the Alert4All research project 4 , which is funded under the European Union Seventh Framework Programme under contract no 261732.…”

Using timed-release cryptography to mitigate the preservation risk of embargo periods

“…Thus, an item can contain more than one record, each containing metadata in a specific metadata format pertaining to one resource. This hierarchy is further clarified by Figure 1 taken from [46]. Sets are repository-defined collections of these items.…”

“…It provides selective harvesting through from-until parameters, so that only the desired metadata may be harvested from a data provider. This feature has enhanced efficiency by facilitating harvest of data from the previous repository update instead of a complete repository harvest with redundant copies of records that may already exist in the harvesters data collection [78,46]. This is achieved by set-based harvesting or by date-based harvesting of records.…”

Generating best-effort preservation metadata for web resources at time of dissemination

“…We can use this format as a container, or CRATE, for our preservation-prepared resource. Since OAI-PMH was already implemented as an Apache web server module [4], mod oai, we used it as our experimental prototype. Like other Apache modules, mod oai activity is controlled through the web server configuration file (httpd.conf).…”