Time-Aware Focused Web Crawling

Research and Advanced Technology for Digital Libraries

2017

Abstract. Web archives are typically very broad in scope and extremely large in scale. This makes data analysis appear daunting, especially for non-computer scientists. These collections constitute an increasingly important source for researchers in the social sciences, the historical sciences and journalists interested in studying past events. However, there are currently no access methods that help users to efficiently access information, in particular about specific events, beyond the retrieval of individual disconnected documents. Therefore we propose a novel method to extract event-centric document collections from large scale Web archives. This method relies on a specialized focused extraction algorithm. Our experiments on the German Web archive (covering a time period of 19 years) demonstrate that our method enables the extraction of event-centric collections for different event types.

“…Our method is related to crawling methods for creating Web Archives (e.g. [17,24]), as well as to methods for temporal information retrieval [5].…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Extracting Event-Centric Document Collections from Large-Scale Web Archives

Research and Advanced Technology for Digital Libraries

2017

“…Extensions of this model use ontologies to incorporate semantic knowledge into the matching process [12,11], 'tunnel' between disjoint page clusters [5,21] or learn navigation structures necessary to find relevant pages [9,14]. In the recently proposed time-aware focused crawling [19] time is used as a primary focusing criteria. Here the crawler is guided to follow links that are related to a target time, but the topical relevance is not considered.…”

Section: Related Workmentioning

confidence: 99%

“…In contrast, focused Web crawlers [e.g. 3,8] take topical or temporal [19] dimensions of the collected pages into consideration.…”

Section: Introductionmentioning

confidence: 99%

iCrawl

Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries

2015

Researchers in the Digital Humanities and journalists need to monitor, collect and analyze fresh online content regarding current events such as the Ebola outbreak or the Ukraine crisis on demand. However, existing focused crawling approaches only consider topical aspects while ignoring temporal aspects and therefore cannot achieve thematically coherent and fresh Web collections. Especially Social Media provide a rich source of fresh content, which is not used by state-of-the-art focused crawlers. In this paper we address the issues of enabling the collection of fresh and relevant Web and Social Web content for a topic of interest through seamless integration of Web and Social Media in a novel integrated focused crawler. The crawler collects Web and Social Media content in a single system and exploits the stream of fresh Social Media content for guiding the crawler.

“…Focused crawlers [1,4] enable the efficient creation of topically and temporally coherent document collections from the Web and Social Media. Such collections are increasingly used in many domains such as digital sociology, history, politics, and journalism [2,5].…”

Section: Introductionmentioning

confidence: 99%

The iCrawl Wizard – Supporting Interactive Focused Crawl Specification

Lecture Notes in Computer Science

2015

Abstract. Collections of Web documents about specific topics are needed for many areas of current research. Focused crawling enables the creation of such collections on demand. Current focused crawlers require the user to manually specify starting points for the crawl (seed URLs). These are also used to describe the expected topic of the collection. The choice of seed URLs influences the quality of the resulting collection and requires a lot of expertise. In this demonstration we present the iCrawl Wizard, a tool that assists users in defining focused crawls efficiently and semiautomatically. Our tool uses major search engines and Social Media APIs as well as information extraction techniques to find seed URLs and a semantic description of the crawl intent. Using the iCrawl Wizard even non-expert users can create semantic specifications for focused crawlers interactively and efficiently.