Comparing Topic Coverage in Breadth-First and Depth-First Crawls Using Anchor Texts

Abstract

Abstract. Web archives preserve the fast changing Web by repeatedly crawling its content. The crawling strategy has an influence on the data that is archived. We use link anchor text of two Web crawls created with different crawling strategies in order to compare their coverage of past popular topics. One of our crawls was collected by the National Library of the Netherlands (KB ) using a depth-first strategy on manually selected websites from the .nl domain, with the goal to crawl websites as completes as possible. The second crawl was collected by the Common Crawl foundation using a breadth-first strategy on the entire Web, this strategy focuses on discovering as many links as possible. The two crawls differ in their scope of coverage, while the KB dataset covers mainly the Dutch domain, the Common Crawl dataset covers websites from the entire Web. Therefore, we used three different sources to identify topics that were popular on the Web; both at the global level (entire Web) and at the national level (.nl domain): Google Trends, WikiStats, and queries collected from users of the Dutch historic newspaper archive. The two crawls are different in terms of their size, number of included websites and domains. To allow fair comparison between the two crawls, we created sub-collections from the Common Crawl dataset based on the .nl domain and the KB seeds. Using simple exact string matching between anchor texts and popular topics from the three different sources, we found that the breadth-first crawl covered more topics than the depth-first crawl. Surprisingly, this is not limited to popular topics from the entire Web but also applies to topics that were popular in the .nl domain.

Sistem Pengecekan Toko Online Asli atau Dropship pada Shopee Menggunakan Algoritma Breadth First Search

“…Teknik Breadth First Search dapat digunakan dalam pencarian record dengan menelusuri URL utama hingga ke bagian terdalam link pada halaman artikel yang dilakukan pada media online, dari hasil penelitian menunjukan detik.com merupakan media online yang lebih banyak menghasilkan data yang relevan [8]. Breadth First Search juga lebih unggul daripada Depth First Search dalam mengumpulkan penjelajahan topik popular baik di tingkat global (seluruh Web) dan ditingkat nasional (domain.nl): Google Trends, WikiStats, dan Kueri dikumpulkan dari pengguna arsip surat kabar bersejarah Belanda [9]. Breadth First Search memiliki efisiensi yang baik untuk sisi penjelajahan link URL [10].…”

What's cached is prologue: Reviewing recent web archives research towards supporting scholarly use

“…For example, Brügger (2013a) considers the coverage of material relating to Danish parliamentary elections by comparing historical network graphs available from the Danish Netarkivet collection and the Internet Archive. Samar et al (2016) analyze coverage of trending topics for the Netherlands in 2014 by comparing the National Library of the Netherlands' web archive to the Common Crawl dataset. Milligan et al (2016) use a case study of Canadian federal elections, comparing collections resulting from three different crawling strategies: starting with curated seed lists, collecting URLs from public tweets for a given hashtag, and broad crawls by the Internet Archive.…”