“…Yet, in addition to the owners of commercial search engines, they are increasingly used by a widening section of society including casual Web users, the creators of e‐mail spam lists, and others looking for information of commercial value. In addition, many new types of information science research rely upon Web crawlers or automatically downloading pages (e.g., Björneborn, 2004; Faba‐Perez, Guerrero‐Bote, & De Moya‐Anegon, 2003; Foot, Schneider, Dougherty, Xenos, & Larsen, 2003; Heimeriks, Hörlesberger, & van den Besselaar, 2003; Koehler, 2002; Lamirel, Al Shehabi, Francois, & Polanco, 2004; Leydesdorff, 2004; Rogers, 2004; Vaughan & Thelwall, 2003; Wilkinson, Harries, Thelwall, & Price, 2003; Wouters & de Vries, 2004). Web crawlers are potentially very powerful tools, with the ability to cause network problems and incur financial penalties to the owners of the Web sites crawled.…”