A large-scale study of robots.txt

Abstract

Search engines largely rely on Web robots to collect information from the Web. Due to the unregulated open-access nature of the Web, robot activities are extremely diverse. Such crawling activities can be regulated from the server side by deploying the Robots Exclusion Protocol in a file called robots.txt. Although it is not an enforcement standard, ethical robots (and many commercial) will follow the rules specified in robots.txt. With our focused crawler, we investigate 7,593 websites from education, government, news, and business domains. Five crawls have been conducted in succession to study the temporal changes. Through statistical analysis of the data, we present a survey of the usage of Web robots rules at the Web scale. The results also show that the usage of robots.txt has increased over time. General TermsExperimentation, Measurement.

A larger scale study of robots.txt

“…In [4,5], the bias was measured by counting the number of directories disallowed, i.e., the crawler with the highest (lowest) such count was regarded as having the most unfavorable (favorable) bias. The drawback of this approach is that the number of directories disallowed may not correlate well to the amount of content or the number of URLs disallowed.…”

“…Despite the importance of this protocol for both content providers and search engines, the first reasonably large scale study for its usage was done only recently in 2007 [4,5]. The study was performed over 2,925 distinct robots.txt files from 7,593 sites.…”

A Forecasting Model for the Pages Crawled by Search Engine Crawlers at a Web Site

“…But it cannot identify search engine visits because Google Analytics track users with the help of JavaScripts and search engine crawlers do not enable the JavaScripts embedded in web pages when the crawlers visit the web sites. A large scale study of robots.txt is done by Sun et al [3] and the ethics of web crawlers is studied by Giles [1]. Schwenke et al has performed a study on the relationship between JavaScript usage and web site visibility to identify whether JavaScript based hyperlinks attract or repel crawlers resulting in an increase or decrease in web site visibility is done by [6].Another study is performed with commercial search engines to find whether there is a significant difference in their coverage of commercial web sites [7].…”

“…Certain crawlers are ethical in their behavior while many are not. Because of the highly automated nature of the robots, rules must be made to regulate such crawling activities to manage the server workload and denying access to confidential or private information [3]. A file called robots.txt is placed at the root of the web site directory which specifies the Robots Exclusion Protocol.…”

Web Site Metadata

“…More recently, a study of initially only 8'000 sites [10,11] has been extended in the BotSeer project and now covers 13.2 million sites [12]. Their finding (in the initial 8'000 site study) of a 38.5% adoption rate of robots.txt files is a little bit smaller than our average of 45.1%, which might be explained by the study's date (October 2006), and also by the fact that the study did not start with the most popular domains, which probably have a higher adoption rate.…”