A comparison of methods for collecting web citation data for academic organizations

Abstract

The primary webometric method for estimating the online impact of an organization is to count links to its website. Link counts have been available from commercial search engines for over a decade but this was set to end by early 2012 and so a replacement is needed. This article compares link counts to two alternative methods: URL citations and organization title mentions. New variations of these methods are also introduced. The three methods are compared against each other usingYahoo!.Two of the three methods (URL citations and organization title mentions) are also compared against each other using Bing. Evidence from a case study of 131 UK universities and 49 US Library and Information Science (LIS) departments suggests that Bing's Hit Count Estimates (HCEs) for popular title searches are not useful for webometric research but that Yahoo!'s HCEs for all three types of search and Bing's URL citation HCEs seem to be consistent. For exact URL counts the results of all three methods in Yahoo! and both methods in Bing are also consistent. Four types of accuracy factors are also introduced and defined: search engine coverage, search engine retrieval variation, search engine retrieval anomalies, and query polysemy.

Scholarly Impact Revisited

“…Third, the information science and technology literature suggests that the most important threat to the validity of results based on webometric research is the presence of spurious matches (Thelwall & Sud, 2011). Specifically related to our study, Google may return results that should not be credited to the specific author in question.…”

“…Our methodology followed best-practice recommendations derived from the field of information science and technology, specifically a subfield of study called webometric research (Thelwall & Sud, 2011).…”

“…Just like number of citations, number of pages on the Web is a measure of impact regardless of the reason that a researcher is mentioned. Also, just as number of citations is an indicator of the level of impact on inside stakeholders (i.e., members of the Academy), having very few entries on Google means that outside stakeholders (i.e., people outside of the Academy) are not paying much attention (Thelwall & Sud, 2011).…”

“…Management scholars are very interested in providing answers to these questions because performance management systems in universities (Aguinis, 2013) and the allocation of resources and rewards to individuals and also departments are determined, at least in part, by the impact of their scholarly work (Gomez-Mejia & Balkin, 1992; Higher Education Funding Council of England [HEFCE], 2011). For example, the Academy of Management Review bestows its best-article-of-thedecade award to the article with the greatest relative impact (Crossan, Maurer, & White, 2011), and many universities classify journals in terms of their relative impact (Blackburn & Mitchell, 1981;van Fleet, McWilliams, & Siegel, 2000).…”

Influence of the Academic Library on US University Reputation: A Webometric Approach

“…At present, some research is being pursued on the validity of this indicator as a predictor of external links [35]. Although there are some problems, Google and Yahoo!…”