Are Mendeley reader counts high enough for research evaluations when articles are published?

Abstract

Design/methodology/approach -Average Mendeley reader counts were compared to average Scopus citation counts for 104520 articles from ten disciplines during the second half of 2016. Findings -Articles attracted, on average, between 0.1 and 0.8 Mendeley readers per article in the month in which they first appeared in Scopus. This is about ten times more than the average Scopus citation count. Research limitations/implications -Other subjects may use Mendeley more or less than the ten investigated here. The results are dependent on Scopus's indexing practices, and Mendeley reader counts can be manipulated and have national and seniority biases. Practical implications -Mendeley reader counts during the month of publication are more powerful than Scopus citations for comparing the average impacts of groups of documents but are not high enough to differentiate between the impacts of typical individual articles. Originality/value -This is the first multi-disciplinary and systematic analysis of Mendeley reader counts from the publication month of an article. IntroductionAcademic research is evaluated for appointment, promotion, tenure, for university league tables, for national research evaluation exercises and for self-reflection purposes. Some of these use quantitative data or are supported by numerical evidence of impact. Citation counts to refereed journal articles are a common source of this quantitative data, including in the form of Journal Impact Factors (JIFs) and field normalised citation counts (Garfield, 2006;Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011;Wilsdon, et al., 2015). Citation counts are not suitable for helping to evaluate new research because articles may take three years to attract a substantial number of citations due to publication delays. For this reason, formal evaluations often use a citation window of considerable length, such as three years (Wang, 2013), which excludes newer articles from evaluations. This means that the most recent and, therefore, most relevant research cannot be evaluated with the help of most citation-based indicators because they cannot differentiate effectively between different levels of impact for individual articles.Two solutions to this problem are to use publishing journal JIFs (or journal rankings: Kulczycki, 2017) as a proxy for citation impact or to use web-based early impact indicators. JIFs can avoid citing article publication delays if it is accepted that the average impact of a journal is an appropriate proxy for the impact of its articles (but see: Lozano, Larivière, & Gingras, 2012; and note also the time dimension: Larivière, Archambault, & Gingras, 2008) 1 Thelwall, M. (in press). Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib Journal of Information Management, 69(2).

Are Mendeley reader counts useful impact indicators in all fields?

“…From one survey, 8% of scientists and engineers have joined and 5% in the social sciences, arts and humanities (Van Noorden, 2014). This is not a problem in practice, however, because reader counts tend to be higher than citation counts for recent articles (Thelwall, 2017a) and, as mentioned above, the two tend to correlate. From these percentages, the number of readers of a social sciences paper could be estimated by multiplying its Mendeley reader count by 1/0.05=20 and for a science paper multiplying by 1/0.08=12.5.…”

“…Researchers may add an article to their Mendeley library at the time that they read it or even beforehand (Mohammadi, Thelwall, & Kousha, 2016), which largely removes the publication lag (Maflahi & Thelwall, 2016, in press;Thelwall & Sud, 2016;Thelwall, 2017a). Mendeley reader counts are suitable substitutes for citation counts because they correlate highly with them in many fields (Haustein, Larivière, Thelwall, Amyot, & Peters, 2014;Mohammadi, & Thelwall, 2014;Thelwall & Wilson, 2016;Zahedi, Costas, & Wouters, 2014;Zahedi, Costas, & Wouters, 2017), suggesting that they also indicate both scholarly influence and peer quality judgements.…”

Testing for universality of Mendeley readership distributions

“…These two considerationsthe tradeoff between timeliness and robustness of impact evaluation, and the ability to observe impact even outside the scholarly communitybring about the evident interest of scientometricians in the so-called "altmetrics" (Fenner, 2014;Sugimoto, Work, Larivière, & Haustein, 2017;Thelwall, 2017a;Thelwall, 2017b). Known mainly for their ease of use and the wide range of indicators, many scholars are persuaded that altmetrics offer an alternative view of impact, compared to traditional bibliometric indicators.…”

Research on the impact evaluation of academic journals based on altmetrics and citation indicators

“…In addition, academic paper citations are lagging behind online browsing, sharing, comments and other behaviors, which also leads to lagging results for journals being cited. However, the timeliness of journals evaluated based on altmetrics is significantly better than citation indicators (Maflahi & Thelwall, ; Thelwall, ). At the same time, because of their strong positive correlation, it is known that the Altmetrics score has the certain predictability to the citation score.…”