Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level

Abstract

When research groups are evaluated by an expert panel, it is an open question how one can determine the match between panel and research groups. In this paper, we outline two quantitative approaches that determine the cognitive distance between evaluators and evaluees, based on the journals they have published in. We use example data from four research evaluations carried out between 2009 and 2014 at the University of Antwerp.While the barycenter approach is based on a journal map, the similarity-adapted publication vector (SAPV) approach is based on the full journal similarity matrix. Both approaches determine an entity's profile based on the journals in which it has published. Subsequently, we determine the Euclidean distance between the barycenter or SAPV profiles of two entities as an indicator of the cognitive distance between them. Using a bootstrapping approach, we determine confidence intervals for these distances. As such, the present article constitutes a refinement of a previous proposal that operates on the level of Web of Science subject categories.

Measuring cognitive distance between publication portfolios

“…We therefore use a bootstrapping method (Efron & Tibshirani, 1998) leading to 95% confidence intervals for distances and similarities. Details of the bootstrapping method we applied are explained in (Rahman et al, 2016a are included in Appendix A. These results are recalculated (leading to small differences) and information about the calculated confidence intervals is added.…”

“…Here, as in (Rahman et al, , 2016aWang & Sandström, 2015) we consider the publication portfolio of the involved researchers to reflect the position of the unit in cognitive space and, hence, to determine cognitive distance. Expressed in general terms we measure cognitive distance between units based on how often they published in the same or similar journals.…”

“…In that article, publications were assigned to Web of Science Subject Categories, in short WoS SCs. We admit that the use of WoS SCs was a convenience approach, which has meanwhile been refined by applying a journal level approach (Rahman, Guns, Leydesdorff, & Engels, 2016a). More precisely, instead of assigning publications to WoS SCs, publications were assigned to the journal in which they were published.…”

Cognitive Distances between Evaluators and Evaluees in Research Evaluation: A Comparison between Three Informetric Methods at the Journal and Subject Category Aggregation Level

“…We started to study the problem of quantifying cognitive distance, such that individual PMs' expertise covers the research domains in the discipline where the units of assessment (in our case: research groups) have publications. In our own work, we focused on determining the cognitive distances between publication portfolios of an expert panel and research groups (Rahman et al, 2015(Rahman et al, , 2016Rousseau et al, 2017), while Wang and Sandström (2015) used bibliographic coupling and topic modeling to determine cognitive distance.…”

“…For this purpose, we used the similarity matrix of WoS SCs and a 2-dimensional base map derived from it [for details see Leydesdorff and Rafols (2009), Rafols et al (2010), and Leydesdorff et al (2013a)] and also the similarity matrix of journals and its 2-dimensional base map [for details see Leydesdorff and Rafols (2012) and ]. Hence, we proposed five different approaches namely a barycenter approach using WoS SCs and journals (Rahman et al, 2015(Rahman et al, , 2016, a similarity-adapted publication vector (SAPV) using WoS SCs and journals (Rahman et al, 2016;Rousseau et al, 2017) and a weighted cosine similarity (WCS) approach using WoS SCs (Rousseau et al, 2017). The SAPV and WCS methods use the similarity matrix of WoS SCs/journals while the barycenter method uses the respective 2-dimensional base map derived from the similarity matrix of WoS SCs/journals.…”