Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models

Abstract

ObjectivesEffective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.DesignWe systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.Data sourcesAll Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017.Eligibility criteria(1) English language, (2) published in the last 10 years (2007–2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher’s achievements.Data extraction and synthesisArticles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen’s Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848–0.9039). Following screening, selected articles underwent full-text review and bias was assessed.ResultsFour hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or ‘altmetric’ data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.ConclusionsAll metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.

Three Commonly Utilized Scholarly Databases and a Social Network Site Provide Different, But Related, Metrics of Pharmacy Faculty Publication

“…This may suggest that the impact of scholarly activity (by citations alone) is not completely correlated with and not fully dependent on the number of publications by a given faculty member [40]. This consideration has been explored by others with proposed alternatives to explore scholarly impact, such as combined article-level metrics, web-based metrics, social-media-based metrics, and comprehensive approaches [41][42][43]. Future work should identify the predictors of total citations or highly cited articles amongst college of pharmacy faculty along with the effect of the database.…”

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

“…Many authors advocated using a mixture of methods and in particular, the triangulation of surveys, interviews (of researchers or research users), and documentary analysis [ 20 , 30 – 32 ]. A large number of reviews cautioned against the use of quantitative metrics, such as bibliometrics, alone [ 29 , 30 , 41 – 48 ]. Concerns included that these metrics were often not designed to be comparable between research programmes [ 49 ], their use may incentivise researchers to focus on quantity rather than quality [ 42 ], and these metrics could be gamed and used in the wrong context to make decisions about researcher funding, employment and promotion [ 41 , 43 , 45 ].…”

Evaluating Scholars’ Impact and Influence: Cross-sectional Study of the Correlation Between a Novel Social Media–Based Score and an Author-Level Citation Metric

“…The altmetrics approach offers new ways to analyze and inform scholarship [ 7 ]. It complements rather than replaces traditional indicators of a scholar’s performance [ 8 ]. Altmetrics have even been adopted aggressively by traditional publishing companies [ 9 ].…”

“…Academic merit and achievement should be appraised using frameworks such as the comprehensive researcher achievement model (CRAM) [ 8 ], encompassing a combination of four aspects: quantity of researcher outputs (productivity), value of outputs (quality), outcomes of research outputs (impact), and relations between publications or authors and the wider world (influence). Current traditional benchmarks focus mostly on productivity and quality, while alternative metrics focus on influence and impact.…”