Value dissonance in research(er) assessment: Individual and institutional priorities in review, promotion and tenure criteria

Ross-Hellauer, Tony; Klebel, Thomas; Knoth, Petr; Pontika, Nancy

doi:10.31235/osf.io/dcr8v

Cited by 4 publications

(6 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data was collected via a survey instrument targeting active researchers, distributed internationally from 29th June to 30th July 2021. The full survey, elements of which have been published in (Ross-Hellauer et al 2023), examined participants' perceptions of review, promotion and tenure criteria and practices related to Open Science and Responsible Research and Innovation. The instrument is included within the published dataset ).…”

Section: Methodsmentioning

confidence: 99%

“…RPT criteria are usually centred on three main types of activities: teaching, service and research. Considering criteria regarding research, although a quickly growing body of work demonstrates the need for greater support for open and responsible practices in RPT criteria (Alperin et al 2022;Rice et al 2020;Schimanski and Alperin 2018;Pontika, Klebel, Correia, et al 2022;Ross-Hellauer et al 2023;Moher et al 2018;, or general support for the aim of reducing quantification, in terms of number of publications, research funding, or proxy measures for quality like the Journal Impact Factor (DORA 2012;Hicks et al 2015;Wilsdon et al 2015), less is known about how criteria are applied in practice.…”

Section: Introductionmentioning

confidence: 99%

“…Full analyses of respondent demographics are available in Supplement 1, section 1 of(Ross-Hellauer et al 2023).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Ross-Hellauer¹,

Bonn²,

Horbach³

2023

Preprint

View full text Add to dashboard Cite

Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal assessment criteria, building to a consensus that less emphasis should be placed on quantification, while supporting open and diverse sets of criteria. Yet the theory and evidence upon which such policy reform depends is still surprisingly sparse. Based on qualitative free-text responses from 121 respondents gathered during an international survey of active researchers, this study examines how criteria are applied in practice and how those being assessed perceive informal criteria to determine the outcomes of assessments. While confirming the general critique on over-emphasising quantification, respondents particularly identify a mismatch between formal criteria and actual evaluation practices. Hidden criteria, including social, political, and demographic factors, are perceived important, especially in intransparent assessment procedures, opening up for assessors’ interpretative flexibility. This adds to ongoing discussions on the performativity of assessment criteria and lays bare a tension between the rigidity and flexibility of criteria and the extent to which these can be transparently communicated.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Ross-Hellauer¹,

Bonn²,

Horbach³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Despite the growing visibility of assessment reform movements in some quarters, the fields of science studies and research on research have only just begun to venture into this emerging reform landscape (Pontika et al, 2022, Schönbrodt et al, 2022, Ross-Hellauer et al, 2023. Prominent interventions in the responsible metrics movement include the San Francisco Declaration on Research Assessment (DORA, 2013), the Leiden Manifesto (Hicks et al, 2015) and the Metric Tide report (Wilsdon, 2016).…”

Section: Introductionmentioning

confidence: 99%

Practicing Responsible Research Assessment: Qualitative study of Faculty Hiring, Promotion, and Tenure Assessments in the United States

Rushforth¹,

Rijcke²

2023

Preprint

View full text Add to dashboard Cite

Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a major problem and seeks to steer and improve practices around their use. It is a multi-actor, multi-disciplinary reform movement premised upon engendering a sense of responsibility among academic evaluators to approach metrics with caution and avoid certain poor practices. In this article we identify how academic evaluators engage with the responsible metrics agenda, via semi-structured interview and open-text survey responses on professorial hiring, tenure and promotion assessments among senior academics in the United States – a country that has so far been less visibly engaged with the responsible metrics reform agenda. We explore how notions of ‘responsibility’ are experienced and practiced among the very types of professionals international reform initiatives such as the San Francisco Declaration on Research Assessment (DORA) are hoping to mobilize into their cause. In doing so, we draw on concepts from science studies, including from literatures on responsible research and innovation and ‘folk theories’ of citation. We argue that literature on citation folk theories should extend its scope beyond simply asking researchers how they view the role and validity of these tools as performance measures, by asking them also what they consider are their professional obligations to handle bibliometrics appropriately.

show abstract

“…Change in research assessment was also singled out as the most substantial obstacle for change in scholarly publishing and scholarly communication in the final report of a recent expert group of which I was part (European Commission, Directorate‐General for Research and Innovation, 2019). Despite the buzz around the San Franscisco Declaration of Research Assessment, it is still common for institutions to lack integration of open research practices as part of research assessment (Ross‐Hellauer et al., 2023). One issue has been the lack of taking tangible steps to fundamentally changing practices rather than just signaling support towards the greater cause of diversification in research assessment.…”

mentioning

confidence: 99%

Alignment of Top‐Down Policies With Emerging Bottom‐Up Practices: A Commentary on “(Why) Are Open Research Practices the Future for the Study of Language Learning?”

null

2023

Language Learning

View full text Add to dashboard Cite

In their article, Marsden and Morgan-Short comprehensively review the current state and development trajectories for key areas within open research practices, both in general as well as more particularly in the context of language sciences. As the article reveals, the scope of open research practices is enormous and essentially touches upon every aspect of performing and interacting with research. The authors touch upon the lack of an established metascience within language sciences that would help inform and guide development of research practices, but, as I see it, the problem is universal, and there would be benefit in creating a stronger and more cohesive metascience discipline in general. While researchers have established practices of research, education, and dedicated scholarly communication outlets within the philosophy of science, history of science, information science, and higher education policy, metascience has remained an area where the discussion is highly distributed and appears sporadically across diverse research disciplines. As Marsden and Morgan-Short's review demonstrates, there are a lot of open questions relating

show abstract

Value dissonance in research(er) assessment: Individual and institutional priorities in review, promotion and tenure criteria

Cited by 4 publications

References 63 publications

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Practicing Responsible Research Assessment: Qualitative study of Faculty Hiring, Promotion, and Tenure Assessments in the United States

Alignment of Top‐Down Policies With Emerging Bottom‐Up Practices: A Commentary on “(Why) Are Open Research Practices the Future for the Study of Language Learning?”

Contact Info

Product

Resources

About